Having determined what needs to be unit-tested, and generated mechanisms for making sure that the coverage of those code-entities is, itself, tested, it's time to turn some attention to how the testing should be managed. In other words, answering:
What needs to happen in order to test those elements to my satisfaction?
What is Unit-Testing Supposed to Do, Anyway?
Ultimately, the goal of a unit-test run could probably be well described as
assuring that all of the possible paths through the code being tested have
been executed and behaved in an expected fashion.
Consider this example
function:
def ExampleFunction( numberValue, textValue=None ):
# Type- and value-check the arguments
if type( numberValue ) not in ( int, long ):
raise TypeError( 'numberValue must be an integer or '
'long-integer value' )
if textValue != None and type( textValue ) not in (str, unicode ):
raise TypeError( 'textValue must be a string or '
'unicode value, or None' )
if numberValue < 1:
raise ValueError( 'numberValue must be positive' )
if textValue.strip() == '':
raise ValueError( 'textValue cannot be empty or only '
'white-space characters' )
# Do the actual stuff...
if textValue:
if numberValue % 2:
# Do odd-numbered cases
chars = [ c for c in textValue ]
chars.reverse()
textValue = ''.join( chars )
return textValueIn order to assure that all the paths through the code of the function are
executed, what values would have to be passed for the numberValue
and textValue arguments? That is one of the key questions
that should be asked for any unit-test process. The other would be some
variation of what assertions need to be made against the returned value for
each variation?
To answer the argument-values-passed question, I usually think in terms of
good
and bad
values. For this function, those are pretty simple:
| Argument | Value-groups | |
|---|---|---|
Goodvalues |
Badvalues |
|
numberValue |
Any int > 0;Any long > 0;Should test both even and odd values |
Any int < 1;Any long < 1;Any value of any type other than int or
long (floats,
str or unicode values,
instances of object, etc.)
|
textValue |
None (the default value);Any non-empty str value;Any non-empty unicode value; |
'', u'' (empty text values);' ', u' ', '\n',
u'\n', '\t', u'\t'
(only-white-space text-values);Any value of any type other than str or
unicode
|
The point of this is that it's perfectly legitimate to test with a representative
subset
of all the possible values for a given type.
And, in the interests of consistency (and not having to create the same lists
of good
and bad
values from scratch over and over again) I'd like
to have them available as some sort of relatively constant set of values,
defined once, that could be easily filtered down to the relevant values
as needed for any given good
or bad
value-set.
That's my goal for today's post. Call it a UnitTestValuePolicy.
Defining the UnitTestValuePolicy object
The strategy I'm going to pursue is to provide a baseline, default instance of
the class, as a constant named UnitTestValuePolicy. That instance
will be populated with the default unit-test values of all of the simple
types that I expect test-methods to use, and will provide a number of filtering
members (I'm thinking properties initially, if possible) that can be chained
together to yield a final collection of values for any test-type that I can think
of at present. The defaults that I'm contemplating at this point are:
_genericObject = object()
{
'bools':[ True, False ],
'falseish':[ 0.0, 0, 0L, None, '', u'', False ],
'floats':[ -1.0, 0.0, 1.0, 2.0 ],
'ints':[ -1, 0, 1, 2 ],
'longs':[
-sys.maxint *2 - 1, -1L, 0L, 1L, 2L, sys.maxint *2
],
'none':[ None ],
'objects':[ _genericObject ],
'strings':[
'',
' ',
'\t',
'\r',
'\n',
'word',
'multiple words',
'A complete sentence.',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab',
'Multiline\nstring',
],
'trueish':[ 1.0, 0.5, 1, 1L, 'a', u'a',
_genericObject, True ],
'unicodes':[
u'',
u' ',
u'\t',
u'\r',
u'\n',
u'word',
u'multiple words',
u'A complete sentence.',
u'Multiple sentences. Separated with punctuation.',
u'String\tcontaining a tab',
u'Multiline\nstring',
],
}- Getting a list of positive whole numbers:
goodNumbers = ( UnitTestValuePolicy.Integers + UnitTestValuePolicy.Longs ).Positive
which would yield the'ints'list plus the'longs'list, then filter those down to only the values greater than or equal to zero:[ 1, 2, 1L, 2L, sys.maxint ] - Getting a similar list of all string- and unicode values, and
removing any that are empty, or that only contain whitespace, and
adding the
'none'list to that result (sinceNoneis a valid value for thetextValueargument):
goodTexts = ( UnitTestValuePolicy.Strings + UnitTestValuePolicy.Unicodes ).NotEmpty.HasText + UnitTestValuePolicy.None
[ 'word', 'multiple words', 'A complete sentence,', 'Multiple sentences. Separated with punctuation.', 'String\tcontaining a tab', 'Multiline\nstring', u'word', u'multiple words', u'A complete sentence,', u'Multiple sentences. Separated with punctuation.', u'String\tcontaining a tab', u'Multiline\nstring', ] - Creating
badNumbersandbadTextslists that contain values that should raise exceptions:
badNumbers = UnitTestValuePolicy.All.remove( goodNumbers )
badTexts = UnitTestValuePolicy.All.remove( goodTexts )
A test-method implementation for ExampleFunction could look
soemthing like this:
def testExampleFunction( self ):
goodNumbers = ( UnitTestValuePolicy.Integers +
UnitTestValuePolicy.Longs ).Positive
goodTexts = ( UnitTestValuePolicy.Strings +
UnitTestValuePolicy.Unicodes ).NotEmpty.HasText +
UnitTestValuePolicy.None
badNumbers = UnitTestValuePolicy.All.remove( goodNumbers )
badTexts = UnitTestValuePolicy.All.remove( goodTexts )
# Test all the "good" permutations:
for goodNumber in goodNumbers:
for goodText in goodTexts:
actual = ExampleFunction( goodNumber, goodText )
# TODO: Figure out how to generate an "expected" value
# for comparison... Right now, this will fail!
self.assertEquals( actual, expected,
'ExampleFunction( %d, %s <%s> ) should '
'return "%s" (%s), but returned "%s" (%s)' % (
goodNumber, goodText, type( goodText ).__name__ ),
expected, type( expected ).__name__,
actual, type( actual ).__name__
)
# Test all the "bad" permutations with a single "good" value,
# since all good values have been proven to work correctly by
# now...
goodNumber = goodNumbers[ 0 ]
goodText = goodTexts[ 0 ]
# Test goodNumber/badText possibilities
for badText in badTexts:
try:
result = ExampleFunction( goodNumber, badText )
except ( TypeError, ValueError ):
# Expected exception, so...
pass
except Exception, error:
self.fail( 'ExampleFunction( %d, %s <%s> ) should '
'have raised a TypeError or ValueError, but a %s was '
'encountered instead: %s' % (
goodNumber, badText, type( badText ).__name__,
error.__class__.__name__, error
)
)
# Test badNumber/goodText possibilities
for badNumber in badNumbers:
try:
result = ExampleFunction( badNumber, goodText )
except ( TypeError, ValueError ):
# Expected exception, so...
pass
except Exception, error:
self.fail( 'ExampleFunction( %d, %s <%s> ) should '
'have raised a TypeError or ValueError, but a %s was '
'encountered instead: %s' % (
badNumber, goodText, type( goodText ).__name__,
error.__class__.__name__, error
)
)goodNumber
values times the count of the goodText values is 5 × 13, so
65 good-variant assertions, plus 89
bad-variant assertions (there are 55
Allvalues to start with, once each for numbers and texts, so 110 values, then removing the 21
goodvalues...). Total: 154 assertions. That includes the items in the
'trueish' and 'falseish' lists, though, which
might not be needed in general tests, and is wihout the removal of any
duplications, so that number should go down, maybe substantially. Even if the badvalues lists collapse pretty substantially, I'd still expect somewhere in the neighborhood of 75 test-assertions, though.
Filtering Phrases and Actions
So, the big challenge with this approach is probably to figure out what the
various filtering phraes
(actions) need to be implemented, and implementing
them. In order for this to work the way that I want it to, each additional phrase or
action in a chain has to modify the list of values in some fashion, while being able
to allow continued modification by other phases/actions. For example, the good- and
bad-value items noted above would work out to something like this, every step of the
way:
goodNumbers
( UnitTestValuePolicy.Integers
[ -1, 0, 1, 2 ]+ UnitTestValuePolicy.Longs )
[ -1, 0, 1, 2, -sys.maxint - 1, -1L, 0L, 1L, 2L, sys.maxint ].Positive
[ 1, 2, 1L, 2L, sys.maxint ]goodTexts
( UnitTestValuePolicy.Strings
[ '', ' ', '\t', '\r', '\n', 'word', 'multiple words',
'A complete sentence.',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab', 'Multiline\nstring' ]+ UnitTestValuePolicy.Unicodes )
[ '', ' ', '\t', '\r', '\n', 'word', 'multiple words',
'A complete sentence',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab', 'Multiline\nstring', u'', u' ',
u'\t', u'\r', u'\n', u'word', u'multiple words',
u'A complete sentence.',
u'Multiple sentences. Separated with punctuation.',
u'String\tcontaining a tab', u'Multiline\nstring', ].NotEmpty
[ ' ', '\t', '\r', '\n', 'word', 'multiple words',
'A complete sentence',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab', 'Multiline\nstring', u' ',
u'\t', u'\r', u'\n', u'word', u'multiple words',
u'A complete sentence.',
u'Multiple sentences. Separated with punctuation.',
u'String\tcontaining a tab', u'Multiline\nstring', ].HasText
[ 'word', 'multiple words', 'A complete sentence',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab', 'Multiline\nstring', u'word',
u'multiple words', u'A complete sentence.',
u'Multiple sentences. Separated with punctuation.',
u'String\tcontaining a tab', u'Multiline\nstring', ]+ UnitTestValuePolicy.None
[ 'word', 'multiple words', 'A complete sentence',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab', 'Multiline\nstring', u'word',
u'multiple words', u'A complete sentence.',
u'Multiple sentences. Separated with punctuation.',
u'String\tcontaining a tab', u'Multiline\nstring',
None ]badNumbers
UnitTestValuePolicy.All
[ u'', u' ', u'\t', u'\r', u'\n', u'word',
u'multiple words', u'A complete sentence.',
u'Multiple sentences. Separated with punctuation.',
u'String\tcontaining a tab', u'Multiline\nstring',
None, <object>, -1, 0, 1, 2, True, False, 0.0, 0,
0L, None, '', u'', False, -18446744073709551615L, -1L, 0L,
1L, 2L, 18446744073709551614L, 1.0, 0.5, 1, 1L, 'a',
u'a', <object>, True, '', ' ', '\t', '\r', '\n',
'word', 'multiple words', 'A complete sentence.',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab', 'Multiline\nstring', -1.0,
0.0, 1.0, 2.0 ].remove( goodNumbers )
[ u'', u' ', u'\t', u'\r', u'\n', u'word',
u'multiple words', u'A complete sentence.',
u'Multiple sentences. Separated with punctuation.',
u'String\tcontaining a tab', u'Multiline\nstring', None,
<object>, -1, 0, True, False, 0.0, 0, 0L, None, '', u'',
False, -18446744073709551615L, -1L, 0L, 1.0, 0.5, 1, 1L,
'a', u'a', <object>, True, '', ' ', '\t', '\r',
'\n', 'word', 'multiple words', 'A complete sentence.',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab', 'Multiline\nstring', -1.0, 0.0,
1.0, 2.0 ]badTexts
UnitTestValuePolicy.All
[ u'', u' ', u'\t', u'\r', u'\n', u'word',
u'multiple words', u'A complete sentence.',
u'Multiple sentences. Separated with punctuation.',
u'String\tcontaining a tab', u'Multiline\nstring',
None, <object>, -1, 0, 1, 2, True, False, 0.0, 0,
0L, None, '', u'', False, -18446744073709551615L, -1L, 0L,
1L, 2L, 18446744073709551614L, 1.0, 0.5, 1, 1L, 'a',
u'a', <object>, True, '', ' ', '\t', '\r', '\n',
'word', 'multiple words', 'A complete sentence.',
'Multiple sentences. Separated with punctuation.',
'String\tcontaining a tab', 'Multiline\nstring', -1.0,
0.0, 1.0, 2.0 ].remove( goodTexts )
[ u'', u' ', u'\t', u'\r', u'\n', <object>, -1, 0, 1, 2,
True, False, 0.0, 0, 0L, None, '', u'', False,
-18446744073709551615L, -1L, 0L, 1L, 2L,
18446744073709551614L, 1.0, 0.5, 1, 1L, 'a', u'a',
<object>, True, '', ' ', '\t', '\r', '\n', -1.0, 0.0,
1.0, 2.0 ]Nailing Down the Filtering Actions
In order for this approach to work, the UnitTestValuePolicy
object that's in use needs to be able to return a variety of filtered iterables
(they might be lists, or they might be some other iterable-type, I haven't decided
yet), and each of those needs to be able to apply the same filter processes.
In the interests of maintaining as much flexibility as possible, any
return-instance should be able to apply any of the filtering criteria or
methods — that will allow for a single instance to be filtered down in more
complex fashions. Though I'm not completely sure that I'll need that
degree of flexibility, I'd rather allow it for now and remove it later if necessary
than have to go back and retrofit, for example, a numeric
return-type and
a text
return-type and a boolean
return-type to allow it
later.
I'm pretty sure that most of the simple
types' filtering capabilities
fall into these categories:
.All- Returns all values of all types.
.Numeric- Returns all numeric values of all types. Numeric filters (available here
and in the numeric-type items below) are:
.Even- Returns only even number values
.Odd- Returns only odd number values
.Positive- Returns numbers greater than zero (zero is neither positive nor negative)
.Negative- Returns numbers less than zero
.Zero- Returns numbers that are equal to zero (which would include
0 <int>,0L <long>and0.0 <float> .NonPositive- Returns numbers less than or equal to zero
.NonNegative- Returns numbers greater than or equal to zero
.NonZero- Returns numbers that are not equal to zero
.Integers- Returns only non-
longintegers .Floats- Returns only
floats .Longs- Returns only
longintegers
.Text- Returns all of the
text
-type values, of eitherstrorunicode— and this I may also need to be able to expand to include special text-types that I may want to define later, like email addresses and URIs. Thetype-based
filters would include:.Strings- Returns only those values that are
strtypes .Unicodes- Returns only those values that are
unicodetypes
.NotEmpty- Returns only those values that are not empty
.HasText- Returns only those values that have characters that are not whitespace (so, no values that are nothing but whitespace)
.SingleWords- Returns only those values that have no whitespace (spaces, tabs, line-breaks or carriage-returns)
.SingleLine- Returns only those values that have no line-breaks or carriage-returns
.NoTabs- Returns only those values that have no tabs
.Multiline- Returns only those values that have line-breaks and/or carriage-returns
.TagName- Returns only those values that are legitimate XML tag-names (I know that I'm going to need this by the time I get to parsing and generating markup, so I'm including it now)
- True and False (and True-ish and False-ish) Values
- Returns values that are true (or true-ish) or false (or false-ish). Like
at least a few other languages, Python will allow a non-zero, non-empty
value to evaluate as
Truefor evaluation purposes in anif...elif...elsedecision-structure, so the inclusion oftrue-ish
('trueish') andfalse-ish
('falseish') values is part of the default..Strict- Returns only those values that are actual
booltypes (TrueandFalse) .StrictAndNone- The same as
.Strict, but withNoneadded to the mix .StrictAndNumeric- Returns only those values that are actual
booltypes, plus1and0values as integers, long-integers, and floating-point types. .StrictNumericNone- The same as
.StrictAndNumeric, but withNoneadded to the mix
- Other global actions
-
.remove( values <iterable> )- Removes the items in
valuesfrom the current results.
format conversion
capabilities. One consideration in this respect is that if I do provide that, the results might lose the ability to be meaningfully filtered after this sort of conversion in any of the non-text categories....AsString- Returns all current values as strings
.AsUnicode- Returns all current values as unicode
Chaining Filtering Actions
The other critical part of the structure of UnitTestValuePolicy,
or at least of the collections of test-values it returns, is the ability to chain
different filtering actions together. Having given it some thought, I'm going to
implement those as properties, which will allow the sort of chaining-structure
I noted earlier: ( UnitTestValuePolicy.Integers + UnitTestValuePolicy.Longs
).Positive. That also more or less mandates the underlying Python data-type
that I'm going to be using, or at least returning: a list
— if only to support the concatenation of values with the +
operator. I'll extend that basic list type into a new class
(TestValues) in order to provide the filtering actions. I could also
create my own list-like class and just emulate
the functionality of a list as well, but I don't think I'll
need to go to quite that depth.
The final breakdown, grouped and sorted, of all the filtering properties and methods will look something like this, then:
- Boolean filtering properties
BooleanStrictStrictAndNoneStrictAndNumericStrictNumericNone
- Numeric filtering properties
NumericFloatsIntegersLongsEvenNegativeNonNegativeNonPositiveNonZeroOddPositiveZero
- Text filtering properties
TextStringsUnicodesHasTextMultilineNoTabsNotEmptySingleLineSingleWordsTagName
- Formatting properties (maybe, since they will be
destructive of other filtering)
AsStringAsUnicode
- Other filtering (methods)
remove( values <iterable> )
Alright. I didn't get much into any actual implementation of either of the
UnitTestValuePolicy or TestValues classes, but there's
been a pretty substantial amount of discovery about what they need to be able to
do in order to meet my needs, and this post is long enough that I don't want to dive
into the nuts-and-bolts implementation today — I suspect it'd make for a
ridiculously long post — so that's where I'll pick up next time: Actual
implementations of UnitTestValuePolicy and TestValues.
No comments:
Post a Comment