Unit-testing and Code-coverage in Python [5]

I hadn't actually shown this aspect of the TestValues class yet, but it's an extension of the built-in Python list type:

@describe.InitClass()
class TestValues( list, object ):
    """
Represents a collection of standard test-values, and provides filtering of 
those values."""
    #-----------------------------------#
    # Class attributes (and instance-   #
    # attribute default values)         #
    #-----------------------------------#

As such, it's got a lot of functionality already available that originates with the underlying list class that it extends. The decision to extend TestValues from list was one that I went back and forth on for a while — On one hand, I didn't want to have to implement all of the functionality that I expected TestValues to need from scratch if basing it off of list. Chances are good that I'd have ended up using a list as an internal data-storage for TestValues values anyway, and just wrapping the TestValues class around it.

On the other hand, I wasn't really sure what functionality a list brought to the table that I didn't want. A list has 30-odd methods for various purposes, and I ended up having to take a substantial look at a fair number of them before deciding which ones were still valid and which ones weren't. Here's what that process looked like...

What does a list do already?

Before I could really determine what (if any) properties and methods of the base list type I wanted to override or otherwise tweak, I needed to know what it's got, and what all it's members do. Fortunately, between the documentation about emulating container types and some bits and pieces from the __doc__s of each member, that turned out to be a pretty easy task. The properties and methods that I evaluated are:

__add__

Called to implement the + operator, used when concatenating instances.
Will need to be examined in order to assure that concatenation yields an instance of TestValues rather than a generic list, but that should be the case by default as long as the object being added to is a TestValues instance.

__contains__

Called to perform membership-test operations. Shouldn't need to be altered.

__delitem__

Called to delete individual items or slices from the instance. Shouldn't need to be altered.

__delslice__

Called to delete a slice from the instance. Shouldn't need to be altered.

__eq__

Called to implement the == operator, testing for equality.
Will need to be examined to determine whether comparison will work as-is, and whether it needs to be altered for comparison of instances with equivalent non-instance sequence-objects.

__ge__

Called to implement the >= operator, testing for relative size. Will need to be examined to determine whether comparison will work as-is, and whether it needs to be altered for comparison of instances with equivalent non-instance sequence-objects.

__getitem__

Called to get individual items or slices from the instance. Shouldn't need to be altered.

__getslice__

Called to get a slice of the instance. Shouldn't need to be altered.

__gt__

Called to implement the > operator, testing for relative size. Will need to be examined to determine whether comparison will work as-is, and whether it needs to be altered for comparison of instances with equivalent non-instance sequence-objects.

__iadd__

Called to implement the += operator, used when concatenating instances.
Will need to be examined in order to assure that concatenation yields an instance of TestValues rather than a generic list, but that should be the case by default as long as the object being added to is a TestValues instance.

__imul__

Called to implement the *= operator. Since this normally yields a number of duplicates of the members of the instance in a list, and that's not something that seems useful (all test-values should ideally be distinct), I may want to override this to have it throw an error.

__iter__

Called to return an iterator for the instance. Shouldn't need to be altered.

__le__

Called to implement the <= operator, testing for relative size. Will need to be examined to determine whether comparison will work as-is, and whether it needs to be altered for comparison of instances with equivalent non-instance sequence-objects.

__len__

Called to implement the built-in len function. Shouldn't need to be altered.

__lt__

Called to implement the < operator, testing for relative size. Will need to be examined to determine whether comparison will work as-is, and whether it needs to be altered for comparison of instances with equivalent non-instance sequence-objects.

__mul__

Called to implement the * operator. Since this normally yields a number of duplicates of the members of the instance in a list, and that's not something that seems useful (all test-values should ideally be distinct), I may want to override this to have it throw an error.

__ne__

Called to implement the != and <> operators, testing for inequality.
Will need to be examined to determine whether comparison will work as-is, and whether it needs to be altered for comparison of instances with equivalent non-instance sequence-objects.

__reversed__

Called to return a reversed iterator for the instance. Shouldn't need to be altered.

__rmul__

Called to implement the * operator. Since this normally yields a number of duplicates of the members of the instance in a list, and that's not something that seems useful (all test-values should ideally be distinct), I may want to override this to have it throw an error.

__setitem__

Called to set individual items or slices in the instance. Shouldn't need to be altered.

__setslice__

Called to set a slice in the instance. Shouldn't need to be altered.

append

Called to append a member to an instance. Shouldn't need to be altered.

count

Called to count the number of occurrences of a value in the members of an instance. Shouldn't need to be altered.

extend

Called to extend the instance by appending elements from the supplied iterable. Shouldn't need to be altered.

index

Called to return the index-position of the first occurrence of the value in the members of the instance. Shouldn't need to be altered.

insert

Called to insert a value into the members of the instance at a given index-position. Shouldn't need to be altered.

pop

Called to remove and return a member-item from the instance, at an optional index-position. Shouldn't need to be altered.

remove

Called to remove the first occurrence of a member value from the members of the instance.
This will need to be altered, if only to allow an iterable of values to be passed, and TextValues.remove should remove all occurrences of the specified value(s) from the member collection.

reverse

Called to reverse the sequence of members in the instance. Shouldn't need to be altered.

sort

Called to sort the members of the instance. Shouldn't need to be altered.

Over half of these (18 of the 30 items noted) fell into the shouldn't need to be altered category, and only one item solidly fell into the will need to be altered category (remove). There were three categories of list members that needed alteration or examination at a minimum:

Comparison-operator-related (various "magic methods")

__eq__, __ge__, __gt__, __le__, __lt__ and __ne__

In these cases, all I felt I really needed to do was make sure that they behaved the same with all of the possible comparisons between TestValues and list instances.

Mutation-operator-related (also various "magic methods")

__add__ and __iadd__

These methods, relating to the + and += operators as applied to lists, really just needed verification that when a TestValues instance was being added to, that the result was still a TestValues instance. My expectation was that it would be the case, but I had to make sure. Though I don't expect much use of either during unit-testing, I can't rule them out either.

__imul__, __mul__ and __rmul__

These relate to the * and *= operators as they apply to list instances, though I'm not quite certain when __rmul__ gets called as opposed to __mul__.
Since a TestValues instance is really intended to provide a reasonably-distinct set of values, and these operations effectively duplicate members in the list they are applied to, I had to think through whether I wanted them to override their inherited implementations in order to, perhaps, raise an error of some sort, thus preventing their use.

As a side note: A truly distinct set of values isn't really possible in a TestValues' member-list because str and unicode values containing the same text will evaluate as equal.

Mutation methods

remove

As noted above, I want to alter TestValues.remove to facilitate the removal of multiple values in a single pass.

Amusingly enough, the first two groups are exactly the kinds of items that I'd be looking to unit-test if they cropped up in the implementation of other classes. Because of the way the AddMethodTesting and AddPropertyTesting decorators work, though, they would not be automatically detected as testable members unless they were actively overridden in the derived class. It would be feasible to simply write overriding methods for each of them that call the parent list methods, though, and the decorators would pick up on those as local members that required testing. That feels kind of... wasteful, maybe... so I don't know if I'll take that approach, but even without doing so, just knowing that they should be tested is enough to prompt writing those test-methods.

While testing those, at first I believed that I'd need to create an empty _UnitTestValuePolicy instance — one whose All value had no values — which turned out not to be possible because of the way the default values were being populated:

    def __init__( self, **values ):
        """
Instance initializer"""
        # ...

        # Set _defaults values from **values members, if they are provided
        # - bools
        bools = values.get( 'bools' )
        if bools:
            self._defaults[ 'bools' ] = bools

        # - falseish

        # ...

In order to allow defaults for the various value-categories to be defined as empty, I had to alter _UnitTestValuePolicy.__init__ to specifically check for None values, as opposed to empty lists (which is what I specified while trying to create that empty instance:

    def __init__( self, **values ):
        """
Instance initializer"""
        # - bools
        bools = values.get( 'bools' )
        if bools == None:
            self._defaults[ 'bools' ] = self.__class__._defaults[ 'bools' ]
        else:
            self._defaults[ 'bools' ] = bools

        # - falseish

        # ...

That allowed the creation of the empty instance I thought I needed:

emptySource = _UnitTestValuePolicy( bools=[], 
    falseish=[], floats=[], ints=[], longs=[], none=[], 
    objects=[], strings=[], trueish=[], unicodes=[] )

print emptySource.All
[]

Checking the Comparison-operator-related Members

For all of the comparison-operation checks, I basically just needed two lists and equivalent TestValues instances:

testValuesList1 = [ 1, 2, 3, 4 ]
testValuesInst1 = TestValues( emptySource, testValuesList1 )
testValuesList2 = [ 4, 3, 2, 1 ]
testValuesInst2 = TestValues( emptySource, testValuesList2 )

Once those were created, the basic checks were fairly simple: Perform the comparison against all of the relevant list and TestValues instance-combinations and make sure that what happened in comparing two lists also happened when comparing the list with its equivalent TestValues instance. Since each pair of checks should return the same result, it was an easy matter to just skim downthe list and look for mismatched result-pairs:

print 'Testing __eq__:'
print 'testValuesList1 == testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 == testValuesList1 )
print 'testValuesList1 == testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 == testValuesInst1 )

Testing __eq__:
testValuesList1 == testValuesList1 .... True
testValuesList1 == testValuesInst1 .... True

print 'Testing __ge__:'
print 'testValuesList1 >= testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 >= testValuesList1 )
print 'testValuesList1 >= testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 >= testValuesInst1 )
print 'testValuesList1 >= testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 >= testValuesList2 )
print 'testValuesList1 >= testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 >= testValuesInst2 )
print 'testValuesList2 >= testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 >= testValuesList1 )
print 'testValuesList2 >= testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 >= testValuesInst1 )
print 'testValuesList2 >= testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 >= testValuesList2 )
print 'testValuesList2 >= testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 >= testValuesInst2 )

Testing __ge__:
testValuesList1 >= testValuesList1 .... True
testValuesList1 >= testValuesInst1 .... True
testValuesList1 >= testValuesList2 .... False
testValuesList1 >= testValuesInst2 .... False
testValuesList2 >= testValuesList1 .... True
testValuesList2 >= testValuesInst1 .... True
testValuesList2 >= testValuesList2 .... True
testValuesList2 >= testValuesInst2 .... True

print 'Testing __gt__:'
print 'testValuesList1 > testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 > testValuesList1 )
print 'testValuesList1 > testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 > testValuesInst1 )
print 'testValuesList1 > testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 > testValuesList2 )
print 'testValuesList1 > testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 > testValuesInst2 )
print 'testValuesList2 > testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 > testValuesList1 )
print 'testValuesList2 > testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 > testValuesInst1 )
print 'testValuesList2 > testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 > testValuesList2 )
print 'testValuesList2 > testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 > testValuesInst2 )

Testing __gt__:
testValuesList1 > testValuesList1 ..... False
testValuesList1 > testValuesInst1 ..... False
testValuesList1 > testValuesList2 ..... False
testValuesList1 > testValuesInst2 ..... False
testValuesList2 > testValuesList1 ..... True
testValuesList2 > testValuesInst1 ..... True
testValuesList2 > testValuesList2 ..... False
testValuesList2 > testValuesInst2 ..... False

print 'Testing __le__:'
print 'testValuesList1 <= testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 <= testValuesList1 )
print 'testValuesList1 <= testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 <= testValuesInst1 )
print 'testValuesList1 <= testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 <= testValuesList2 )
print 'testValuesList1 <= testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 <= testValuesInst2 )
print 'testValuesList2 <= testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 <= testValuesList1 )
print 'testValuesList2 <= testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 <= testValuesInst1 )
print 'testValuesList2 <= testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 <= testValuesList2 )
print 'testValuesList2 <= testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 <= testValuesInst2 )

Testing __le__:
testValuesList1 <= testValuesList1 .... True
testValuesList1 <= testValuesInst1 .... True
testValuesList1 <= testValuesList2 .... True
testValuesList1 <= testValuesInst2 .... True
testValuesList2 <= testValuesList1 .... False
testValuesList2 <= testValuesInst1 .... False
testValuesList2 <= testValuesList2 .... True
testValuesList2 <= testValuesInst2 .... True

print 'Testing __lt__:'
print 'testValuesList1 < testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 < testValuesList1 )
print 'testValuesList1 < testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 < testValuesInst1 )
print 'testValuesList1 < testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 < testValuesList2 )
print 'testValuesList1 < testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 < testValuesInst2 )
print 'testValuesList2 < testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 < testValuesList1 )
print 'testValuesList2 < testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 < testValuesInst1 )
print 'testValuesList2 < testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 < testValuesList2 )
print 'testValuesList2 < testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList2 < testValuesInst2 )

Testing __lt__:
testValuesList1 < testValuesList1 ..... False
testValuesList1 < testValuesInst1 ..... False
testValuesList1 < testValuesList2 ..... True
testValuesList1 < testValuesInst2 ..... True
testValuesList2 < testValuesList1 ..... False
testValuesList2 < testValuesInst1 ..... False
testValuesList2 < testValuesList2 ..... False
testValuesList2 < testValuesInst2 ..... False

print 'Testing __ne__:'
print 'testValuesList1 != testValuesList1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 != testValuesList1 )
print 'testValuesList1 != testValuesInst1 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 != testValuesInst1 )
print 'testValuesList1 != testValuesList2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 != testValuesList2 )
print 'testValuesList1 != testValuesInst2 '.ljust( 39, '.' ) + ' %s' % str(
    testValuesList1 != testValuesInst2 )

Testing __ne__:
testValuesList1 != testValuesList1 .... False
testValuesList1 != testValuesInst1 .... False
testValuesList1 != testValuesList2 .... True
testValuesList1 != testValuesInst2 .... True

All of the comparison-operator-related magic methods checked out just fine, so there's no need for me to override them.

The irony of this manual, brute-force testing in a series of posts about unit-testing does not escape me. At some point, I'll try to work out a good way to actually unit-test these, perhaps, but that would distract from the goal of today's post, so I'll leave that for later...

Checking the Mutation-operator-related Members

The mutation operators actually make changes to the object they are being applied to: __add__ and __iadd__ append members to a list, so they should also add members, in the exact same fashion, to a TestValues instance.

print 'Testing __add__:'
List1 = [ 1, 3 ]
Inst1 = TestValues( emptySource, List1 )
List2 = [ 2, 4 ]
Inst2 = TestValues( emptySource, List2 )
print 'type( List1 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( List1 ).__name__ )
print 'type( Inst1 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( Inst1 ).__name__ )
print 'List1 + List2 '.ljust( 39, '.' ) + ' %s' % str(
    List1 + List2 )
print 'type( List1 + List2 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( List1 + List2 ).__name__ )
print 'Inst1 + Inst2 '.ljust( 39, '.' ) + ' %s' % str(
    Inst1 + Inst2 )
print 'type( Inst1 + Inst2 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( Inst1 + Inst2 ).__name__ )
print 'Inst1 + List2 '.ljust( 39, '.' ) + ' %s' % str(
    Inst1 + List2 )
print 'type( Inst1 + List2 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( Inst1 + List2 ).__name__ )

Testing __add__:
type( List1 ) ......................... list
type( Inst1 ) ......................... TestValues
List1 + List2 ......................... [1, 3, 2, 4]
type( List1 + List2 ) ................. list
Inst1 + Inst2 ......................... [1, 3, 2, 4]
type( Inst1 + Inst2 ) ................. list
Inst1 + List2 ......................... [1, 3, 2, 4]
type( Inst1 + List2 ) ................. list

So, the short story behind the __add__ method is that it apparently always returns a list-instance, whether any of the instances being added are not lists themselves or not. That means that I will need to write a TestValues.__add__ method, in order to return a TestValues instance.

print 'Testing __iadd__:'
List1 = [ 1, 3 ]
Inst1 = TestValues( emptySource, List1 )
List2 = [ 2, 4 ]
Inst2 = TestValues( emptySource, List2 )
print 'type( List1 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( List1 ).__name__ )
print 'type( Inst1 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( Inst1 ).__name__ )
List1 += List2
print 'List1 += List2 '.ljust( 39, '.' ) + ' %s' % str(
    List1 )
print 'type( List1 += List2 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( List1 ).__name__ )
Inst1 += Inst2
print 'Inst1 += Inst2 '.ljust( 39, '.' ) + ' %s' % str(
    Inst1 )
print 'type( Inst1 += Inst2 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( Inst1 ).__name__ )
List2 += Inst2
print 'List2 += Inst2 '.ljust( 39, '.' ) + ' %s' % str(
    List2 )
print 'type( List2 += Inst2 ) '.ljust( 39, '.' ) + ' %s' % str(
    type( List2 ).__name__ )

Testing __iadd__:
type( List1 ) ......................... list
type( Inst1 ) ......................... TestValues
List1 += List2 ........................ [1, 3, 2, 4]
type( List1 += List2 ) ................ list
Inst1 += Inst2 ........................ [1, 3, 2, 4]
type( Inst1 += Inst2 ) ................ TestValues
List2 += Inst2 ........................ [2, 4, 2, 4]
type( List2 += Inst2 ) ................ list

I was expecting __iadd__ to return a TestValues instance if the left item in the applicable code were itself an instance of TestValues. I was not expecting a TestValues instance if the left value was not a TestValues instance. In both cases, my expectations panned out.

That does raise a potential concern, though — In any case where a TestValues instance is part of a += operation, if the result needs to be a TestValues then the TestValues instance must be the left item in the assignment. I don't expect this will be much of a concern, but it does mean that I may want to give some thought to creating some sort of conversion-method for lists in order to cast them as TestValues. It's possible to do so by simply creating a new TestValues and passing the list, but that also requires some awareness of which _UnitTestValuePolicy is applicable, or creating a new one just for the new instance. In a case like that, it might be easier to create an instance-method since the isnstance is already aware of its _UnitTestValuePolicy.

By the time I'd finished checking the first two mutation-operator items, I'd given some thought to the remaining ones (__imul__, __mul__ and __rmul__). I couldn't think of a case where I'd actually want to duplicate the value-list behind a TestValues instance. Ever. So those methods will also be overridden — I planned on raising some Exception, but hadn't decided which one.

Since I had already planned to override remove, there was no checking needed.

The Final TestValues Implementation

With only four of the inherited magic-method overrides to be written, and one override of another non-magic method, the additions are pretty short:

@describe.AttachDocumentation()
@describe.argument( 'values', 
    'the values to add to the members', 
    object )
def __add__( self, values ):
    """
Override of the "+" operator callback for lists. Returns an instance of the 
class, populated with the members of the original instance, and with the 
provided value appended to it."""
    selfValues = list( self )
    result = self.__class__( self.ValueSource, 
        selfValues + values )
    return result

@describe.AttachDocumentation()
@describe.raises( RuntimeError, 
    'if the *= operator is executed against an instance of the class'
)
def __imul__( self, value ):
    """
Override of the "*=" operator callback for lists. Prevents the use of the 
operator on instances of the class."""
    raise RuntimeError( '%s does not support the "*=" operator' % ( 
        self.__class__.__name__ ) )

@describe.AttachDocumentation()
@describe.raises( RuntimeError, 
    'if the * operator is executed against an instance of the class'
)
def __mul__( self, value ):
    """
Override of the "*=" operator callback for lists. Prevents the use of the 
operator on instances of the class."""
    raise RuntimeError( '%s does not support the "*=" operator' % ( 
        self.__class__.__name__ ) )

@describe.AttachDocumentation()
@describe.raises( RuntimeError, 
    'if the * operator is executed against an instance of the class'
)
def __rmul__( self, value ):
    """
Override of the "*=" operator callback for lists. Prevents the use of the 
operator on instances of the class."""
    raise RuntimeError( '%s does not support the "*=" operator' % ( 
        self.__class__.__name__ ) )

@describe.AttachDocumentation()
@describe.argument( 'values', 
    'the value (a single value that is not a list or tuple) or values '
    '(a list or tuple of single values that are not lists or tuples) to '
    'remove from the members of the instance', 
    list, tuple, object )
def remove( self, values ):
    """
Removes all instances of the value(s) supplied from the members of the 
instance."""
    if isinstance( values, ( list, tuple ) ):
        for value in values:
            while self.count( value ) != 0:
                list.remove( self, value )
    else:
        while self.count( values ) != 0:
            list.remove( self, values )

The magic-method overrides are almost brutally simple:

• The implementation of __add__ renders the instance being added to down to a list, then returns an instance of the class populated with the results of + applied to that list plus the values supplied. Simple. It also keeps to the implementation pattern of the list type, by retruning a new instance.
• The __imul__, __mul__ and __rmul__ overrides all raise a RuntimeError (for lack of a better Exception type to use) if they are called, and the error-messaging explains that the instance doesn't support the operation attempted.

The override of remove does break, a bit, from the behavior of list.remove — list.remove will raise errors if the item specified for removal isn't present in the list to begin with. After some consideration, and weighing my desire to be able to remove all instances of a given value from a TestValues instance, I discarded that behavior, so a TestValues.remove shouldn't ever fail because a value being removed doesn't exist. There may be other implications of this that I haven't thought of yet, but for now, I'm happy with this implementation: It allows the removal of all values specified, and accepts a list or tuple of values to be removed, as well as single values.

That, then, wraps up the unit_testing module as far as implementation is concerned. The next post, and the last one dealing with unit-testing for the time being, will focus on actually unit-testing the serialization module — finally.

unit_testing.py

47.2kB

Unit-testing and Code-coverage in Python [4]

Long Post

There's a fair length of code in today's post, though it's mostly pretty simple stuff.

Implementation time!

With a set of requirements for the UnitTestValuePolicy and TestValues classes (mostly?) solidified, there's not much more to discuss before diving in to the implementation-details. I expect this post will be code-heavy, and possibly quite long. If it gets too long, I'll spend my next post in actually writing a unit-test module for the serialization module — which is how this all got started.

The filtering-action properties will make frequent use of Python's list comprehensions, rather than the filter() function. In my experience, list comprehensions tend to be faster in most cases, and I've found, after making extensive use of them over the past several years, that they are often easier for me to write and maintain, rather than generating a separate free-standing function to perform evaluations of filtering criteria (which would also then live somewhere else in the code), while still allowing a certain degree of complexity that is often difficult to create with a lambda expression.

By way of example, consider this code:

import time

source = range( -1000000, 2000001 )

def isEven( num ):
    if num % 2 == 0:
        return True
    return False

for trialNumber in range( 1,4 ):
    filterFuncStart = time.time()
    results = filter( isEven, source )
    filterFuncRun = time.time() - filterFuncStart

    isEven2 = lambda n: n % 2 == 0
    filterLambdaStart = time.time()
    results = filter( isEven2, source )
    filterLambdaRun = time.time() - filterLambdaStart

    listCompStart = time.time()
    results = [ n for n in source if n % 2 == 0 ]
    listCompRun = time.time() - listCompStart

    print 'trial-run #%d' % ( trialNumber )
    print 'filter w/ function ............. %0.3f sec.' % ( filterFuncRun )
    print 'filter w/ lambda ............... %0.3f sec.' % ( filterLambdaRun )
    print 'list comp. ..................... %0.3f sec.' % ( listCompRun )
    print 'list-comp ÷ filter w/ function ... %0.2f%%' % ( 
        float( int( listCompRun *10000 / filterFuncRun ) / 100.0 ) )
    print 'list-comp ÷ filter w/ lambda ..... %0.2f%%' % ( 
        float( int( listCompRun *10000 / filterLambdaRun ) / 100.0 ) )
    print

When this is run on my main development laptop, the results are pretty close to this:

trial-run #1
filter w/ function ................ 0.790 sec.
filter w/ lambda .................. 0.652 sec.
list comp. ........................ 0.565 sec.
list-comp ÷ filter w/ function ... 71.53%
list-comp ÷ filter w/ lambda ..... 86.73%

trial-run #2
filter w/ function ................ 0.729 sec.
filter w/ lambda .................. 0.616 sec.
list comp. ........................ 0.538 sec.
list-comp ÷ filter w/ function ... 73.79%
list-comp ÷ filter w/ lambda ..... 87.39%

trial-run #3
filter w/ function ................ 0.725 sec.
filter w/ lambda .................. 0.625 sec.
list comp. ........................ 0.551 sec.
list-comp ÷ filter w/ function ... 76.04%
list-comp ÷ filter w/ lambda ..... 88.16%

If you're interested in seeing what the timing-difference looks like on your own machine, here's the script-file:

filter-vs-comp.py

1.1kB

The list-comprehension approach clocks in at somewhere between 70 and 75% of the run-time for an equivalent filter() call with a dedicated function, and 85 to 90% of the run-time of a lambda-based filter() equivalent.

The _UnitTestValuePolicy Class

As a class, _UnitTestValuePolicy isn't too complex. It's really just a collection of lists of test-values, organized internally into groups by (roughly) formal value-types or by purposes served by the members of those lists of values as applied to unit-testing test-methods. There are a fair few values in the defaults of an instance of the object, so the code perhaps looks long, but it's still pretty simple:

@describe.InitClass()
class _UnitTestValuePolicy( object ):
    """
Represents a collection of standard unit-testing test-method values to be 
tested."""
    #-----------------------------------#
    # Class attributes (and instance-   #
    # attribute default values)         #
    #-----------------------------------#

    _genericObject = object()
    _defaults = {
        'bools':[ True, False ],
        'falseish':[ 0.0, 0, 0L, None, '', u'', False ],
        'floats':[ -1.0, 0.0, 1.0, 2.0 ],
        'ints':[ -1, 0, 1, 2 ],
        'longs':[ -sys.maxint *2 - 1, -1L, 0L, 1L, 2L, sys.maxint *2 ],
        'none':[ None ],
        'objects':[ _genericObject ],
        'strings':[
            '',
            ' ',
            '\t',
            '\r',
            '\n',
            'word',
            'multiple words',
            'A complete sentence,',
            'Multiple sentences. Separated with punctuation.',
            'String\tcontaining a tab',
            'Multiline\nstring',
        ],
        'trueish':[ 1.0, 0.5, 1, 1L, 'a', u'a', _genericObject, True ],
        'unicodes':[
            u'',
            u' ',
            u'\t',
            u'\r',
            u'\n',
            u'word',
            u'multiple words',
            u'A complete sentence,',
            u'Multiple sentences. Separated with punctuation.',
            u'String\tcontaining a tab',
            u'Multiline\nstring',
        ],
    }

    #-----------------------------------#
    # Instance property-getter methods  #
    #-----------------------------------#

    def _GetAll( self ):
        try:
            return self._all
        except AttributeError:
            self._all = []
            for key in self._defaults:
                self._all += self._defaults[ key ]
            return TestValues( self )

    #-----------------------------------#
    # Instance Properties               #
    #-----------------------------------#

    All = describe.makeProperty( _GetAll, None, None, 
        'the complete collection of all test-values', 
        list
    )

    #-----------------------------------#
    # Instance Initializer              #
    #-----------------------------------#
    @describe.AttachDocumentation()
    @describe.keywordargs( 'The collection of values to be populated '
        'in the instance and made available for unit-testing "good" and '
        '"bad" values' )
    @describe.keyword( 'bools', 
        'the values to store as boolean test-values in the instance',
        bool,
        default=_defaults[ 'bools' ]
    )
    @describe.keyword( 'falseish', 
        'the values to store as "false-ish" test-values in '
        'the instance',
        object,
        default=_defaults[ 'falseish' ]
    )
    @describe.keyword( 'floats', 
        'the values to store as floating-point-number test-values in '
        'the instance',
        float,
        default=_defaults[ 'floats' ]
    )
    @describe.keyword( 'ints', 
        'the values to store as integer test-values in the instance',
        int,
        default=_defaults[ 'ints' ]
    )
    @describe.keyword( 'longs', 
        'the values to store as long-integer test-values in the instance '
        '-- by default includes the minimum negative number value '
        'available to the system, and the maximum positive value '
        'available to the system',
        long,
        default=_defaults[ 'longs' ]
    )
    @describe.keyword( 'none', 
        'the values to store as "null" test-values in the instance',
        type( None ),
        default=_defaults[ 'none' ]
    )
    @describe.keyword( 'objects', 
        'the values to store as generic object test-values in the instance',
        object,
        default=_defaults[ 'objects' ]
    )
    @describe.keyword( 'strings', 
        'the values to store as string test-values in the instance',
        str,
        default=_defaults[ 'strings' ]
    )
    @describe.keyword( 'trueish', 
        'the values to store as "true-ish" test-values in '
        'the instance',
        object,
        default=_defaults[ 'trueish' ]
    )
    @describe.keyword( 'unicodes', 
        'the values to store as unicode test-values in the instance',
        str,
        default=_defaults[ 'unicodes' ]
    )
    def __init__( self, **values ):
        """
Instance initializer"""
        # Call parent initializers, if applicable.
        # Set default instance property-values with _Del... methods as needed.
        # Set instance property values from arguments if applicable.
        # Set _defaults values from **values members, if they are provided
        # - bools
        bools = values.get( 'bools' )
        if bools:
            self._defaults[ 'bools' ] = bools

        # - falseish
        falseish = values.get( 'falseish' )
        if falseish:
            self._defaults[ 'falseish' ] = falseish

        # - floats
        floats = values.get( 'floats' )
        if floats:
            self._defaults[ 'floats' ] = floats

        # - ints
        ints = values.get( 'ints' )
        if ints:
            self._defaults[ 'ints' ] = ints

        # - longs
        longs = values.get( 'longs' )
        if longs:
            self._defaults[ 'longs' ] = longs

        # - none
        none = values.get( 'none' )
        if none:
            self._defaults[ 'none' ] = none

        # - objects
        objects = values.get( 'objects' )
        if objects:
            self._defaults[ 'objects' ] = objects

        # - strings
        strings = values.get( 'strings' )
        if strings:
            self._defaults[ 'strings' ] = strings

        # - trueish
        trueish = values.get( 'trueish' )
        if trueish:
            self._defaults[ 'trueish' ] = trueish

        # - unicodes
        unicodes = values.get( 'unicodes' )
        if unicodes:
            self._defaults[ 'unicodes' ] = unicodes

        # Other set-up

The actual _UnitTestValuePolicy class is not explicitly added to the __all__ list of the unit_testing module. Instead, a default instance, UnitTestValuePolicy is created by the module that uses the default values of the class for its test-values. A non-default instance could still be created, it would just require an explicit import of the _UnitTestValuePolicy class, and creation of a new instance with whatever test-values need to be overridden from the default. For example:

from unit_testing import _UnitTestValuePolicy

myTestingValues = {
    # Populate this accordingly, using the keywords expected
    }
myTestValues = _UnitTestValuePolicy( **myTestingValues )

myOtherTestValues = _UnitTestValuePolicy( 
    ints=[ -12, -6, -2, -1, 
        0, 1, 2, 6, 12 ],
    longs=[ -12L, -6L, -2L, -1L, 
        0L, 1L, 2L, 6L, 12L ],
    floats=[ -12.0, -6.0, -2.0, -1.0, 
        0.0, 1.0, 2.0, 6.0, 12.0 ],
)

The intention is to allow both a standard, common set of test-values with minimal required set-up, that can be applied to the majority of test-method values, and the ability to customize some or all of the standard values for specific test-method needs and implementations, without having to do too much customization of test-method structures and logic. It's inevitable that some customization of values will be needed, especially as the structures being tested start having more properties and methods that utilize collections of values or custom classes. But with less customization needed for the simple values, that can be put off for a while.

The other advantage to having a single, standard collection of test-values available and in use as widely as possible is that if (when) a need for a new test-value arises, it can simply be added to the master collection of values, without having to alter each and every test-method. The larger the body of code being tested, the greater an impact this will have from a time-saving standpoint.

_UnitTestValuePolicy has a single property, All, that will gather up and return all of the test-values available to the instance, collected into a TestValues instance.

The default UnitTestValuePolicy Instance

The All value/TestValues-instance of the default _UnitTestValuePolicy instance is what's actually exposed by the module as its UnitTestValuePolicy constant:

# Define a standard UnitTestValuePolicy constant
UnitTestValuePolicy = _UnitTestValuePolicy().All

__all__.append( 'UnitTestValuePolicy' )

That allows the UnitTestValuePolicy constant to be immediately usable for generating lists of test-values with all of the filtering actions provided by any instance of TestValues.

The TestValues Class

The important aspects of TestValues are the various filtering-action properties. Their set-up as properties is pretty typical of any class that uses the describe.makeProperty process from several weeks ago — I've already listed all of the properties themselves several times, but here's a few of them as they were expressed in the code:

    # -- Boolean properties------------------#
    Boolean = describe.makeProperty( _GetBoolean, None, None, 
        'the current test-values that evaluate to True or False when used '
        'in comparison logic',
        list
    )
    Strict = describe.makeProperty( _GetStrict, None, None, 
        'the current test-values that are True or False',
        list
    )

    # ...

    # -- Numeric properties------------------#
    Numeric = describe.makeProperty( _GetNumeric, None, None, 
        'the current test-values that are numbers',
        list
    )
    Floats = describe.makeProperty( _GetFloats, None, None, 
        'the current test-values that are float-type numbers',
        list
    )
    Integers = describe.makeProperty( _GetIntegers, None, None, 
        'the current test-values that are int-type numbers',
        list
    )
    Longs = describe.makeProperty( _GetLongs, None, None, 
        'the current test-values that are long-type numbers',
        list
    )
    Even = describe.makeProperty( _GetEven, None, None, 
        'the current test-values that are even numbers (int and '
        'long-int types only)',
        list
    )
    Odd = describe.makeProperty( _GetOdd, None, None, 
        'the current test-values that are odd numbers (int and '
        'long-int types only)',
        list
    )

    # ...

    # -- Text properties---------------------#
    Text = describe.makeProperty( _GetText, None, None, 
        'the current test-values that are str- or unicode-type text-values',
        list
    )
    Strings = describe.makeProperty( _GetStrings, None, None, 
        'the current test-values that are str-type text-values',
        list
    )

    # ...

The real magic of these properties, if there is any, is all contained in their getter-methods...

There are a couple of properties that aren't directly related to filtering of test-values: All and ValueSource. The ValueSource property is a reference to the _UnitTestValuePolicy instance that contains the complete collection of all available test-values, allowing the individual instances to be able to look at that object and its _defaults items if necessary. The All property, then, allows the instances to acquire the All value from the original _UnitTestValuePolicy instance.

@describe.AttachDocumentation()
def _GetAll( self ):
    """
Gets the complete collection of all available test-values"""
    return self._valueSource.All

@describe.AttachDocumentation()
def _GetValueSource( self ):
    """
Gets the source of all available standard test-values"""
    return self._valueSource

The various Boolean filter-properties are pretty straightforward:

# -- Boolean property-getters------------#
@describe.AttachDocumentation()
def _GetBoolean( self ):
    """
Gets the current test-values that evaluate to True or False when used 
in comparison logic"""
    checkValues = ( self.ValueSource._defaults[ 'bools' ] + 
        self.ValueSource._defaults[ 'trueish' ] + 
        self.ValueSource._defaults[ 'falseish' ]
    )
    newValues = [ v for v in self if v in checkValues ]
    self._checkValues( newValues, 'Boolean' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetStrict( self ):
    """
Gets the current test-values that are True or False"""
    newValues = [ v for v in self if v in ( True, False ) ]
    self._checkValues( newValues, 'Strict' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetStrictAndNone( self ):
    """
Gets the current test-values that are True, False, or None"""
    newValues = [ v for v in self if v in ( True, False, None ) ]
    self._checkValues( newValues, 'StrictAndNone' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetStrictAndNumeric( self ):
    """
Gets the current test-values that are True, False, or any numeric 
value that is equivalent to True or False"""
    newValues = [
        v for v in self 
        if v in ( True, False, 1, 0, 1L, 0L, 1.0, 0.0 )
    ]
    self._checkValues( newValues, 'StrictAndNumeric' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetStrictNumericNone( self ):
    """
Gets the current test-values that are True, False, None, or any numeric 
value that is equivalent to True or False"""
    newValues = [
        v for v in self 
        if v in ( True, False, None, 1, 0, 1L, 0L, 1.0, 0.0 )
    ]
    self._checkValues( newValues, 'StrictNumericNone' )
    return self.__class__( self.ValueSource, newValues )

As noted earlier, these make extensive use of list comprehensions, so as long as those are understood, there's nothing too spectacular about the actual generation of values being returned. All of these getter-methods make use of a protected helper-method to check that the results being returned are going to actually be useful, though:

@describe.AttachDocumentation()
@describe.argument( 'newValues', 
    'the sequence of values to test', 
    list )
@describe.argument( 'name', 
    'the name of the property being tested', 
    str, unicode )
@describe.raises( TypeError,
    'if newValues is not a list'
)
@describe.raises( ValueError,
    'if newValues is an empty list'
)
def _checkValues( self, newValues, name ):
    if not isinstance( newValues, list ):
        raise TypeError( '%s.%s yielded a non-list value' % ( 
            self.__class__.__name__, name ) )
    if len( newValues ) == 0:
        raise ValueError( '%s.%s yielded an empty list' % ( 
            self.__class__.__name__, name ) )

The thought here is that any filtering-process that yields an empty list of test-values is going to be invalid in the context of writing a unit-test. As an example, consider what would happen if a list were generated by using UnitTestValuePolicy.Even.Odd — There are, by definition, no numbers that are both even and odd. Trying to write a test-case that relied on that is... nonsensical at best, really. The same concern would arise from mixing filtering of different types, say like UnitTestValuePolicy.Strings.Negative.

The boolean test-value filter-properties are very value-oriented — boolean values are simple, though, so that's perhaps no great surprise. There's not a whole lot of variation between True and False (and True-ish and False-ish) values, so doing direct filtering based entirely on whether values being filtered are or are not members of fixed, magic value-sets makes sense.

The numeric filter-getters are a bit more process-oriented than their boolean brethren. Numbers and numeric values are more easily processed by examining their mathematical properties, and that makes the list-comprehensions that generate the final filtered output both easy to implement and flexible enough to handle any values that might get added in on the fly, or as overrides to the defaults in the parent _UnitTestValuePolicy instance.

# -- Numeric property-getters------------#
@describe.AttachDocumentation()
def _GetNumeric( self ):
    """
Gets the current test-values that are numeric values (float, int or long types)"""
    newValues = [
        v for v in self 
        if type( v ) in ( int, float, long )
    ]
    self._checkValues( newValues, 'Numeric' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
@describe.todo( 'Document _GetFloats' )
def _GetFloats( self ):
    """
Gets the current test-values that are float-type numeric values"""
    newValues = [
        v for v in self 
        if type( v ) == float
    ]
    self._checkValues( newValues, 'Floats' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetIntegers( self ):
    """
Gets the current test-values that are int-type numeric values"""
    newValues = [
        v for v in self 
        if type( v ) == int
    ]
    self._checkValues( newValues, 'Integers' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetLongs( self ):
    """
Gets the current test-values that are long-type numeric values"""
    newValues = [
        v for v in self 
        if type( v ) == long
    ]
    self._checkValues( newValues, 'Longs' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetEven( self ):
    """
Gets the current test-values that are even numbers (int and long-int types only)"""
    newValues = [
        v for v in self 
        if type( v ) in ( int, long ) 
        and v % 2 == 0
    ]
    self._checkValues( newValues, 'Even' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetNegative( self ):
    """
Gets the current test-values that are negative numeric values"""
    newValues = [
        v for v in self 
        if v < 0
    ]
    self._checkValues( newValues, 'Negative' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetNonNegative( self ):
    """
Gets the current test-values that are non-negative numeric values"""
    newValues = [
        v for v in self 
        if v >= 0
    ]
    self._checkValues( newValues, 'NonNegative' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetNonPositive( self ):
    """
Gets the current test-values that are non-positive numeric values"""
    newValues = [
        v for v in self 
        if v <= 0
    ]
    self._checkValues( newValues, 'NonPositive' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetNonZero( self ):
    """
Gets the current test-values that are non-zero numeric values"""
    newValues = [
        v for v in self 
        if v != 0
    ]
    self._checkValues( newValues, 'NonZero' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetOdd( self ):
    """
Gets the current test-values that are odd numbers (int and long-int types only)"""
    newValues = [
        v for v in self 
        if type( v ) in ( int, long ) 
        and v % 2 == 1
    ]
    self._checkValues( newValues, 'Odd' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetPositive( self ):
    """
Gets the current test-values that are negative numeric values"""
    newValues = [
        v for v in self 
        if v > 0
    ]
    self._checkValues( newValues, 'Positive' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
@describe.todo( 'Document _GetZero' )
def _GetZero( self ):
    """
Gets the current test-values that are numeric values equal to zero"""
    newValues = [
        v for v in self 
        if v == 0
    ]
    self._checkValues( newValues, 'Zero' )
    return self.__class__( self.ValueSource, newValues )

A couple of mathematical factoids to bear in mind:

• Zero is an even number — If divided by two, there is no remainder;
• Zero is neither positive nor negative;

The various text-type filters have a lost of qualitative testing involved once the basic type-filters are taken care of. Those type-based filters, though are pretty much wht you might expect given the examples of the Floats, Integers and Longs propeties for numeric values:

# -- Text property-getters---------------#
    @describe.AttachDocumentation()
    def _GetText( self ):
        """
Gets the current test-values that are str- or unicode-type text-values"""
        newValues = [
            v for v in self 
            if type( v ) in ( str, unicode )
        ]
        self._checkValues( newValues, 'Text' )
        return self.__class__( self.ValueSource, newValues )

    @describe.AttachDocumentation()
    def _GetStrings( self ):
        """
Gets the current test-values that are str-type text-values"""
        newValues = [
            v for v in self 
            if type( v ) == str
        ]
        self._checkValues( newValues, 'Strings' )
        return self.__class__( self.ValueSource, newValues )

    @describe.AttachDocumentation()
    def _GetUnicodes( self ):
        """
Gets the current test-values that are unicode-type text-values"""
        newValues = [
            v for v in self 
            if type( v ) == unicode
        ]
        self._checkValues( newValues, 'Unicodes' )
        return self.__class__( self.ValueSource, newValues )

Empty- and non-empty text-values are pretty easy:

@describe.AttachDocumentation()
def _GetEmpty( self ):
    """
Gets the current test-values that are empty str- or unicode-values"""
    newValues = [
        v for v in self 
        if v == ''
    ]
    self._checkValues( newValues, 'Empty' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetNotEmpty( self ):
    """
Gets the current test-values that are non-empty str- or unicode-values"""
    newValues = [
        v for v in self 
        if v != ''
    ]
    self._checkValues( newValues, 'NotEmpty' )
    return self.__class__( self.ValueSource, newValues )

Single- and multi-line text-values are also pretty straightforward, though they rely on an aspect of the split method of str and unicode values that may not be obvious:

@describe.AttachDocumentation()
def _GetMultiline( self ):
    """
Gets the current test-values that are str- or unicode-type values that have at 
least one line-break or carriage-return in them"""
    newValues = [
        v for v in self 
        if type( v ) in  (str, unicode )
        and (
            len( v.split( '\n' ) ) > 1
            or
            len( v.split( '\r' ) ) > 1
        )
    ]
    self._checkValues( newValues, 'Multiline' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetSingleLine( self ):
    """
Gets the current test-values that are str- or unicode-type values that have 
no line-breaks or carriage-returns in them"""
    newValues = [
        v for v in self 
        if type( v ) in ( str, unicode )
        and len( v.split( '\n' ) ) == 1
        and len( v.split( '\r' ) ) == 1
        )
    ]
    self._checkValues( newValues, 'SingleLine' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetNoTabs( self ):
    """
Gets the current test-values that are CRITERIA"""
    newValues = [
        v for v in self 
        if type( v ) in  (str, unicode )
        and len( v.split( '\t' ) ) == 1
    ]
    self._checkValues( newValues, 'NoTabs' )
    return self.__class__( self.ValueSource, newValues )

@describe.AttachDocumentation()
def _GetSingleWords( self ):
    """
Gets the current test-values that have no spaces in them"""
    newValues = [
        v for v in self 
        if type( v ) in  (str, unicode )
        and len( v.split( ' ' ) ) == 1
    ]
    self._checkValues( newValues, 'SingleWords' )
    return self.__class__( self.ValueSource, newValues )

The same mechanism used for determining whether a text-value has line-breaking characters can also be used to determine if that value has tabs ('\t'), or spaces so I also included the _GetNoTabs and _GetSingleWords filter-getter methods, since they use the same basic mechanism...

The perhaps-not-obvious aspect of split() is that when a text-value is split on a character that does not exist in the original text, the result is still a list, with one member containing the original string. That is:

print 'This is a single-line string'.split( '\n' )
['This is a single-line string']

print 'This is a \nmulti-line string'.split( '\n' )
['This is a ', 'multi-line string']

The inclusion of both '\n' and '\r' is because I expect that I'll need to be able to check against values that have either or both new-lines and carriage-returns in them, particularly once I start delving into HTTP request-response functionality, where the presence of both in a response has been part of the protocol-standards for quite some time.

At present, that leaves the HasText and TagName filter-properties unimplemented:

@describe.AttachDocumentation()
@describe.todo( 'Document _GetHasText' )
@describe.todo( 'Implement _GetHasText' )
def _GetHasText( self ):
    """
Gets the current test-values that are CRITERIA"""
    # TODO: Implement me
    raise NotImplementedError( '%s.HasText has not been implemented '
        'yet' % ( self.__class__.__name__ ) )

@describe.AttachDocumentation()
@describe.todo( 'Document _GetTagName' )
@describe.todo( 'Implement _GetTagName' )
def _GetTagName( self ):
    """
Gets the current test-values that are CRITERIA"""
    # TODO: Implement me
    raise NotImplementedError( '%s.TagName has not been implemented '
        'yet' % ( self.__class__.__name__ ) )

In the time that it's taken to work out all of the other filter-getters, I've started to question whether _GetHasText (and its corresponding HasText property) are necessary. _GetTagName and TagName I'm going to leave for later (once I start in on markup handling), and I expect I'll want/need a _GetAttributeName/AttributeName filter-getter as well at that point, but I don't see a need for them now.

That covers most of the functionality of both of these classes, though, so even with the length of the post as it stands here, I'm pretty happy. There is one aspect to the TestValues object that I haven't covered, though — since it's an extension of the built-in list type, I'll need to take a look at what needs to be implemented so that it will still behave like a list while still being a TestValues.

Unit-testing and Code-coverage in Python [3]

Having determined what needs to be unit-tested, and generated mechanisms for making sure that the coverage of those code-entities is, itself, tested, it's time to turn some attention to how the testing should be managed. In other words, answering:

What needs to happen in order to test those elements to my satisfaction?

What is Unit-Testing Supposed to Do, Anyway?

Ultimately, the goal of a unit-test run could probably be well described as assuring that all of the possible paths through the code being tested have been executed and behaved in an expected fashion. Consider this example function:

def ExampleFunction( numberValue, textValue=None ):
    # Type- and value-check the arguments
    if type( numberValue ) not in ( int, long ):
        raise TypeError( 'numberValue must be an integer or '
            'long-integer value' )
    if textValue != None and type( textValue ) not in (str, unicode ):
        raise TypeError( 'textValue must be a string or '
            'unicode value, or None' )
    if numberValue < 1:
        raise ValueError( 'numberValue must be positive' )
    if textValue.strip() == '':
        raise ValueError( 'textValue cannot be empty or only '
            'white-space characters' )
    # Do the actual stuff...
    if textValue:
        if numberValue % 2:
            # Do odd-numbered cases
            chars = [ c for c in textValue ]
            chars.reverse()
            textValue = ''.join( chars )
        return textValue

This is a fairly simple function — two arguments in, a handful of acceptable types for each argument, some basic restrictions on argument values, and two possible return-value processes.

In order to assure that all the paths through the code of the function are executed, what values would have to be passed for the numberValue and textValue arguments? That is one of the key questions that should be asked for any unit-test process. The other would be some variation of what assertions need to be made against the returned value for each variation?

To answer the argument-values-passed question, I usually think in terms of good and bad values. For this function, those are pretty simple:

Argument	Value-groups
	Good values	Bad values
numberValue	Any int > 0; Any long > 0; Should test both even and odd values	Any int < 1; Any long < 1; Any value of any type other than int or long (floats, str or unicode values, instances of object, etc.)
textValue	None (the default value); Any non-empty str value; Any non-empty unicode value;	'', u'' (empty text values); ' ', u' ', '\n', u'\n', '\t', u'\t' (only-white-space text-values); Any value of any type other than str or unicode

Realistically, it's not always practical to test every legitimate value, though. There are, for example, an infinite number of integer and long-integer values that could be tested, and even if not all of those can be represented by a computer and the set of values were limited to the 2⁶⁴-1 values that a 64-bit machine could represent, there's not much point to testing with all of them — 1,023 and 1,024 will behave just like 1 and 2 would. I'm not even going to try to calculate the number of possible legitimate text-values...

The point of this is that it's perfectly legitimate to test with a representative subset of all the possible values for a given type.

And, in the interests of consistency (and not having to create the same lists of good and bad values from scratch over and over again) I'd like to have them available as some sort of relatively constant set of values, defined once, that could be easily filtered down to the relevant values as needed for any given good or bad value-set.

That's my goal for today's post. Call it a UnitTestValuePolicy.

Defining the UnitTestValuePolicy object

The strategy I'm going to pursue is to provide a baseline, default instance of the class, as a constant named UnitTestValuePolicy. That instance will be populated with the default unit-test values of all of the simple types that I expect test-methods to use, and will provide a number of filtering members (I'm thinking properties initially, if possible) that can be chained together to yield a final collection of values for any test-type that I can think of at present. The defaults that I'm contemplating at this point are:

_genericObject = object()
{
    'bools':[ True, False ],
    'falseish':[ 0.0, 0, 0L, None, '', u'', False ],
    'floats':[ -1.0, 0.0, 1.0, 2.0 ],
    'ints':[ -1, 0, 1, 2 ],
    'longs':[ 
        -sys.maxint *2 - 1, -1L, 0L, 1L, 2L, sys.maxint *2
        ],
    'none':[ None ],
    'objects':[ _genericObject ],
    'strings':[
        '',
        ' ',
        '\t',
        '\r',
        '\n',
        'word',
        'multiple words',
        'A complete sentence.',
        'Multiple sentences. Separated with punctuation.',
        'String\tcontaining a tab',
        'Multiline\nstring',
    ],
    'trueish':[ 1.0, 0.5, 1, 1L, 'a', u'a', 
        _genericObject, True ],
    'unicodes':[
        u'',
        u' ',
        u'\t',
        u'\r',
        u'\n',
        u'word',
        u'multiple words',
        u'A complete sentence.',
        u'Multiple sentences. Separated with punctuation.',
        u'String\tcontaining a tab',
        u'Multiline\nstring',
    ],
}

In use, applying the object to the example method above would involve

• Getting a list of positive whole numbers:
  goodNumbers = ( UnitTestValuePolicy.Integers + UnitTestValuePolicy.Longs ).Positive
  which would yield the 'ints' list plus the 'longs' list, then filter those down to only the values greater than or equal to zero:
  [ 1, 2, 1L, 2L, sys.maxint ]
• Getting a similar list of all string- and unicode values, and removing any that are empty, or that only contain whitespace, and adding the 'none' list to that result (since None is a valid value for the textValue argument):
  goodTexts = ( UnitTestValuePolicy.Strings + UnitTestValuePolicy.Unicodes ).NotEmpty.HasText + UnitTestValuePolicy.None
[ 'word', 'multiple words', 'A complete sentence,', 'Multiple sentences. Separated with punctuation.', 'String\tcontaining a tab', 'Multiline\nstring', u'word', u'multiple words', u'A complete sentence,', u'Multiple sentences. Separated with punctuation.', u'String\tcontaining a tab', u'Multiline\nstring', ]
• Creating badNumbers and badTexts lists that contain values that should raise exceptions:
  badNumbers = UnitTestValuePolicy.All.remove( goodNumbers )
badTexts = UnitTestValuePolicy.All.remove( goodTexts )
  

A test-method implementation for ExampleFunction could look soemthing like this:

def testExampleFunction( self ):
    goodNumbers = ( UnitTestValuePolicy.Integers + 
        UnitTestValuePolicy.Longs ).Positive
    goodTexts = ( UnitTestValuePolicy.Strings + 
        UnitTestValuePolicy.Unicodes ).NotEmpty.HasText + 
        UnitTestValuePolicy.None
    badNumbers = UnitTestValuePolicy.All.remove( goodNumbers )
    badTexts = UnitTestValuePolicy.All.remove( goodTexts )
    # Test all the "good" permutations:
    for goodNumber in goodNumbers:
        for goodText in goodTexts:
            actual = ExampleFunction( goodNumber, goodText )
            # TODO: Figure out how to generate an "expected" value
            # for comparison... Right now, this will fail!
            self.assertEquals( actual, expected, 
                'ExampleFunction( %d, %s <%s> ) should '
                'return "%s" (%s), but returned "%s" (%s)' % ( 
                    goodNumber, goodText, type( goodText ).__name__ ),
                    expected, type( expected ).__name__,
                    actual, type( actual ).__name__
                )
    # Test all the "bad" permutations with a single "good" value, 
    # since all good values have been proven to work correctly by 
    # now...
    goodNumber = goodNumbers[ 0 ]
    goodText = goodTexts[ 0 ]
    # Test goodNumber/badText possibilities
    for badText in badTexts:
        try:
            result = ExampleFunction( goodNumber, badText )
        except ( TypeError, ValueError ):
            # Expected exception, so...
            pass
        except Exception, error:
            self.fail( 'ExampleFunction( %d, %s <%s> ) should '
                'have raised a TypeError or ValueError, but a %s was '
                'encountered instead: %s' % (
                goodNumber, badText, type( badText ).__name__, 
                error.__class__.__name__, error
                )
            )
    # Test badNumber/goodText possibilities
    for badNumber in badNumbers:
        try:
            result = ExampleFunction( badNumber, goodText )
        except ( TypeError, ValueError ):
            # Expected exception, so...
            pass
        except Exception, error:
            self.fail( 'ExampleFunction( %d, %s <%s> ) should '
                'have raised a TypeError or ValueError, but a %s was '
                'encountered instead: %s' % (
                badNumber, goodText, type( goodText ).__name__, 
                error.__class__.__name__, error
                )
            )

If you do the math, that's maybe a pretty formidable set of assertions that will be made for a function with only two arguments: The count of the goodNumber values times the count of the goodText values is 5 × 13, so 65 good-variant assertions, plus 89 bad-variant assertions (there are 55 All values to start with, once each for numbers and texts, so 110 values, then removing the 21 good values...). Total: 154 assertions. That includes the items in the 'trueish' and 'falseish' lists, though, which might not be needed in general tests, and is wihout the removal of any duplications, so that number should go down, maybe substantially. Even if the bad values lists collapse pretty substantially, I'd still expect somewhere in the neighborhood of 75 test-assertions, though.

Filtering Phrases and Actions

So, the big challenge with this approach is probably to figure out what the various filtering phraes (actions) need to be implemented, and implementing them. In order for this to work the way that I want it to, each additional phrase or action in a chain has to modify the list of values in some fashion, while being able to allow continued modification by other phases/actions. For example, the good- and bad-value items noted above would work out to something like this, every step of the way:

goodNumbers

( UnitTestValuePolicy.Integers

[ -1, 0, 1, 2 ]

+ UnitTestValuePolicy.Longs )

[ -1, 0, 1, 2, -sys.maxint - 1, -1L, 0L, 1L, 2L, sys.maxint ]

.Positive

[ 1, 2, 1L, 2L, sys.maxint ]

goodTexts

( UnitTestValuePolicy.Strings

[ '', ' ', '\t', '\r', '\n', 'word', 'multiple words', 
'A complete sentence.', 
'Multiple sentences. Separated with punctuation.', 
'String\tcontaining a tab', 'Multiline\nstring' ]

+ UnitTestValuePolicy.Unicodes )

[ '', ' ', '\t', '\r', '\n', 'word', 'multiple words', 
'A complete sentence', 
'Multiple sentences. Separated with punctuation.', 
'String\tcontaining a tab', 'Multiline\nstring', u'', u' ', 
u'\t', u'\r', u'\n', u'word', u'multiple words', 
u'A complete sentence.', 
u'Multiple sentences. Separated with punctuation.', 
u'String\tcontaining a tab', u'Multiline\nstring', ]

.NotEmpty

[ ' ', '\t', '\r', '\n', 'word', 'multiple words', 
'A complete sentence', 
'Multiple sentences. Separated with punctuation.', 
'String\tcontaining a tab', 'Multiline\nstring', u' ', 
u'\t', u'\r', u'\n', u'word', u'multiple words', 
u'A complete sentence.', 
u'Multiple sentences. Separated with punctuation.',
 u'String\tcontaining a tab', u'Multiline\nstring', ]

.HasText

[ 'word', 'multiple words', 'A complete sentence', 
'Multiple sentences. Separated with punctuation.', 
'String\tcontaining a tab', 'Multiline\nstring', u'word', 
u'multiple words', u'A complete sentence.', 
u'Multiple sentences. Separated with punctuation.', 
u'String\tcontaining a tab', u'Multiline\nstring', ]

+ UnitTestValuePolicy.None

[ 'word', 'multiple words', 'A complete sentence', 
'Multiple sentences. Separated with punctuation.', 
'String\tcontaining a tab', 'Multiline\nstring', u'word', 
u'multiple words', u'A complete sentence.', 
u'Multiple sentences. Separated with punctuation.', 
u'String\tcontaining a tab', u'Multiline\nstring', 
None ]

badNumbers

UnitTestValuePolicy.All

[ u'', u' ', u'\t', u'\r', u'\n', u'word', 
u'multiple words', u'A complete sentence.', 
u'Multiple sentences. Separated with punctuation.', 
u'String\tcontaining a tab', u'Multiline\nstring', 
None, <object>, -1, 0, 1, 2, True, False, 0.0, 0, 
0L, None, '', u'', False, -18446744073709551615L, -1L, 0L, 
1L, 2L, 18446744073709551614L, 1.0, 0.5, 1, 1L, 'a', 
u'a', <object>, True, '', ' ', '\t', '\r', '\n', 
'word', 'multiple words', 'A complete sentence.', 
'Multiple sentences. Separated with punctuation.', 
'String\tcontaining a tab', 'Multiline\nstring', -1.0, 
0.0, 1.0, 2.0 ]

.remove( goodNumbers )

[ u'', u' ', u'\t', u'\r', u'\n', u'word', 
u'multiple words', u'A complete sentence.', 
u'Multiple sentences. Separated with punctuation.', 
u'String\tcontaining a tab', u'Multiline\nstring', None, 
<object>, -1, 0, True, False, 0.0, 0, 0L, None, '', u'', 
False, -18446744073709551615L, -1L, 0L, 1.0, 0.5, 1, 1L, 
'a', u'a', <object>, True, '', ' ', '\t', '\r', 
'\n', 'word', 'multiple words', 'A complete sentence.', 
'Multiple sentences. Separated with punctuation.', 
'String\tcontaining a tab', 'Multiline\nstring', -1.0, 0.0, 
1.0, 2.0 ]

badTexts

UnitTestValuePolicy.All

[ u'', u' ', u'\t', u'\r', u'\n', u'word', 
u'multiple words', u'A complete sentence.', 
u'Multiple sentences. Separated with punctuation.', 
u'String\tcontaining a tab', u'Multiline\nstring', 
None, <object>, -1, 0, 1, 2, True, False, 0.0, 0, 
0L, None, '', u'', False, -18446744073709551615L, -1L, 0L, 
1L, 2L, 18446744073709551614L, 1.0, 0.5, 1, 1L, 'a', 
u'a', <object>, True, '', ' ', '\t', '\r', '\n', 
'word', 'multiple words', 'A complete sentence.', 
'Multiple sentences. Separated with punctuation.', 
'String\tcontaining a tab', 'Multiline\nstring', -1.0, 
0.0, 1.0, 2.0 ]

.remove( goodTexts )

[ u'', u' ', u'\t', u'\r', u'\n', <object>, -1, 0, 1, 2, 
True, False, 0.0, 0, 0L, None, '', u'', False, 
-18446744073709551615L, -1L, 0L, 1L, 2L, 
18446744073709551614L, 1.0, 0.5, 1, 1L, 'a', u'a', 
<object>, True, '', ' ', '\t', '\r', '\n', -1.0, 0.0, 
1.0, 2.0 ]

Nailing Down the Filtering Actions

In order for this approach to work, the UnitTestValuePolicy object that's in use needs to be able to return a variety of filtered iterables (they might be lists, or they might be some other iterable-type, I haven't decided yet), and each of those needs to be able to apply the same filter processes. In the interests of maintaining as much flexibility as possible, any return-instance should be able to apply any of the filtering criteria or methods — that will allow for a single instance to be filtered down in more complex fashions. Though I'm not completely sure that I'll need that degree of flexibility, I'd rather allow it for now and remove it later if necessary than have to go back and retrofit, for example, a numeric return-type and a text return-type and a boolean return-type to allow it later.

I'm pretty sure that most of the simple types' filtering capabilities fall into these categories:

.All

Returns all values of all types.

.Numeric

Returns all numeric values of all types. Numeric filters (available here and in the numeric-type items below) are:

.Even

Returns only even number values

.Odd

Returns only odd number values

.Positive

Returns numbers greater than zero (zero is neither positive nor negative)

.Negative

Returns numbers less than zero

.Zero

Returns numbers that are equal to zero (which would include 0 <int>, 0L <long> and 0.0 <float>

.NonPositive

Returns numbers less than or equal to zero

.NonNegative

Returns numbers greater than or equal to zero

.NonZero

Returns numbers that are not equal to zero

Type-based numeric filters are:

.Integers

Returns only non-long integers

.Floats

Returns only floats

.Longs

Returns only long integers

.Text

Returns all of the text-type values, of either str or unicode — and this I may also need to be able to expand to include special text-types that I may want to define later, like email addresses and URIs. The type-based filters would include:

.Strings

Returns only those values that are str types

.Unicodes

Returns only those values that are unicode types

Text-values cover a lot of variations that can and will show up in unit-tests, including:

.NotEmpty

Returns only those values that are not empty

.HasText

Returns only those values that have characters that are not whitespace (so, no values that are nothing but whitespace)

.SingleWords

Returns only those values that have no whitespace (spaces, tabs, line-breaks or carriage-returns)

.SingleLine

Returns only those values that have no line-breaks or carriage-returns

.NoTabs

Returns only those values that have no tabs

.Multiline

Returns only those values that have line-breaks and/or carriage-returns

.TagName

Returns only those values that are legitimate XML tag-names (I know that I'm going to need this by the time I get to parsing and generating markup, so I'm including it now)

This list will almost certainly grow, and perhaps grow quickly, as new text-types to be tested surface.

True and False (and True-ish and False-ish) Values

Returns values that are true (or true-ish) or false (or false-ish). Like at least a few other languages, Python will allow a non-zero, non-empty value to evaluate as True for evaluation purposes in an if...elif...else decision-structure, so the inclusion of true-ish ('trueish') and false-ish ('falseish') values is part of the default.

.Strict

Returns only those values that are actual bool types (True and False)

.StrictAndNone

The same as .Strict, but with None added to the mix

.StrictAndNumeric

Returns only those values that are actual bool types, plus 1 and 0 values as integers, long-integers, and floating-point types.

.StrictNumericNone

The same as .StrictAndNumeric, but with None added to the mix

Other global actions

.remove( values <iterable> )

Removes the items in values from the current results.

I'm also going to consider whether I'd want/need some format conversion capabilities. One consideration in this respect is that if I do provide that, the results might lose the ability to be meaningfully filtered after this sort of conversion in any of the non-text categories...

.AsString

Returns all current values as strings

.AsUnicode

Returns all current values as unicode

Chaining Filtering Actions

The other critical part of the structure of UnitTestValuePolicy, or at least of the collections of test-values it returns, is the ability to chain different filtering actions together. Having given it some thought, I'm going to implement those as properties, which will allow the sort of chaining-structure I noted earlier: ( UnitTestValuePolicy.Integers + UnitTestValuePolicy.Longs ).Positive. That also more or less mandates the underlying Python data-type that I'm going to be using, or at least returning: a list — if only to support the concatenation of values with the + operator. I'll extend that basic list type into a new class (TestValues) in order to provide the filtering actions. I could also create my own list-like class and just emulate the functionality of a list as well, but I don't think I'll need to go to quite that depth.

The final breakdown, grouped and sorted, of all the filtering properties and methods will look something like this, then:

• Boolean filtering properties
  • Boolean
  • Strict
  • StrictAndNone
  • StrictAndNumeric
  • StrictNumericNone
• Numeric filtering properties
  • Numeric
  • Floats
  • Integers
  • Longs
  • Even
  • Negative
  • NonNegative
  • NonPositive
  • NonZero
  • Odd
  • Positive
  • Zero
• Text filtering properties
  • Text
  • Strings
  • Unicodes
  • HasText
  • Multiline
  • NoTabs
  • NotEmpty
  • SingleLine
  • SingleWords
  • TagName
• Formatting properties (maybe, since they will be destructive of other filtering)
  • AsString
  • AsUnicode
• Other filtering (methods)
  • remove( values <iterable> )

Alright. I didn't get much into any actual implementation of either of the UnitTestValuePolicy or TestValues classes, but there's been a pretty substantial amount of discovery about what they need to be able to do in order to meet my needs, and this post is long enough that I don't want to dive into the nuts-and-bolts implementation today — I suspect it'd make for a ridiculously long post — so that's where I'll pick up next time: Actual implementations of UnitTestValuePolicy and TestValues.

Unit-testing and Code-coverage in Python [2]

Ideally, if possible, I'd very much like for code-coverage testing to be a trivial thing to put in place in a unit-test module. The best-case scenario I can think of would be to have some common unittest.TestCase-derived class that can just be imported into a given unit-test module, and it would take care of running (perhaps even generating) all the coverage-related testing that I want to have in place.

I'm still using the same copy of the serialization module, with an additional concrete class (JSONSerializableStub) added in to assure that I'm testing an interface, and abstract class, and a concrete class. I've also added a BogusFunction function, because I realized that I also wanted to make sure I was forcing testing of module-level functions as well as classes — though I use them rarely, they should also be thoroughly tested...

Testing for Test-Cases

The first task is to gather up all of that example inspect-based code from the last post, and come up with a test-mechanism to assert that all classes and functions present in the module being tested have a corresponding unittest.TestCase test-case class. Since the module being tested is an unknown quantity as far as the definition of the module-coverage-test class is concerned, I'll need to make sure that it's available to the test-case class. I'll accomplish that by making it a class-attribute. The balance of the code for doing all of the examination and generating the final testCodeCoverage test-method looks like this:

class moduleCoverageTest( unittest.TestCase ):
    """
Unit-test that checks to make sure that all classes in the module being tested 
have corresponding test-case classes in the unit-test module where the derived 
class is defined."""

    #-----------------------------------#
    # Default class constants that      #
    # point to the namespace and module #
    # being tested                      #
    #-----------------------------------#
    _testNamespace = None
    _testModule = None

    @classmethod
    def setUpClass( cls ):
        # Get all the classes available in the module
        cls._moduleClasses = inspect.getmembers( 
            cls._testModule, inspect.isclass )
        # Get all the functions available in the module
        cls._moduleFunctions = inspect.getmembers( 
            cls._testModule, inspect.isfunction )
        # Collect all the *LOCAL* items
        cls._testModuleName = cls._testModule.__name__
        # Find and keep track of all of the test-cases that relate to 
        # classes in the module being tested
        cls._classTests = dict(
            [
                ( 'test%s' % m[ 0 ], m[ 1 ] ) 
                for m in cls._moduleClasses
                if m[ 1 ].__module__ == cls._testModuleName
            ]
        )
        # Ditto for the functions in the module being tested
        cls._functionTests = dict(
            [
                ( 'test%s' % m[ 0 ], m[ 1 ] ) 
                for m in cls._moduleFunctions
                if m[ 1 ].__module__ == cls._testModuleName
            ]
        )
        # The list of required test-case class-names is the aggregated 
        # list of all class- and function-test-case-class names
        cls._requiredTestCases = sorted( 
            cls._classTests.keys() + cls._functionTests.keys()
        )
        # Find and keep track of all of the actual test-case classes in 
        # the module the class resides in
        cls._actualTestCases = dict(
            [ 
                item for item in 
                inspect.getmembers( inspect.getmodule( cls ), 
                  inspect.isclass ) 
                if item[ 1 ].__name__[ 0:4 ] == 'test'
                and issubclass( item[ 1 ], unittest.TestCase )
            ]
        )
        # Calculate the missing test-case-class names, for use by 
        # the testCodeCoverage test-method
        cls._missingTestCases = sorted( 
            set( cls._requiredTestCases ).difference( 
              set( cls._actualTestCases.keys() ) ) )

    def testCodeCoverage(self):
        self.assertEquals( [], self._missingTestCases, 
            'Unit-testing policies require test-cases for all classes and '
            'functions in the %s module, but the following have not been '
            'defined: (%s)' % ( 
                self.__class__._testModule.__name__, 
                ', '.join( self._missingTestCases )
            )
        )

There is a fair amount of pre-processing in the setUpClass method of moduleCoverageTest. setUpClass is called during a unit-test run on each class before any of its test* methods, providing a hook for operations that need to take place before any tests are actually run. In this case, I'm doing a fair amount of inspect-based examination of the module that the class resides in, and the module that the test-module is testing, in order to pre-calculate various data that I expect to use in testing for the presence of test-case classes and (later) property- and method-coverage tests. Those various calculations and aggregations should, ideally, happen only once for any given coverage-test, and need to persist across the life of the test-case class, so the setUpClass method seemed a good place to do all that. It does raise the potential concern that classes derived from moduleCoverageTest could override that method, putting the fidelity of code-coverage testing at risk. Normally, I'd be looking pretty seriously at making the moduleCoverageTest class nominally final, but since subclassing it is kind of key to its purpose, that risk is going to remain unmitigated, at least for now.

With this collected/encapsulated into a common, importable class, it becomes very simple to perform the code-coverage test — All that needs to happen is to import the moduleCoverageTest class, then define a subclass of it, with the namespace and source-module set as class properties of that subclass. A bare-bones test-module for the serialization module starts taking shape as:

import unittest
import serialization

from unit_testing import moduleCoverageTest

#-----------------------------------#
# Set up a local test-suite         #
#-----------------------------------#
LocalSuite = unittest.TestSuite()

#-----------------------------------#
# Import the classes, etc., from    #
# the module/package being tested   #
#-----------------------------------#

#-----------------------------------#
# Test-cases in the module          #
#-----------------------------------#
class testModuleCoverage( moduleCoverageTest ):
    _testNamespace = 'idic'
    _testModule = serialization

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testModuleCoverage
    )
)

#-----------------------------------#
# Run local unit-tests if the       #
# module is called directly         #
#-----------------------------------#

if __name__ == '__main__':
    results = unittest.TestResult()
    LocalSuite.run( results )

    print results
    if results.errors:
        for testCase, error in results.errors:
            print testCase
            print error
    if results.failures:
        for testCase, failure in results.failures:
            print testCase
            print failure

There are a few items omitted from this listing, in order to keep it short and relevant, but they mostly center around making sure that imports of the module being tested, and of all its members are done. The output is crude, but functional enough for now. Running this test-code yields a single failure:

<unittest.result.TestResult run=1 errors=0 failures=1>
testCodeCoverage (__main__.testModuleCoverage)
Traceback (most recent call last):
  File "unit_testing.py", line 144, in testCodeCoverage

AssertionError: Unit-testing policies require test-cases for 
all classes and functions in the serialization module, but 
the following have not been defined: 
  (testBogusFunction, testHasSerializationDict, 
  testIsJSONSerializable, testJSONSerializableStub, 
  testUnsanitizedJSONWarning)

which, at this point, is exactly what I want. It's identified that there are no test-case classes for any of the serialization classes, and caused the test to fail.

Perfect!

If it's working as expected, then all that need happen to allow the test-case coverage test to pass is to create the missing test-case classes, like so:

#-----------------------------------#
# Test-cases in the module          #
#-----------------------------------#
class testModuleCoverage( moduleCoverageTest ):
    _testNamespace = 'idic'
    _testModule = serialization

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testModuleCoverage
    )
)

class testBogusFunction( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testBogusFunction
    )
)

class testHasSerializationDict( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testHasSerializationDict
    )
)

class testIsJSONSerializable( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testIsJSONSerializable
    )
)

class testJSONSerializableStub( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testJSONSerializableStub
    )
)

class testUnsanitizedJSONWarning( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testUnsanitizedJSONWarning
    )
)

With those additional test-cases, even as useless as they currently are, re-running the test-code yields no failures:

<unittest.result.TestResult run=1 errors=0 failures=0>

That part, then, is good to go.

Testing Coverage of Class Members

In similar unit-testing code that I wrote for work, the process that got implemented ended up performing a check for the presence of all test-cases, listing any that were expected but not created across the entire module (just like what's happening in this implementation so far). It then did a similarly-scoped check across all test-cases looking for missing test-methods for properties and methods in the source classes, but because it was searching the entire module, it would only return one failure at a time. While that was workable, and still achieved the amount of code-coverage that I wanted to accomplish, it did get somewhat tedious, having to run the test-suite, find the single reported missing property and/or method test-results, correct that, then re-run again and repeat until there were no more missing test-methods.

I'd consider that to be a viable fall-back position, but I'd much rather have a separate assertion-failure report for each test-case class, and separate assertion-failures for missing property tests and method tests both. Implementing something at that level of granularity is possible, but there are at least two distinct approaches that could be taken, both with their advantages and potential drawbacks.

The first option, to my thinking, would be to subclass the existing unittest.TestCase class, and add in methods for testing whether the class has all of the appropriate test-methods for the properties and methods of the source-class being tested. That's certainly feasible. The drawback to that, as I see it, is that there might well be other reasons to need to subclass unittest.TestCase, and if/when those arise, that would then mean, potentially, subclassing the original property- and method-test-aware subclass of unittest.TestCase. That has the potential of getting unweildy pretty quickly. There's also considerable potential, I think, for introducing unit-testing failures if that sort of scenario were to play out. Nevertheless, it's still a viable option.

Arguably, a better approach would be to have some mechanism that examines the source-classes for properties and methods that should have corresponding test-methods, and coming up with some way to require a test-method that tests the coverage of those test-methods in comparison to the source-class' members. I believe that the ideal approach would be for the existing moduleCoverageTest class to be able to insert property- and method-coverage tests directly into the test-case classes that it's already keeping track of.

Unfortunately, when I tested that approach, although it certainly was possible to add testMethodCoverage and testPropertyCoverage test-methods to the individual test-case classes, those test-methods did not fire — I'm guessing that it's a sequencing issue of some sort, that all of a TestCase-class' test-methods have to be defined in some fashion at the class level, in the code, so that they are members of their respective classes during the interpretation-run of against the code. Even using the same sort of decoration-like approach that I used for overriding the various json-module functions in the serialization module didn't have the desired effect. The test-methods existed, as far as I could tell, but they didn't fire. A cursory examination of the members of unittest.TestCase didn't lead me to believe that there was any way to register or add a new test-method on the fly, though there may well be one that just wasn't apparent to me.

One more possibility occurred to me, and when I tried it out, it apparently worked — another Python decoration approach. Specifically, I found that if I defined decorator functions like so:

def AddPropertyTesting( cls ):
    def testPropertyCoverage( self ):
        self.assertEquals( '%s.testPropertyCoverage called' % cls.__name__, None )
    cls.testPropertyCoverage = testPropertyCoverage
    return cls

def AddMethodTesting( cls ):
    def testMethodCoverage( self ):
        self.assertEquals( '%s.testMethodCoverage called' % cls.__name__, None )
    cls.testMethodCoverage = testMethodCoverage
    return cls

and decorated the test-case classes accordingly:

@AddPropertyTesting
@AddMethodTesting
class testHasSerializationDict( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testHasSerializationDict
    )
)

I at least got some potentially-viable results:

testMethodCoverage (__main__.testHasSerializationDict)
Traceback (most recent call last):
  File "unit_testing.py", line 71, in testMethodCoverage

AssertionError: 
  'testHasSerializationDict.testMethodCoverage called' != None

testPropertyCoverage (__main__.testHasSerializationDict)
Traceback (most recent call last):
  File "unit_testing.py", line 65, in testPropertyCoverage

AssertionError: 
  'testHasSerializationDict.testPropertyCoverage called' != None

These don't look like much, but they at least prove that a decorator-based approach might solve the issue. The one remaining missing piece that I needed was a way to determine if the individual test-case classes had been appropriately decorated, which could be managed as an additional test-method of the moduleCoverageTest class:

def testPropertyCoverage( self ):
    for testCaseClassName in self._classTests:
        # Get the source class, the class being tested
        sourceClass = self._classTests[ testCaseClassName ]
        # Get the corresponding test-case class, locally
        testClass = self._actualTestCases[ testCaseClassName ]
        # Determine if the source class has any properties that 
        # should be tested with the AddPropertyTesting decorator
        sourceProps = [
                pt for pt in inspect.getmembers( sourceClass, 
                    inspect.isdatadescriptor )
                if pt[ 0 ][ 0:2 ] != '__'
            ]
        if sourceProps:
            # If there are properties, then require the decorator
            self.assertNotEqual( 
                testClass.__dict__.get( 'testPropertyCoverage' ),
                None, 
                'Unit-testing policy requires that %s implement a '
                'testPropertyCoverage test-method, since %s has '
                'active properties (%s). Please decorate %s with the '
                '@AddPropertyTesting function.' % ( 
                    testClass.__name__, sourceClass.__name__, 
                    ', '.join( sorted( 
                        [ sp[ 0 ] for sp in sourceProps ]
                    ) ), testClass.__name__
                )
            )

That at least allows the test-process to determine that something needs the property- or method-coverage test-decorators, and cause an assertion-failure if they aren't provided:

<unittest.result.TestResult run=2 errors=0 failures=1>
testPropertyCoverage (__main__.testModuleCoverage)
Traceback (most recent call last):
  File "unit_testing.py", line 185, in testPropertyCoverage

AssertionError: Unit-testing policy requires that 
testJSONSerializableStub implement a testPropertyCoverage 
test-method, since JSONSerializableStub has active properties 
  (FieldName1, FieldName2, PythonNamespace, SanitizedJSON). 
Please decorate testJSONSerializableStub with the 
  @AddPropertyTesting function.

After some tinkering, I managed to get the decoration-based process to work rather well, I think. In use, it will probably feel a little weird for a while, but it appears to be surprisingly robust even this early in the game, and it provides much more granular test-results for missing test-methods than the viable fall-back position mentioned at the start of this post — one test each that checks for source-class property- and method-tests in the test-case class, with a nice, itemized list of what's missing for that test-case.

The Final setUpClass method

Support-data for the AddMethodTesting and AddPropertyTesting methods had to be added to setUpClass, providing the sets of expected test-method names for the methods and properties of the source classes. Picking up where it left off above, these additional items are built like this (after the calculation of cls._missingTestCases:

# Calculate the missing test-case-class names, for use by 
# the testCodeCoverage test-method
cls._missingTestCases = sorted( 
    set( cls._requiredTestCases ).difference( 
        set( cls._actualTestCases.keys() ) ) )

# Calculate the property test-case names for all the 
# module's classes
cls._propertyTestsByClass = {}
for testClass in cls._classTests:
    cls._propertyTestsByClass[ testClass ] = set()
    sourceClass = cls._classTests[ testClass ]
    sourceMRO = list( sourceClass.__mro__ )
    sourceMRO.reverse()
    # Get all the item's properties
    properties = [
        member for member in inspect.getmembers( 
            sourceClass, inspect.isdatadescriptor )
        if member[ 0 ][ 0:2 ] != '__'
    ]
    # Create and populate data-structures that keep track of where 
    # property-members originate from, and what their implementation 
    # looks like. Initially populated with None values:
    propSources = {}
    propImplementations = {}
    for name, value in properties:
        propSources[ name ] = None
        propImplementations[ name ] = None
    for memberName in propSources:
        implementation = sourceClass.__dict__.get( memberName )
        if implementation and propImplementations[ memberName ] != implementation:
            propImplementations[ memberName ] = implementation
            propSources[ memberName ] = sourceClass
    cls._propertyTestsByClass[ testClass ] = set(
        [
            'test%s' % key for key in propSources 
            if propSources[ key ] == sourceClass
        ]
    )

# Calculate the method test-case names for all the module's classes
cls._methodTestsByClass = {}
for testClass in cls._classTests:
    cls._methodTestsByClass[ testClass ] = set()
    sourceClass = cls._classTests[ testClass ]
    sourceMRO = list( sourceClass.__mro__ )
    sourceMRO.reverse()
    # Get all the item's methods
    methods = [
        member for member in inspect.getmembers( 
            sourceClass, inspect.ismethod )
    ] + [
        member for member in inspect.getmembers( 
            sourceClass, inspect.isfunction )
    ]
    # Create and populate data-structures that keep track of where 
    # method-members originate from, and what their implementation 
    # looks like. Initially populated with None values:
    methSources = {}
    methImplementations = {}
    for name, value in methods:
        methSources[ name ] = None
        methImplementations[ name ] = None
    for memberName in methSources:
        implementation = sourceClass.__dict__.get( memberName )
        if implementation and methImplementations[ memberName ] != implementation:
            methImplementations[ memberName ] = implementation
            methSources[ memberName ] = sourceClass
    cls._methodTestsByClass[ testClass ] = set(
        [
            'test%s' % key for key in methSources 
            if methSources[ key ] == sourceClass
        ]
    )

In both cases, the new data is populated for all the source-classes being tested. Each class is retrieved, the Method Resolution Order (MRO) for the class is retrieved, and each superclass of the source-class is checked for each member in the source-class. This allows the expected test-cases for any given class to be built around the members that are implemented within the source class only. For example, the JSONSerializableStub class, though it has all of the inherited members from IsJSONSerializable and so on, it only implements the members that are required (as abstract members from the classes it derives from). The test-methods for IsJSONSerializable will be responsible for testing the members present there, and until or unless JSONSerializableStub overrides a non-abstract member of IsJSONSerializable (taking ownership of it, as it were), there's no requirement to test the members inherited from IsJSONSerializable in JSONSerializableStub.

The Final AddMethodTesting method

The final AddMethodTesting decorator-method relies on the main class that it's bound to storing a fair chunk of information about the module being tested, the members of that module, and the members of those members. All that data is still compiled in the test-case class derived from moduleCoverageTest, during its setUpClass method.

@classmethod
def AddMethodTesting( cls, target ):
    if cls.__name__ == 'moduleCoverageTest':
        raise RuntimeError( 'moduleCoverageTest should be extended '
            'into a local test-case class, not used as one directly.' )
    if not cls._testModule:
        raise AttributeError( '%s does not have a _testModule defined '
            'as a class attribute. Check that the decorator-method is '
            'being called from the extended local test-case class, not '
            'from moduleCoverageTest itself.' % ( cls.__name__ ) )
    try:
        if cls._methodTestsByClass:
            populate = False
        else:
            populate = True
    except AttributeError:
        populate = True
    if populate:
        cls.setUpClass()
    def testMethodCoverage( self ):
        requiredTestMethods = cls._methodTestsByClass[ target.__name__ ]
        activeTestMethods = set(
            [
                m[ 0 ] for m in 
                inspect.getmembers( target, inspect.ismethod )
                if m[ 0 ][ 0:4 ] == 'test'
            ]
        )
        missingMethods = sorted( 
            requiredTestMethods.difference( activeTestMethods )
        )
        self.assertEquals( [], missingMethods, 
            'Unit-testing policy requires test-methods to be created for '
            'all public and protected methods, but %s is missing the '
            'following test-methods: %s' % ( 
                target.__name__, missingMethods
            )
        )
    target.testMethodCoverage = testMethodCoverage
    return target

In order to (hopefully) eliminate any possibility of accidentally using the original moduleCoverageTest decorator-methods, a set of checks is performed first, raising errors if the class is moduleCoverageTest, or if no _testModule class-attribute has been specified. Although it should never be needed, I'm also checking for class-attributes created and populated by setUpClass, and explicilty calling setUpClass if those attributes don't exist.

After that, it's all just getting the test-method names expected/required, removing any test-method names that exist, and checking that the remaining set of test-method names is empty. If it is, then all the required methods exist, and the test will pass. If there are any missing test-method names remaining, then the test will fail, and display the missing test-method names as part of its failure message. All of this happens inside a function defined inside the closure of the wrapping class-method, so all of the class-properties of the moduleCoverageTest-derived class are available.

The Final AddPropertyTesting method

Structurally, this is pretty much identical to the AddMethodTesting method already described. The only substantive differences are which data-structure in the owner class is being used as the check-source, and the messaging:

@classmethod
def AddPropertyTesting( cls, target ):
    if cls.__name__ == 'moduleCoverageTest':
        raise RuntimeError( 'moduleCoverageTest should be extended '
            'into a local test-case class, not used as one directly.' )
    if not cls._testModule:
        raise AttributeError( '%s does not have a _testModule defined '
            'as a class attribute. Check that the decorator-method is '
            'being called from the extended local test-case class, not '
            'from moduleCoverageTest itself.' % ( cls.__name__ ) )
    try:
        if cls._propertyTestsByClass:
            populate = False
        else:
            populate = True
    except AttributeError:
        populate = True
    if populate:
        cls.setUpClass()
    def testPropertyCoverage( self ):
        requiredTestMethods = cls._propertyTestsByClass[ target.__name__ ]
        activeTestMethods = set(
            [
                m[ 0 ] for m in 
                inspect.getmembers( target, inspect.ismethod )
                if m[ 0 ][ 0:4 ] == 'test'
            ]
        )
        missingMethods = sorted( 
            requiredTestMethods.difference( activeTestMethods )
        )
        self.assertEquals( [], missingMethods, 
            'Unit-testing policy requires test-methods to be created for '
            'all public properties, but %s is missing the following test-'
            'methods: %s' % ( target.__name__, missingMethods )
        )
    target.testPropertyCoverage = testPropertyCoverage
    return target

Example serialization Test-Cases

Before I break for the day, let me just dump the results of the following test-case classes, with the applicable decoration in place:

class testSerializationModuleCoverage( moduleCoverageTest ):
    _testNamespace = 'idic'
    _testModule = serialization

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testSerializationModuleCoverage
    )
)

class testBogusFunction( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testBogusFunction
    )
)

# ...

@testSerializationModuleCoverage.AddMethodTesting
@testSerializationModuleCoverage.AddPropertyTesting
class testIsJSONSerializable( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testIsJSONSerializable
    )
)

@testSerializationModuleCoverage.AddMethodTesting
@testSerializationModuleCoverage.AddPropertyTesting
class testJSONSerializableStub( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testJSONSerializableStub
    )
)

# ...

These test-cases, with no test-methods in them at all, yield the following test-failures:

testMethodCoverage (__main__.testIsJSONSerializable)
Traceback (most recent call last):
  File "unit_testing.py", line 283, in testMethodCoverage

AssertionError: Unit-testing policy requires test-methods 
to be created for all public and protected methods, but 
testIsJSONSerializable is missing the following test-methods: 
  ['testFromJSON', 'testRegisterLoadable', 'testSanitizeDict', 
  'test_GetPythonNamespace', 'test_GetSanitizedJSON', 
  'test__init__', 'testwrapjsondump', 'testwrapjsondumps', 
  'testwrapjsonload', 'testwrapjsonloads']

testPropertyCoverage (__main__.testIsJSONSerializable)
Traceback (most recent call last):
  File "unit_testing.py", line 324, in testPropertyCoverage

AssertionError: Unit-testing policy requires test-methods 
to be created for all public properties, but 
testIsJSONSerializable is missing the following test-methods: 
  ['testPythonNamespace', 'testSanitizedJSON']

testMethodCoverage (__main__.testJSONSerializableStub)
Traceback (most recent call last):
  File "unit_testing.py", line 283, in testMethodCoverage

AssertionError: Unit-testing policy requires test-methods 
to be created for all public and protected methods, but 
testJSONSerializableStub is missing the following 
test-methods: 
  ['testFromDict', 'testGetSerializationDict', 'test__init__']

testPropertyCoverage (__main__.testJSONSerializableStub)
Traceback (most recent call last):
  File "unit_testing.py", line 324, in testPropertyCoverage

AssertionError: Unit-testing policy requires test-methods 
to be created for all public properties, but 
testJSONSerializableStub is missing the following 
test-methods:
  ['testFieldName1', 'testFieldName2']

As soon as the applicable test-methods are defined (testFieldName1 and testFieldName2 in the last result, for example), those failures go away, ensuring that all the class-members that I require test-methods for have test-methods (even if they don't do anything).

That, I think, pretty solidly answers

How can I identify what code-elements need tested?

In my next post, I'll start considering

What needs to happen in order to test those elements to my satisfaction?