Unit-testing and Code-coverage in Python [2]

Ideally, if possible, I'd very much like for code-coverage testing to be a trivial thing to put in place in a unit-test module. The best-case scenario I can think of would be to have some common unittest.TestCase-derived class that can just be imported into a given unit-test module, and it would take care of running (perhaps even generating) all the coverage-related testing that I want to have in place.

I'm still using the same copy of the serialization module, with an additional concrete class (JSONSerializableStub) added in to assure that I'm testing an interface, and abstract class, and a concrete class. I've also added a BogusFunction function, because I realized that I also wanted to make sure I was forcing testing of module-level functions as well as classes — though I use them rarely, they should also be thoroughly tested...

Testing for Test-Cases

The first task is to gather up all of that example inspect-based code from the last post, and come up with a test-mechanism to assert that all classes and functions present in the module being tested have a corresponding unittest.TestCase test-case class. Since the module being tested is an unknown quantity as far as the definition of the module-coverage-test class is concerned, I'll need to make sure that it's available to the test-case class. I'll accomplish that by making it a class-attribute. The balance of the code for doing all of the examination and generating the final testCodeCoverage test-method looks like this:

class moduleCoverageTest( unittest.TestCase ):
    """
Unit-test that checks to make sure that all classes in the module being tested 
have corresponding test-case classes in the unit-test module where the derived 
class is defined."""

    #-----------------------------------#
    # Default class constants that      #
    # point to the namespace and module #
    # being tested                      #
    #-----------------------------------#
    _testNamespace = None
    _testModule = None

    @classmethod
    def setUpClass( cls ):
        # Get all the classes available in the module
        cls._moduleClasses = inspect.getmembers( 
            cls._testModule, inspect.isclass )
        # Get all the functions available in the module
        cls._moduleFunctions = inspect.getmembers( 
            cls._testModule, inspect.isfunction )
        # Collect all the *LOCAL* items
        cls._testModuleName = cls._testModule.__name__
        # Find and keep track of all of the test-cases that relate to 
        # classes in the module being tested
        cls._classTests = dict(
            [
                ( 'test%s' % m[ 0 ], m[ 1 ] ) 
                for m in cls._moduleClasses
                if m[ 1 ].__module__ == cls._testModuleName
            ]
        )
        # Ditto for the functions in the module being tested
        cls._functionTests = dict(
            [
                ( 'test%s' % m[ 0 ], m[ 1 ] ) 
                for m in cls._moduleFunctions
                if m[ 1 ].__module__ == cls._testModuleName
            ]
        )
        # The list of required test-case class-names is the aggregated 
        # list of all class- and function-test-case-class names
        cls._requiredTestCases = sorted( 
            cls._classTests.keys() + cls._functionTests.keys()
        )
        # Find and keep track of all of the actual test-case classes in 
        # the module the class resides in
        cls._actualTestCases = dict(
            [ 
                item for item in 
                inspect.getmembers( inspect.getmodule( cls ), 
                  inspect.isclass ) 
                if item[ 1 ].__name__[ 0:4 ] == 'test'
                and issubclass( item[ 1 ], unittest.TestCase )
            ]
        )
        # Calculate the missing test-case-class names, for use by 
        # the testCodeCoverage test-method
        cls._missingTestCases = sorted( 
            set( cls._requiredTestCases ).difference( 
              set( cls._actualTestCases.keys() ) ) )

    def testCodeCoverage(self):
        self.assertEquals( [], self._missingTestCases, 
            'Unit-testing policies require test-cases for all classes and '
            'functions in the %s module, but the following have not been '
            'defined: (%s)' % ( 
                self.__class__._testModule.__name__, 
                ', '.join( self._missingTestCases )
            )
        )

There is a fair amount of pre-processing in the setUpClass method of moduleCoverageTest. setUpClass is called during a unit-test run on each class before any of its test* methods, providing a hook for operations that need to take place before any tests are actually run. In this case, I'm doing a fair amount of inspect-based examination of the module that the class resides in, and the module that the test-module is testing, in order to pre-calculate various data that I expect to use in testing for the presence of test-case classes and (later) property- and method-coverage tests. Those various calculations and aggregations should, ideally, happen only once for any given coverage-test, and need to persist across the life of the test-case class, so the setUpClass method seemed a good place to do all that. It does raise the potential concern that classes derived from moduleCoverageTest could override that method, putting the fidelity of code-coverage testing at risk. Normally, I'd be looking pretty seriously at making the moduleCoverageTest class nominally final, but since subclassing it is kind of key to its purpose, that risk is going to remain unmitigated, at least for now.

With this collected/encapsulated into a common, importable class, it becomes very simple to perform the code-coverage test — All that needs to happen is to import the moduleCoverageTest class, then define a subclass of it, with the namespace and source-module set as class properties of that subclass. A bare-bones test-module for the serialization module starts taking shape as:

import unittest
import serialization

from unit_testing import moduleCoverageTest

#-----------------------------------#
# Set up a local test-suite         #
#-----------------------------------#
LocalSuite = unittest.TestSuite()

#-----------------------------------#
# Import the classes, etc., from    #
# the module/package being tested   #
#-----------------------------------#

#-----------------------------------#
# Test-cases in the module          #
#-----------------------------------#
class testModuleCoverage( moduleCoverageTest ):
    _testNamespace = 'idic'
    _testModule = serialization

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testModuleCoverage
    )
)

#-----------------------------------#
# Run local unit-tests if the       #
# module is called directly         #
#-----------------------------------#

if __name__ == '__main__':
    results = unittest.TestResult()
    LocalSuite.run( results )

    print results
    if results.errors:
        for testCase, error in results.errors:
            print testCase
            print error
    if results.failures:
        for testCase, failure in results.failures:
            print testCase
            print failure

There are a few items omitted from this listing, in order to keep it short and relevant, but they mostly center around making sure that imports of the module being tested, and of all its members are done. The output is crude, but functional enough for now. Running this test-code yields a single failure:

<unittest.result.TestResult run=1 errors=0 failures=1>
testCodeCoverage (__main__.testModuleCoverage)
Traceback (most recent call last):
  File "unit_testing.py", line 144, in testCodeCoverage

AssertionError: Unit-testing policies require test-cases for 
all classes and functions in the serialization module, but 
the following have not been defined: 
  (testBogusFunction, testHasSerializationDict, 
  testIsJSONSerializable, testJSONSerializableStub, 
  testUnsanitizedJSONWarning)

which, at this point, is exactly what I want. It's identified that there are no test-case classes for any of the serialization classes, and caused the test to fail.

Perfect!

If it's working as expected, then all that need happen to allow the test-case coverage test to pass is to create the missing test-case classes, like so:

#-----------------------------------#
# Test-cases in the module          #
#-----------------------------------#
class testModuleCoverage( moduleCoverageTest ):
    _testNamespace = 'idic'
    _testModule = serialization

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testModuleCoverage
    )
)

class testBogusFunction( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testBogusFunction
    )
)

class testHasSerializationDict( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testHasSerializationDict
    )
)

class testIsJSONSerializable( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testIsJSONSerializable
    )
)

class testJSONSerializableStub( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testJSONSerializableStub
    )
)

class testUnsanitizedJSONWarning( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testUnsanitizedJSONWarning
    )
)

With those additional test-cases, even as useless as they currently are, re-running the test-code yields no failures:

<unittest.result.TestResult run=1 errors=0 failures=0>

That part, then, is good to go.

Testing Coverage of Class Members

In similar unit-testing code that I wrote for work, the process that got implemented ended up performing a check for the presence of all test-cases, listing any that were expected but not created across the entire module (just like what's happening in this implementation so far). It then did a similarly-scoped check across all test-cases looking for missing test-methods for properties and methods in the source classes, but because it was searching the entire module, it would only return one failure at a time. While that was workable, and still achieved the amount of code-coverage that I wanted to accomplish, it did get somewhat tedious, having to run the test-suite, find the single reported missing property and/or method test-results, correct that, then re-run again and repeat until there were no more missing test-methods.

I'd consider that to be a viable fall-back position, but I'd much rather have a separate assertion-failure report for each test-case class, and separate assertion-failures for missing property tests and method tests both. Implementing something at that level of granularity is possible, but there are at least two distinct approaches that could be taken, both with their advantages and potential drawbacks.

The first option, to my thinking, would be to subclass the existing unittest.TestCase class, and add in methods for testing whether the class has all of the appropriate test-methods for the properties and methods of the source-class being tested. That's certainly feasible. The drawback to that, as I see it, is that there might well be other reasons to need to subclass unittest.TestCase, and if/when those arise, that would then mean, potentially, subclassing the original property- and method-test-aware subclass of unittest.TestCase. That has the potential of getting unweildy pretty quickly. There's also considerable potential, I think, for introducing unit-testing failures if that sort of scenario were to play out. Nevertheless, it's still a viable option.

Arguably, a better approach would be to have some mechanism that examines the source-classes for properties and methods that should have corresponding test-methods, and coming up with some way to require a test-method that tests the coverage of those test-methods in comparison to the source-class' members. I believe that the ideal approach would be for the existing moduleCoverageTest class to be able to insert property- and method-coverage tests directly into the test-case classes that it's already keeping track of.

Unfortunately, when I tested that approach, although it certainly was possible to add testMethodCoverage and testPropertyCoverage test-methods to the individual test-case classes, those test-methods did not fire — I'm guessing that it's a sequencing issue of some sort, that all of a TestCase-class' test-methods have to be defined in some fashion at the class level, in the code, so that they are members of their respective classes during the interpretation-run of against the code. Even using the same sort of decoration-like approach that I used for overriding the various json-module functions in the serialization module didn't have the desired effect. The test-methods existed, as far as I could tell, but they didn't fire. A cursory examination of the members of unittest.TestCase didn't lead me to believe that there was any way to register or add a new test-method on the fly, though there may well be one that just wasn't apparent to me.

One more possibility occurred to me, and when I tried it out, it apparently worked — another Python decoration approach. Specifically, I found that if I defined decorator functions like so:

def AddPropertyTesting( cls ):
    def testPropertyCoverage( self ):
        self.assertEquals( '%s.testPropertyCoverage called' % cls.__name__, None )
    cls.testPropertyCoverage = testPropertyCoverage
    return cls

def AddMethodTesting( cls ):
    def testMethodCoverage( self ):
        self.assertEquals( '%s.testMethodCoverage called' % cls.__name__, None )
    cls.testMethodCoverage = testMethodCoverage
    return cls

and decorated the test-case classes accordingly:

@AddPropertyTesting
@AddMethodTesting
class testHasSerializationDict( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testHasSerializationDict
    )
)

I at least got some potentially-viable results:

testMethodCoverage (__main__.testHasSerializationDict)
Traceback (most recent call last):
  File "unit_testing.py", line 71, in testMethodCoverage

AssertionError: 
  'testHasSerializationDict.testMethodCoverage called' != None

testPropertyCoverage (__main__.testHasSerializationDict)
Traceback (most recent call last):
  File "unit_testing.py", line 65, in testPropertyCoverage

AssertionError: 
  'testHasSerializationDict.testPropertyCoverage called' != None

These don't look like much, but they at least prove that a decorator-based approach might solve the issue. The one remaining missing piece that I needed was a way to determine if the individual test-case classes had been appropriately decorated, which could be managed as an additional test-method of the moduleCoverageTest class:

def testPropertyCoverage( self ):
    for testCaseClassName in self._classTests:
        # Get the source class, the class being tested
        sourceClass = self._classTests[ testCaseClassName ]
        # Get the corresponding test-case class, locally
        testClass = self._actualTestCases[ testCaseClassName ]
        # Determine if the source class has any properties that 
        # should be tested with the AddPropertyTesting decorator
        sourceProps = [
                pt for pt in inspect.getmembers( sourceClass, 
                    inspect.isdatadescriptor )
                if pt[ 0 ][ 0:2 ] != '__'
            ]
        if sourceProps:
            # If there are properties, then require the decorator
            self.assertNotEqual( 
                testClass.__dict__.get( 'testPropertyCoverage' ),
                None, 
                'Unit-testing policy requires that %s implement a '
                'testPropertyCoverage test-method, since %s has '
                'active properties (%s). Please decorate %s with the '
                '@AddPropertyTesting function.' % ( 
                    testClass.__name__, sourceClass.__name__, 
                    ', '.join( sorted( 
                        [ sp[ 0 ] for sp in sourceProps ]
                    ) ), testClass.__name__
                )
            )

That at least allows the test-process to determine that something needs the property- or method-coverage test-decorators, and cause an assertion-failure if they aren't provided:

<unittest.result.TestResult run=2 errors=0 failures=1>
testPropertyCoverage (__main__.testModuleCoverage)
Traceback (most recent call last):
  File "unit_testing.py", line 185, in testPropertyCoverage

AssertionError: Unit-testing policy requires that 
testJSONSerializableStub implement a testPropertyCoverage 
test-method, since JSONSerializableStub has active properties 
  (FieldName1, FieldName2, PythonNamespace, SanitizedJSON). 
Please decorate testJSONSerializableStub with the 
  @AddPropertyTesting function.

After some tinkering, I managed to get the decoration-based process to work rather well, I think. In use, it will probably feel a little weird for a while, but it appears to be surprisingly robust even this early in the game, and it provides much more granular test-results for missing test-methods than the viable fall-back position mentioned at the start of this post — one test each that checks for source-class property- and method-tests in the test-case class, with a nice, itemized list of what's missing for that test-case.

The Final setUpClass method

Support-data for the AddMethodTesting and AddPropertyTesting methods had to be added to setUpClass, providing the sets of expected test-method names for the methods and properties of the source classes. Picking up where it left off above, these additional items are built like this (after the calculation of cls._missingTestCases:

# Calculate the missing test-case-class names, for use by 
# the testCodeCoverage test-method
cls._missingTestCases = sorted( 
    set( cls._requiredTestCases ).difference( 
        set( cls._actualTestCases.keys() ) ) )

# Calculate the property test-case names for all the 
# module's classes
cls._propertyTestsByClass = {}
for testClass in cls._classTests:
    cls._propertyTestsByClass[ testClass ] = set()
    sourceClass = cls._classTests[ testClass ]
    sourceMRO = list( sourceClass.__mro__ )
    sourceMRO.reverse()
    # Get all the item's properties
    properties = [
        member for member in inspect.getmembers( 
            sourceClass, inspect.isdatadescriptor )
        if member[ 0 ][ 0:2 ] != '__'
    ]
    # Create and populate data-structures that keep track of where 
    # property-members originate from, and what their implementation 
    # looks like. Initially populated with None values:
    propSources = {}
    propImplementations = {}
    for name, value in properties:
        propSources[ name ] = None
        propImplementations[ name ] = None
    for memberName in propSources:
        implementation = sourceClass.__dict__.get( memberName )
        if implementation and propImplementations[ memberName ] != implementation:
            propImplementations[ memberName ] = implementation
            propSources[ memberName ] = sourceClass
    cls._propertyTestsByClass[ testClass ] = set(
        [
            'test%s' % key for key in propSources 
            if propSources[ key ] == sourceClass
        ]
    )

# Calculate the method test-case names for all the module's classes
cls._methodTestsByClass = {}
for testClass in cls._classTests:
    cls._methodTestsByClass[ testClass ] = set()
    sourceClass = cls._classTests[ testClass ]
    sourceMRO = list( sourceClass.__mro__ )
    sourceMRO.reverse()
    # Get all the item's methods
    methods = [
        member for member in inspect.getmembers( 
            sourceClass, inspect.ismethod )
    ] + [
        member for member in inspect.getmembers( 
            sourceClass, inspect.isfunction )
    ]
    # Create and populate data-structures that keep track of where 
    # method-members originate from, and what their implementation 
    # looks like. Initially populated with None values:
    methSources = {}
    methImplementations = {}
    for name, value in methods:
        methSources[ name ] = None
        methImplementations[ name ] = None
    for memberName in methSources:
        implementation = sourceClass.__dict__.get( memberName )
        if implementation and methImplementations[ memberName ] != implementation:
            methImplementations[ memberName ] = implementation
            methSources[ memberName ] = sourceClass
    cls._methodTestsByClass[ testClass ] = set(
        [
            'test%s' % key for key in methSources 
            if methSources[ key ] == sourceClass
        ]
    )

In both cases, the new data is populated for all the source-classes being tested. Each class is retrieved, the Method Resolution Order (MRO) for the class is retrieved, and each superclass of the source-class is checked for each member in the source-class. This allows the expected test-cases for any given class to be built around the members that are implemented within the source class only. For example, the JSONSerializableStub class, though it has all of the inherited members from IsJSONSerializable and so on, it only implements the members that are required (as abstract members from the classes it derives from). The test-methods for IsJSONSerializable will be responsible for testing the members present there, and until or unless JSONSerializableStub overrides a non-abstract member of IsJSONSerializable (taking ownership of it, as it were), there's no requirement to test the members inherited from IsJSONSerializable in JSONSerializableStub.

The Final AddMethodTesting method

The final AddMethodTesting decorator-method relies on the main class that it's bound to storing a fair chunk of information about the module being tested, the members of that module, and the members of those members. All that data is still compiled in the test-case class derived from moduleCoverageTest, during its setUpClass method.

@classmethod
def AddMethodTesting( cls, target ):
    if cls.__name__ == 'moduleCoverageTest':
        raise RuntimeError( 'moduleCoverageTest should be extended '
            'into a local test-case class, not used as one directly.' )
    if not cls._testModule:
        raise AttributeError( '%s does not have a _testModule defined '
            'as a class attribute. Check that the decorator-method is '
            'being called from the extended local test-case class, not '
            'from moduleCoverageTest itself.' % ( cls.__name__ ) )
    try:
        if cls._methodTestsByClass:
            populate = False
        else:
            populate = True
    except AttributeError:
        populate = True
    if populate:
        cls.setUpClass()
    def testMethodCoverage( self ):
        requiredTestMethods = cls._methodTestsByClass[ target.__name__ ]
        activeTestMethods = set(
            [
                m[ 0 ] for m in 
                inspect.getmembers( target, inspect.ismethod )
                if m[ 0 ][ 0:4 ] == 'test'
            ]
        )
        missingMethods = sorted( 
            requiredTestMethods.difference( activeTestMethods )
        )
        self.assertEquals( [], missingMethods, 
            'Unit-testing policy requires test-methods to be created for '
            'all public and protected methods, but %s is missing the '
            'following test-methods: %s' % ( 
                target.__name__, missingMethods
            )
        )
    target.testMethodCoverage = testMethodCoverage
    return target

In order to (hopefully) eliminate any possibility of accidentally using the original moduleCoverageTest decorator-methods, a set of checks is performed first, raising errors if the class is moduleCoverageTest, or if no _testModule class-attribute has been specified. Although it should never be needed, I'm also checking for class-attributes created and populated by setUpClass, and explicilty calling setUpClass if those attributes don't exist.

After that, it's all just getting the test-method names expected/required, removing any test-method names that exist, and checking that the remaining set of test-method names is empty. If it is, then all the required methods exist, and the test will pass. If there are any missing test-method names remaining, then the test will fail, and display the missing test-method names as part of its failure message. All of this happens inside a function defined inside the closure of the wrapping class-method, so all of the class-properties of the moduleCoverageTest-derived class are available.

The Final AddPropertyTesting method

Structurally, this is pretty much identical to the AddMethodTesting method already described. The only substantive differences are which data-structure in the owner class is being used as the check-source, and the messaging:

@classmethod
def AddPropertyTesting( cls, target ):
    if cls.__name__ == 'moduleCoverageTest':
        raise RuntimeError( 'moduleCoverageTest should be extended '
            'into a local test-case class, not used as one directly.' )
    if not cls._testModule:
        raise AttributeError( '%s does not have a _testModule defined '
            'as a class attribute. Check that the decorator-method is '
            'being called from the extended local test-case class, not '
            'from moduleCoverageTest itself.' % ( cls.__name__ ) )
    try:
        if cls._propertyTestsByClass:
            populate = False
        else:
            populate = True
    except AttributeError:
        populate = True
    if populate:
        cls.setUpClass()
    def testPropertyCoverage( self ):
        requiredTestMethods = cls._propertyTestsByClass[ target.__name__ ]
        activeTestMethods = set(
            [
                m[ 0 ] for m in 
                inspect.getmembers( target, inspect.ismethod )
                if m[ 0 ][ 0:4 ] == 'test'
            ]
        )
        missingMethods = sorted( 
            requiredTestMethods.difference( activeTestMethods )
        )
        self.assertEquals( [], missingMethods, 
            'Unit-testing policy requires test-methods to be created for '
            'all public properties, but %s is missing the following test-'
            'methods: %s' % ( target.__name__, missingMethods )
        )
    target.testPropertyCoverage = testPropertyCoverage
    return target

Example serialization Test-Cases

Before I break for the day, let me just dump the results of the following test-case classes, with the applicable decoration in place:

class testSerializationModuleCoverage( moduleCoverageTest ):
    _testNamespace = 'idic'
    _testModule = serialization

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testSerializationModuleCoverage
    )
)

class testBogusFunction( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testBogusFunction
    )
)

# ...

@testSerializationModuleCoverage.AddMethodTesting
@testSerializationModuleCoverage.AddPropertyTesting
class testIsJSONSerializable( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testIsJSONSerializable
    )
)

@testSerializationModuleCoverage.AddMethodTesting
@testSerializationModuleCoverage.AddPropertyTesting
class testJSONSerializableStub( unittest.TestCase ):
    pass

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testJSONSerializableStub
    )
)

# ...

These test-cases, with no test-methods in them at all, yield the following test-failures:

testMethodCoverage (__main__.testIsJSONSerializable)
Traceback (most recent call last):
  File "unit_testing.py", line 283, in testMethodCoverage

AssertionError: Unit-testing policy requires test-methods 
to be created for all public and protected methods, but 
testIsJSONSerializable is missing the following test-methods: 
  ['testFromJSON', 'testRegisterLoadable', 'testSanitizeDict', 
  'test_GetPythonNamespace', 'test_GetSanitizedJSON', 
  'test__init__', 'testwrapjsondump', 'testwrapjsondumps', 
  'testwrapjsonload', 'testwrapjsonloads']

testPropertyCoverage (__main__.testIsJSONSerializable)
Traceback (most recent call last):
  File "unit_testing.py", line 324, in testPropertyCoverage

AssertionError: Unit-testing policy requires test-methods 
to be created for all public properties, but 
testIsJSONSerializable is missing the following test-methods: 
  ['testPythonNamespace', 'testSanitizedJSON']

testMethodCoverage (__main__.testJSONSerializableStub)
Traceback (most recent call last):
  File "unit_testing.py", line 283, in testMethodCoverage

AssertionError: Unit-testing policy requires test-methods 
to be created for all public and protected methods, but 
testJSONSerializableStub is missing the following 
test-methods: 
  ['testFromDict', 'testGetSerializationDict', 'test__init__']

testPropertyCoverage (__main__.testJSONSerializableStub)
Traceback (most recent call last):
  File "unit_testing.py", line 324, in testPropertyCoverage

AssertionError: Unit-testing policy requires test-methods 
to be created for all public properties, but 
testJSONSerializableStub is missing the following 
test-methods:
  ['testFieldName1', 'testFieldName2']

As soon as the applicable test-methods are defined (testFieldName1 and testFieldName2 in the last result, for example), those failures go away, ensuring that all the class-members that I require test-methods for have test-methods (even if they don't do anything).

That, I think, pretty solidly answers

How can I identify what code-elements need tested?

In my next post, I'll start considering

What needs to happen in order to test those elements to my satisfaction?