I Dream In Code: A Unit-testing Walk-through

If you've been following this blog for any length of time, it should come as no great surprise to you that I have set up template-files/code-snippets for writing unit-test modules and individual test-cases. I won't show them all at once this time — they will be exposed sufficiently, I think, as I write the tests for the serialization module — but they are written with an eye towards making my normal unit-testing process as smooth and painless as possible:

Create a test-module from the UnitTestsTemplate.py file, named test_{module being tested}.py, in a project directory named test_{project name};
Set up the various namespace- and module-name-values that the test-module needs to be able to find the source-module being tested;
Run the test-module until there are no failed tests.
- Generate test-case classes to address each failure that stems from a requirement reported from the code-coverage test;
- Generate test-methods to address each failure that stems from a requirement reported from the tests generated by the AddMethodTesting and AddPropertyTesting decorators;
- Resolve any other failures reported;
- Resolve any errors reported;
- If test-methods cannot be usefully generated, apply the unittest.skip decorator to those.
If the test-module should be part of a larger set of tests (say, for an entire package), then add the test-module to the relevant parent module- or package-test file, then run it as above;

Since the serialization module is part of the idic package, the end-result of todays post should be a test_idic.py file that calls the tests from a test_serialization.py file. I'll create the test_serialization.py file first, then work my way up to test_idic.py.

Starting with the `UnitTestsTemplate.py` file

So, following my process, the first things I need to do are create a copy of the unit-test template file in the appropriate location, and change all of the namespace- and module-name strings in it to match what I'm testing. The most relevant chunk of that happens at the start of the file, which looks like this initially:

#!/usr/bin/env python
"""Defines unit-tests for the module at PackagePath.ModuleName."""

# Python unit-test-module template. Copy the template to a new 
# unit-test-module location, and start replacing names as needed:
#
# PackagePath  ==> The path/namespace of the parent of the module/package 
#                  being tested in this file.
# ModuleName   ==> The name of the module being tested
#
# Then remove this comment-block

#-----------------------------------#
# Standard-library imports.         #
#-----------------------------------#

import os
import sys
import unittest

#-----------------------------------#
# Imports of other third-party      #
# libraries and functionality.      #
#-----------------------------------#

#-----------------------------------#
# idic-library imports.             #
#-----------------------------------#
# - Local development path
sys.path.insert( 1, os.path.expanduser( 
    '~/path/to/local/project/lib/project_name' ) )
# - Installed location
sys.path.insert( 1, '/usr/local/lib/idic' )

from idic.unit_testing import *

#-----------------------------------#
# Import the module being tested    #
#-----------------------------------#
LocalSuite = unittest.TestSuite()

#-----------------------------------#
# Import the module being tested    #
#-----------------------------------#
import PackagePath.ModuleName as ModuleName

#-----------------------------------#
# Code-coverage test-case and       #
# decorator-methods                 #
#-----------------------------------#

class testModuleNameCodeCoverage( moduleCoverageTest ):
    _testNamespace = 'PackagePath'
    _testModule = ModuleName

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testModuleNameCodeCoverage
    )
)

#-----------------------------------#
# Test-cases in the module          #
#-----------------------------------#

A couple of quick search-and-replaces in the file are all I need to do to get started:

Replacing every instance of PackagePath with the namespace path to the module being tested (in this case, idic, since the full namespace of the serialization module would be idic.serialization); and
Replacing every instance of ModuleName with the module name of the module being tested (serialization in this case).

After those replacements (and removing the comments near the top as they instruct), the unit-test module starts with:

#!/usr/bin/env python
"""Defines unit-tests for the module at idic.serialization."""

#-----------------------------------#
# Standard-library imports.         #
#-----------------------------------#

import os
import sys
import unittest

#-----------------------------------#
# Imports of other third-party      #
# libraries and functionality.      #
#-----------------------------------#

#-----------------------------------#
# idic-library imports.             #
#-----------------------------------#
# - Local development path
sys.path.insert( 1, os.path.expanduser( 
    '~/path/to/local/project/lib/project_name' ) )
# - Installed location
sys.path.insert( 1, '/usr/local/lib/idic' )

from idic.unit_testing import *

#-----------------------------------#
# Import the module being tested    #
#-----------------------------------#
LocalSuite = unittest.TestSuite()

#-----------------------------------#
# Import the module being tested    #
#-----------------------------------#
import idic.serialization as serialization

#-----------------------------------#
# Code-coverage test-case and       #
# decorator-methods                 #
#-----------------------------------#

class testserializationCodeCoverage( moduleCoverageTest ):
    _testNamespace = 'idic'
    _testModule = serialization

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testserializationCodeCoverage
    )
)

#-----------------------------------#
# Test-cases in the module          #
#-----------------------------------#

Right now, with those changes made, it's executable as long as all the import-paths are correct. Running the test-module generates the following output, which tells what the next steps are:

############################################################
Unit-test results
############################################################
Tests were successful ... False
Number of tests run ..... 2
Number of errors ........ 0
Number of failures ...... 1
############################################################
FAILURES
#----------------------------------------------------------#
Traceback (most recent call last):
  File "unit_testing.py", line 160, in testCodeCoverage
    ', '.join( self._missingTestCases )
AssertionError: Unit-testing policies require test-cases 
    for all classes and functions in the idic.serialization 
    module, but the following have not been defined: 
    (testHasSerializationDict, testIsJSONSerializable, 
    testUnsanitizedJSONWarning)
############################################################
Unit-test results
############################################################
Tests were successful ... False
Number of tests run ..... 2
Number of errors ........ 0
Number of failures ...... 1
############################################################

The output of PrintTestResults includes two copies of the results because the output may well be long enough that the data at the top may be lost off the scrollable area in command-line output. I've toyed with the idea of only generating the end output-information if a certain number of errors or failures has occurred, but that's never really been a high enough priority for me to follow through with it.

That first set of failure-results is nothing more than a list of test-case classes that need to be defined in order for the test-module to provide the required code-coverage.

Creating `TestCase` Classes

The next step is to generate test-case classes for each of the items noted in the initial failure: testHasSerializationDict, testIsJSONSerializable, and testUnsanitizedJSONWarning. For each of those, I start with the code in my TestCaseTemplate.py file. Removing some of the common helper-methods that use at least occastionally, and the optional set-up and tear-down, there's not a lot to that file:

@testModuleNameCodeCoverage.AddMethodTesting
@testModuleNameCodeCoverage.AddPropertyTesting
class testClassName( unittest.TestCase ):
    """Unit-tests the ClassName class."""

    #--------------------------------------#
    # Unit-tests for class constants,      #
    # if any                               #
    #--------------------------------------#

    #--------------------------------------#
    # Unit-tests the object constructor,   #
    # including any property-values set    #
    # during construction of an instance.  #
    #--------------------------------------#

    def test__init__(self):
        """Unit-tests the initialization of a ClassName instance."""
        self.fail( 'test__init__ is not yet implemented' )

    #--------------------------------------#
    # Unit-tests the object destructor, if #
    # one is provided.                     #
    #--------------------------------------#

#    def test__del__(self):
#        """Unit-tests the destruction of a ClassName instance."""
#        self.fail( 'test__del__ is not yet implemented' )

    #--------------------------------------#
    # Unit-tests of object properties      #
    #--------------------------------------#

#    def testPROPERTYNAME(self):
#        """Unit-tests the PROPERTYNAME property of a ClassName instance."""
#        self.fail( 'testPROPERTYNAME is not yet implemented' )

    #--------------------------------------#
    # Unit-tests of object methods         #
    #--------------------------------------#

#    def testMETHODNAME(self):
#        """Unit-tests the METHODNAME method of a ClassName instance."""
#        self.fail( 'testMETHODNAME is not yet implemented' )


LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testClassName 
    )
)

Since I usually use template-files rather than snippets, I generally copy the test-case template, paste it into the test-file, then replace all the instances of ClassName with the name of the class that the test-case relates to. The AddMethodTesting and AddPropertyTesting decorators also need to be bound to the initial code-coverage test-case (testserializationCodeCoverage here), in order to avoid any potential requirements contamination across test-modules. By way of example, after adding a test-case to be pointed at HasSerializationDict, the test-module has a test-case the looks like this:

@testserializationCodeCoverage.AddMethodTesting
@testserializationCodeCoverage.AddPropertyTesting
class testHasSerializationDict( unittest.TestCase ):
    """Unit-tests the HasSerializationDict class."""

    #--------------------------------------#
    # Unit-tests for class constants,      #
    # if any                               #
    #--------------------------------------#

    #--------------------------------------#
    # Unit-tests the object constructor,   #
    # including any property-values set    #
    # during construction of an instance.  #
    #--------------------------------------#

    def test__init__(self):
        """Unit-tests the initialization of a HasSerializationDict instance."""
        self.fail( 'test__init__ is not yet implemented' )

    #--------------------------------------#
    # Unit-tests the object destructor, if #
    # one is provided.                     #
    #--------------------------------------#

#    def test__del__(self):
#        """Unit-tests the destruction of a HasSerializationDict instance."""
#        self.fail( 'test__del__ is not yet implemented' )

    #--------------------------------------#
    # Unit-tests of object properties      #
    #--------------------------------------#

#    def testPROPERTYNAME(self):
#        """Unit-tests the PROPERTYNAME property of a HasSerializationDict instance."""
#        self.fail( 'testPROPERTYNAME is not yet implemented' )

    #--------------------------------------#
    # Unit-tests of object methods         #
    #--------------------------------------#

#    def testMETHODNAME(self):
#        """Unit-tests the METHODNAME method of a HasSerializationDict instance."""
#        self.fail( 'testMETHODNAME is not yet implemented' )


LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testHasSerializationDict 
    )
)

Running the test-module now yields different output (I've removed the end-of-test information for brevity):

############################################################
Unit-test results
############################################################
Tests were successful ... False
Number of tests run ..... 5
Number of errors ........ 0
Number of failures ...... 3
############################################################
FAILURES
#----------------------------------------------------------#
Traceback (most recent call last):
  File "unit_testing.py", line 160, in testCodeCoverage
    ', '.join( self._missingTestCases )
AssertionError: Unit-testing policies require test-cases 
    for all classes and functions in the idic.serialization 
    module, but the following have not been defined: 
    (testIsJSONSerializable, testUnsanitizedJSONWarning)
#----------------------------------------------------------#
Traceback (most recent call last):
  File "unit_testing.py", line 348, in testMethodCoverage
    target.__name__, missingMethods
AssertionError: Unit-testing policy requires test-methods 
    to be created for all public and protected methods, but 
    testHasSerializationDict is missing the following 
    test-methods: 
    ['testFromDict', 'testGetSerializationDict']
#----------------------------------------------------------#
Traceback (most recent call last):
  File "testserialization.py", line 122, in test__init__
    self.fail( 'test__init__ is not yet implemented' )
AssertionError: test__init__ is not yet implemented
############################################################

There are a couple of noteworthy items in this output:

The original list of test-cases required has gone down to two items: testIsJSONSerializable and testUnsanitizedJSONWarning, because the testHasSerializationDict test-case class exists now.
The testMethodCoverage that was attached by the AddMethodTesting decorator is working, and has identified that two test-methods need to be generated: testFromDict and testGetSerializationDict. Those are both abstract (or at least nominally-abstract) methods, but they live in the scope of the class, and can be usefully tested, I think, so the fact that tests are being required is a good thing in my opinion.
Finally, there is a forced failure for the __init__ method of HasSerializationDict.

After adding test-case classes for testIsJSONSerializable and testUnsanitizedJSONWarning, the failures change yet again:

############################################################
Unit-test results
############################################################
Tests were successful ... False
Number of tests run ..... 11
Number of errors ........ 0
Number of failures ...... 6
############################################################
FAILURES
#----------------------------------------------------------#
Traceback (most recent call last):
  File "unit_testing.py", line 348, in testMethodCoverage
    target.__name__, missingMethods
AssertionError: Unit-testing policy requires test-methods 
    to be created for all public and protected methods, 
    but testHasSerializationDict is missing the following 
    test-methods: 
    ['testFromDict', 'testGetSerializationDict']
#----------------------------------------------------------#
Traceback (most recent call last):
  File "testserialization.py", line 122, in test__init__
    self.fail( 'test__init__ is not yet implemented' )
AssertionError: test__init__ is not yet implemented
#----------------------------------------------------------#
Traceback (most recent call last):
  File "unit_testing.py", line 348, in testMethodCoverage
    target.__name__, missingMethods
AssertionError: Unit-testing policy requires test-methods 
    to be created for all public and protected methods, but 
    testIsJSONSerializable is missing the following 
    test-methods: 
    ['testFromJSON', 'testRegisterLoadable', 
    'testSanitizeDict', 'test_GetPythonNamespace', 
    'test_GetSanitizedJSON', 'testwrapjsondump', 
    'testwrapjsondumps', 'testwrapjsonload', 
    'testwrapjsonloads']
#----------------------------------------------------------#
Traceback (most recent call last):
  File "unit_testing.py", line 389, in testPropertyCoverage
    'methods: %s' % ( target.__name__, missingMethods )
AssertionError: Unit-testing policy requires test-methods 
    to be created for all public properties, but 
    testIsJSONSerializable is missing the following 
    test-methods: 
    ['testPythonNamespace', 'testSanitizedJSON']
#----------------------------------------------------------#
Traceback (most recent call last):
  File "testserialization.py", line 174, in test__init__
    self.fail( 'test__init__ is not yet implemented' )
AssertionError: test__init__ is not yet implemented
#----------------------------------------------------------#
Traceback (most recent call last):
  File "testserialization.py", line 226, in test__init__
    self.fail( 'test__init__ is not yet implemented' )
AssertionError: test__init__ is not yet implemented
############################################################

The set-up for the top-level test-module, for the entire idic project-namespace follows the same structure, and uses the same starting-point, but has some minor differences. With all of the empty sections stripped out, this is what it boils down to:

#!/usr/bin/env python
"""Defines unit-tests for the package at idic."""

#-----------------------------------#
# Standard-library imports.         #
#-----------------------------------#

import os
import sys
import unittest

#-----------------------------------#
# idic-library imports.             #
#-----------------------------------#
# - Local development path
sys.path.insert( 1, os.path.expanduser( 
    '~/IDreamInCode/idic/usr/local/lib/idic' ) )
# - Installed location
sys.path.insert( 1, '/usr/local/lib/idic' )

from idic.unit_testing import *

#-----------------------------------#
# Import the module being tested    #
#-----------------------------------#
LocalSuite = unittest.TestSuite()

#-----------------------------------#
# Import the module being tested    #
#-----------------------------------#
import idic

#-----------------------------------#
# Code-coverage test-case and       #
# decorator-methods                 #
#-----------------------------------#

class testidicCodeCoverage( moduleCoverageTest ):
    _testNamespace = 'idic'
    _testModule = idic

LocalSuite.addTests( 
    unittest.TestLoader().loadTestsFromTestCase( 
        testidicCodeCoverage
    )
)

#-----------------------------------#
# Test-cases in the module          #
#-----------------------------------#

#-----------------------------------#
# Child test-cases to run           #
#-----------------------------------#

import test_serialization
LocalSuite.addTests( test_serialization.LocalSuite._tests )

#-----------------------------------#
# Code to execute if file is called #
# or run directly.                  #
#-----------------------------------#
if __name__ == '__main__':
    import time
    results = unittest.TestResult()
    testStartTime = time.time()
    LocalSuite.run( results )
    results.runTime = time.time() - testStartTime
    PrintTestResults( results )
    if not results.errors and not results.failures:
        SaveTestReport( results, 'idic', 
            'idic-test-results.txt' )

The main differences are:

The namespace change (idic), allowing it to test the idic package-header file;
The lack of test-case classes, because the package-header file for the idic namespace is currently an empty file; and
The inclusion of the test_serialization module's test-cases in the LocalSuite test-suite.

Taken together, these allow the test_idic test-module, when executed, to test the idic and idic.serialization namespaces:

###########################################################
Unit-test results
###########################################################
Tests were successful ... True
Number of tests run ..... 23
 + Tests ran in ......... 0.20 seconds
Number of errors ........ 0
Number of failures ...... 0
###########################################################

One additional test runs, the testCodeCoverage provided by the testidicCodeCoverage test-case class. The test_serialization test-module can still be run individually:

###########################################################
Unit-test results
###########################################################
Tests were successful ... True
Number of tests run ..... 22
 + Tests ran in ......... 0.19 seconds
Number of errors ........ 0
Number of failures ...... 0
###########################################################

As long as a similar set-up is put in place for each child module in the idic namespace (importing whichever test_module and adding the LocalSuite._tests from it), the test_idic module can be used to test the entire> idic namespace. Sub-packages inside the idic namespace, if any are eventually built out, can also use the same kind of structure, and will also be included in the tests run by test_idic in those cases.

Creating the Required Test-methods

I had originally planned to go into considerable depth in this post, including a step-by-step walk-through of generating the actual test-methods for all of the classes in the serialization module, and all of the members of those classes. When I'd finished writing it all up, it was a lot longer than I wanted. Also, perhaps because it was about unit-testing, it was really dry stuff.

Long and dry together sounds like a recipe for boring, so I'm going to shelve the discussion of detailed test-method implementation for now. I did generate unit-tests for everything in serialization, though they aren't complete, and if you're curious about them, they can be found in the idic.zip [Snapshot] download at the end of the page.

After I've had some time to think on a better way to present that level of detail, the in-the-weeds unit-testing, I'll come back to those, and probably revisit the current serialization tests.

But I think that's enough on unit-testing for the time being. It gets me where I needed to be to have standard testing policies, and ways to enforce them on my own code. It's also a key component in what I think of as a repeatable build-process — even if that's not a full-on Continuous Integration set-up, it's still in my list of things to accomplish for what I'd consider the minimum viable/bare-bones repeatable build-process:

Run automated tests and stop if any tests fail;
Generate notifications if the tests fail;
Package the build(s) in some fashion so that it's ready to be deployed;
Generate notifications if a build fails for reasons other than test-failures; and
Deploy to an environment where the current build can be executed.

Since I'm to the point where I need to be able to generate snapshots, it feels to me like it's about time to start looking at that repeatable build process idea.

UnitTestsTemplate.py

3.2kB

TestCaseTemplate.py

5.6kB

idic.zip [Snapshot]

41.2kB

I Dream In Code

Tuesday, April 4, 2017

A Unit-testing Walk-through — from the Ground Up

Starting with the `UnitTestsTemplate.py` file

Creating `TestCase` Classes

Creating the Required Test-methods

No comments:

Post a Comment

Tuesday, April 4, 2017

A Unit-testing Walk-through — from the Ground Up

Starting with the UnitTestsTemplate.py file

Creating TestCase Classes

Creating the Required Test-methods

No comments:

Post a Comment

Starting with the `UnitTestsTemplate.py` file

Creating `TestCase` Classes