Documentation Decorators: Keyword Arguments

The last of the argument-related documentation-decorators is for keyword arguments — the **kwargs in the methods of the Ook class that I've been using as a demonstration class for the past few articles:

class Ook( object ):
    """
Test-class."""
    @describe.argument( 'arg1', 'Ook.Fnord (method) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Fnord (method) arg2 description' )
    @describe.arglist( 'Ook.Fnord (method) arglist description', str, unicode )
    def Fnord( self, arg1, arg2, *args, **kwargs ):
        """Ook.Fnord (method) original doc-string"""
        return None

    @classmethod
    @describe.argument( 'arg1', 'Ook.Bleep (classmethod) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Bleep (classmethod) arg2 description' )
    @describe.arglist( 'Ook.Bleep (classmethod) arglist description' )
    def Bleep( cls, arg1, arg2=None, *args, **kwargs ):
        """Ook.Bleep (classmethod) original doc-string"""
        return None

    @staticmethod
    @describe.argument( 'arg1', 'Ook.Flup (staticmethod) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Flup (staticmethod) arg2 description' )
    @describe.arglist( 'Ook.Flup (staticmethod) arglist description', str, unicode )
    def Flup( arg1, arg2, *args, **kwargs ):
        """Ook.Flup (staticmethod) original doc-string"""
        return None

Keyword arguments, in use, are name-value pairs (e.g., keyword=value, keyword2=value2), and they allow a function or method to accept any number of arbitrarily-named arguments, including no arguments. Within the body of the function/method, they are expressed as a dict type. That is, a **kwargs passed keyword=value, keyword2=value2 would be accessable in the body of the function as kwargs, and would contain this dict:

{
    'keyword': value,
    'keyword2': value2
}

inside the function that they were provided to. Generally, in my experience, most functions and methods that use keyword arguments have a specific set of known or accepted keyword names, though this is not always the case. It's not hard to imagine a function that performs some operation against every member of a dict, and leveraging the fact that keyword arguments are a dict inside the function. In cases where a keyword-argument expects certain names/values, or may not expect them, but supplies a default value if they aren't provided, those would be facts worthy of note in the documentation. It's also not uncommon for the existence of one keyword to mandate the provision of another, though that's not a pattern that I see a lot of.

As a bare minimum, then, the documentation for keyword arguments would need to be able to generate output looking something like this:

--------------------------------------------------------------------------------
Fnord( self, arg1, arg2, *args, **kwargs ) [instancemethod]
Ook.Fnord (method) original doc-string

ARGUMENTS:

self .............. (instance, required): The object-instance that the method 
                    will bind to for execution.
[arguments and argument-list removed for brevity]
**kwargs .......... Ook.Fnord (method) keyword-arguments description. 
                    Known/accepted keyword values are:
  - keyword1 ...... (float, required): Ook.Fnord keyword1 description
  - keyword2 ...... (bool, defaults to False): Ook.Fnord keyword2 description
  - keyword3 ...... (int|long): Ook.Fnord keyword3 description.

The one thing this example output does not contain is any sort of keyword-dependency information. Assume, for the moment, that keyword3 is required if keyword2 is True. At first glance it doesn't sound too difficult to set up some process, maybe another decorator-method, perhaps like describe.arglistitem, was generated for argument-lists, that allows a developer to specify a relationship between two known keyword names. The number of possible relationships between two such keyword names is relatively small, at least if it's a simple condition like this keyword is required if that keyword is a given value. However, the number of conditions possible, and the number of relationships possible as more known keywords are added grows very quickly, and I'm not confident that I'd be able to identify even a useful fraction of the possible relationships between any two keyword names. Off the top of my head, and without getting into relationship-conditions that rely on specific values with more than a few simple options (True and False, for example, and maybe None), I can think of several in a few seconds:

• Another keyword exists;
• Another keyword doesn't exist;
• Some specific combination(s) of keywords exist;
• Some specific combination(s) of keywords doesn't exist;
• Another keyword has a specific value out of a very large set of possible values (e.g., an integer value, but only -1 is significant);
• Some specific combination(s) of keywords have one or more specific values, even if those values are a small set;

I suspect that the number of useful possibilities is huge. As a result, the number of conditions, variations, or whatever of decorators to handle those permutations would also be huge.

If the goal of such a keyword-item decorator was to result in documentation that looked like this:

ARGUMENTS:

self .............. (instance, required): The object-instance that the method 
                    will bind to for execution.
[arguments and argument-list removed for brevity]
**kwargs .......... Ook.Fnord (method) keyword-arguments description. 
                    Known/accepted keyword values are:
  - keyword1 ...... (float, required): Ook.Fnord keyword1 description
  - keyword2 ...... (bool, defaults to False): Ook.Fnord keyword2 description
  - keyword3 ...... (int|long): Ook.Fnord keyword3 description. Required if 
                    keyword2 is True.

The decoration involved is really nothing more than a specific description add-on — something that allows the developer to specify, as simple text, some sort of additional description that can then be guaranteed to exist after the normal description of an item.

I must admit that I'm on the fence about this prospect. On one hand, it does enforce structural consistency in documentation. On the other hand, it does so by using yet another decorator-call and complicating the underlying data-structure (slightly) when simply adhering to a certain amount of discipline in writing the description would take care of the need. Admittedly, the idea of simplifying, if not enforcing some common degree of discipline in generating documentation is what this entire exercise is all about. This feels like it might be going too far, though, perhaps for no good reason. Additionally, I cannot think of any real need for the additional descriptions to be distinct elements in the data-structure. And, as an afterthought, since it would be feasible for multiple relationships to exist for a single keyword-name, that implies that there would be, at a minimum, a list of relationships, which might well have a meaningful sequence when output. Yes, I have code already written for the describe.arglistitem decorator that could be adapted to handle the potential sequence-requirement, but then the trade-off is that each and every keyword-relationship would have to have a sequence set. That feels like way more work just in decorating functions and methods than I'd be willing to undertake, especially since I could just as easily just write those relationship-items in the keyword-item decoration.

I may revisit this idea later, but for now, I think I'll leave the relationship decorator idea alone. It just feels too complicated all around for what feels like no real gain.

So, what would the underlying metadata data-structure for keyword arguments look like? Something like this, I think:

{
    # arguments omitted for brevity
    # argument-list omitted for brevity
    'keywords':{
        'description':<str|unicode>,
        'keywords':{<dict <str|unicode>:<dict> >,
            # <str|unicode>,: {
            #     'defaultValue': <object>,
            #     'description': <str|unicode>,
            #     'expects': <tuple <type*> >,
            #     'hasDefault': <bool>,
            #     'name': <str|unicode>
            #     'required': <bool>,
            # },
        'name':<str|unicode>,
        }
    }
}

This structure is very similar to the structures defined for arguments and argument-list items. In fact, apart from the addition of the required member, it is identical to the structure of a normal argument's metadata. The required field in the dictionary is a necessary addition because there is no way to infer from the keyword-argument's specification itself whether it has any expected items, let alone whether those items are required. The hasDefault member, while it is common between the two structures, faces a similar issue in the context of a keyword-item decorator, for much the same reason as the required member: There's no way to ascertain what a default for a given keyword would be, since there's no way to determine that the keyword is known — it has to be specified.

In the interests of consistency, I'd like, if I can, to keep the keyword-item decorator looking as much like the existing decorators as I can manage. I expect the main keyword-argument decorator to be very straightforward: All it needs is a description, since its name can be gleaned from the function or method by inspection. The keyword-item decorator should look very much like athe existing argument decorator, since it will have many of the same elements and requirements. Even if there's nothing else, it just feels good to have the same sequence of arguments (if possible) for similar decoration efforts.

The fact that keyword-items can have either default values or required flags is, I think, the only thing that will potentially interfere with keeping that consistency. My initial thought was to simply add them in as additional standard arguments in the signature, looking something like this:

@classmethod
    def keyworditem( cls, name, description='No description provided', required=False, 
        default=None, *expects ):

There are at least two issues with taking this approach, though. The first is that required and default in this method are almost certainly going to be mutually exclusive. That is, if a keyword-item name has a default value, it cannot be required, since the function or method should be executable without supplying the required name. That means that whichever of the two arguments is first in the argument sequence must always be passed a value if the second is used. In the example above, any item that has a default must still be explicitly passed a required of False. That's do-able, and might even be manageable without being... horribly inelegant, maybe... by generating some sort of constant like describe.NOT_REQUIRED and passing that when necessary. But it just feels stupid to me to have to do that, and I don't like it.

The second flaw, and the more important of the two, I think, is that by setting up default as an argument with a default value, the method would be assuming that either all keyword-items have a default value (of None in this case), or that no keyword-item can have a default value of whatever the argument's default is (None again in this case). Again, setting up some sort of specific constant value that indicates that the default doesn't really exist would be viable as a solution, but it also feels stupid.

I arrived at what I think is a much better solution by asking myself why this wasn't a problem with the describe.argument decorator. The answer to that question is that in describe.argument, the existence of a default value could be ascertained by whether or not a default value actually existed. That is, as part of all of the other decoration methods created so far, a call is made to inspect.getargspec, and the resulting argument-specification is examined looking for the existence of a default value corresponding with each argument. If it doesn't exist, then there is no default value, not even None. There is a way to achieve a similar does/does-not exist state for arguments: Using a keyword-argument list:

@classmethod
def keyworditem( cls, name, description='No description provided', *expects, 
    **requiredOrDefault ):

This approach, I think, is as good as it's going to get:

• Keyword-items that have no default and are not required use the exact same argument-sequence as the argument decorator. It keeps decorator calls consistent (which I really like).
• There is never any requirement to specify either a required flag or default value as not existing, since if they aren't provided, they don't exist. It doesn't require stupid/inelegant argument structure (which is a Good Thing®™).
• If there ever arose an actual need to flag something as required, and provide it with a default value, as unlikely as I think that is, it's still a viable option.
• If there ever arose a need for some other documentation-item that I haven't thought of yet, there is at least a possibility that it could fit as just another keyword in the decorator's arguments.

The implementation-steps for keyword-argument decoration are pretty similar to those for argument-list decoration:

• The existing api_documentation class needs to be altered to set up default keyword-argument metadata structures;
• The describe.keywordargs method needs to be built;
• The describe.keyworditem method needs to be built;
• The __str__ method of the api_documentation class needs to be altered to output the argument-list metadata; and (since I'm documenting these decorators with themselves)
• The documentation-decorators need to be created for anything new that's being added to the mix.

The implementation of these methods was pretty quick and easy, since a fair portion followed a pattern similar to something already in place in earlier argument or arglist implementations. Starting with api_documentation:

class api_documentation( object ):

    # ...

    #####################################
    # Instance property-getter methods  #
    #####################################

    # ...

    def _GetKeywordArgs( self ):
        """
Gets the dictionary of the keyword-arguments that the instance has been used to 
document/describe:
'keywordargs': {
    'description':<str|unicode>,
    'keywords':{
        <str|unicode name>:
        {
            'defaultValue':<None|object>,
            'description':<str|unicode>,
            'expects':<tuple<types>>,
            'hasDefault':<bool>,
            'name':<str|unicode>,
            'required':<bool>,
        }
    },
    'name':<str|unicode>,
}"""
        return self._keywordArgs

    # ...

    #####################################
    # Instance Properties               #
    #####################################

    # ...

    keywordargs = property( _GetKeywordArgs, None, None, 
        _GetKeywordArgs.__doc__.strip() )

    # ...

    #####################################
    # Instance Methods                  #
    #####################################

    # ...


    def AddKeywordargs( self, callable, description ):
        """
Adds keyword-args-metadata for the specified argument to the instance's 
documentation-metadata."""
        if not self._keywordArgs:
            # If it doesn't exist, create it
            self._keywordArgs = self._createKeywordargsMetadata( 
                callable, description )
        else:
            # If it DOES exist, update it (partially)
            self._keywordArgs[ 'description' ] = description

    def AddKeyword( self, callable, name, description, *expects, **options ):
        """
Adds keyword-args-metadata for the specified argument to the instance's 
documentation-metadata."""
        if not self._keywordArgs:
            # If it doesn't exist, create it
            self._keywordArgs = self._createKeywordargsMetadata( callable )
        if self._keywordArgs[ 'keywords' ].get( name ):
            raise DescriptionError( 'The "%s" keyword has already been '
                'described for the %s %s' % ( name, callable.__name__, 
                    type( callable ).__name__ ) )
        metadata = {
            'defaultValue':options.get( 'default' ),
            'description':description,
            'expects':( object, ),
            'hasDefault':( 'default' in options.keys() ),
            'name':name,
            'required':False,
            }
        if expects:
            metadata[ 'expects' ] = expects
        if 'required' in options and 'default' in options:
            raise DescriptionError( 'The "%s" keyword cannot be both required '
                'and optional with a default value' % ( 
                    name, callable.__name__ ) )
        required = options.get( 'required' )
        if required != None and required in ( True, False ):
            metadata[ 'required' ] = required
        else:
            if required != None:
                raise TypeError( 'The "required" keyword expects a boolean '
                    '(True|False) value, but was passed "%s" (%s)' % ( 
                        required, type( required ).__name__ ) )
        self._keywordArgs[ 'keywords' ][ name ] = metadata

    # ...

    def _createKeywordargsMetadata( self, callable, 
        description='No description provided' ):
        """
Creates and returns a metadata-dictionary for a function or method keyword-
arguments list.

RETURNS
 - dict of metadata values

ARGUMENTS:
callable ......... (callable, required) The function or method whose arglist-
                   metadata is being created
description ...... (str|unicode, required) The description of the arglist.

RAISES:
 - TypeError:
   + If the supplied callable is not a function, method, or generator
 - ValueError:
   + If a value for an argument is invalid
"""
        # Type-check callable
        if not inspect.isfunction( callable ) and \
            not inspect.isgenerator( callable ) and \
            not inspect.ismethod( callable ):
            raise TypeError( '%s._createKeywordargsMetadata expects a '
                'callable (function, generator or method) in its "callable" '
                'argument, but was passed "%s" (%s)' % ( 
                    self.__name__, callable, 
                    type( callable ).__name__ ) )
        # Get the argspecs of the callable if necessary
        if not self._argSpecs:
            self._argSpecs = inspect.getargspec( callable )
        # Set up the default structure:
        results = {
            'name':self._argSpecs.keywords,
            'description':None,
            'keywords':{}
            }
        # Type- and value-check description. Any non-empty value is kosher, 
        # since a description should, in theory, be unrestricted.
        if type( description ) not in ( str, unicode ):
            raise TypeError( '%s._createKeywordargsMetadata expects a non-'
                'empty string or unicode text-value for its "description" '
                'argument, but was passed "%s" (%s)' % ( 
                    self.__name__, description, 
                    type( description ).__name__ ) )
        if not description.strip():
            raise ValueError( '%s._createArgumentMetadata expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( self.__name__, description, 
                    type( description ).__name__ ) )
        # Description checks out, so add it to the structure.
        results[ 'description' ] = description
        # Return the structure
        return results

There's not much here that's significantly different than previous implementations for other decorators, though the use and initial population of metadata values in AddKeyword is, perhaps, a more elegant means of generating the data-structure, since it captures almost everything in one chunk of execution.

The additional implementation items in describe also look very similar to previous decorator-implementations:

class describe( object ):
    """
Nominally-static class (not intended to be instantiated) that provides the 
actual functionality for generating documentation-metadata structures."""
    #####################################
    # Class attributes (and instance-   #
    # attribute default values)         #
    #####################################

    # ...

    #####################################
    # Class Methods                     #
    #####################################

    # ...

    @classmethod
    def keyword( cls, name, description='No description provided', 
        *expects, **options ):
        """
Decorates a function or method by attaching documentation-metadata about a 
keyword-argument list item in the metadata for the decorted item's keywords 
metadata."""
        # Type-check name it must be a string or unicode value
        if type( name ) not in ( str, unicode ):
            raise TypeError( 'describe.argument expects a non-empty string or '
                'unicode text-value containing an argument-name for the item '
                'being decorated, but was passed "%s" (%s)' % ( 
                    name, type( name ).__name__ ) )
        # Type- and value-check description
        if type( description ) not in ( str, unicode ):
            raise TypeError( '%s.argument expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( cls.__name__, description, 
                    type( description ).__name__ ) )
        if not description.strip():
            raise ValueError( '%s.argument expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( cls.__name__, description, 
                    type( description ).__name__ ) )
        def _keywordDecorator( decoratedItem ):
            # Make sure the decorated item has a _documentation attribute, and 
            # if it doesn't, create and attach one:
            try:
                _documentation = decoratedItem._documentation
            except AttributeError:
                decoratedItem.__dict__[ '_documentation' ] = api_documentation( 
                    decoratedItem )
                _documentation = decoratedItem._documentation
            # If we reach this point, then we should add the item to the 
            # metadata, but first we have to create the metadata entry...
            _documentation.AddKeyword( decoratedItem, name, description, 
                *expects, **options )
            # Return the decorated item!
            return decoratedItem
        return _keywordDecorator

    @classmethod
    def keywordargs( cls, description='No description provided' ):
        """
Decorates a function or method by attaching documentation-metadata about its 
keyword-argument list - a description of it."""
        # Type- and value-check description
        if type( description ) not in ( str, unicode ):
            raise TypeError( '%s.keywordargs expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( cls.__name__, description, 
                    type( description ).__name__ ) )
        if not description.strip():
            raise ValueError( '%s.keywordargs expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( cls.__name__, description, 
                    type( description ).__name__ ) )
        def _keywordargsDecorator( decoratedItem ):
            """
Performs the actual keyword-args decoration"""
            # Make sure the decorated item has a _documentation attribute, and 
            # if it doesn't, create and attach one:
            try:
                _documentation = decoratedItem._documentation
            except AttributeError:
                decoratedItem.__dict__[ '_documentation' ] = api_documentation( decoratedItem )
                _documentation = decoratedItem._documentation
            # If we reach this point, then we should add the item to the 
            # metadata, but first we have to create the metadata entry...
            _documentation.AddKeywordargs( decoratedItem, description )
            # Return the decorated item!
            return decoratedItem
        return _keywordargsDecorator

Throwing a few keyword-decorations onto the Ook test-class:

class Ook( object ):
    """
Test-class."""
    @describe.argument( 'arg1', 'Ook.Fnord (method) arg1 description', bool, None )
    @describe.argument( 'arg2', 'Ook.Fnord (method) arg2 description' )
    @describe.arglist( 'Ook.Fnord (method) arglist description', int, long, float )
    @describe.arglistitem( 0, 'arg1', 'Ook.Fnord.args[0] description', float )
    @describe.arglistitem( 1, 'arg2', 'Ook.Fnord.args[1] description', int, long )
    @describe.arglistitem( 2, 'arg3', 'Ook.Fnord.args[2] description', bool )
    @describe.arglistitem( -1, 'values', 'Ook.Fnord.args[3] (values) description', str, unicode )
    @describe.keywordargs( 'Ook.Fnord keyword-arguments list description' )
    @describe.keyword( 'keyword1', 'Ook.Fnord (method) "keyword1" description',
        int, long, float, 
        required=True )
    @describe.keyword( 'keyword2', 'Ook.Fnord (method) "keyword2" description',
        None, str, unicode, 
        default=None )
    @describe.keyword( 'keyword3', 'Ook.Fnord (method) "keyword3" description',
        None, str, unicode )
    def Fnord( self, arg1, arg2, *args, **kwargs ):
        """Ook.Fnord (method) original doc-string"""
        return None

The resulting documentation-output is:

--------------------------------------------------------------------------------
Fnord( self, arg1, arg2, *args, **kwargs ) [function]
Ook.Fnord (method) original doc-string

ARGUMENTS:

self .............. (instance, required): The object-instance that the method will bind to for execution.
arg1 .............. (bool|None, required): Ook.Fnord (method) arg1 description
arg2 .............. (any, required): Ook.Fnord (method) arg2 description
*args ............. (int|long|float): Ook.Fnord (method) arglist description
  - arg1 .......... (float): Ook.Fnord.args[0] description
  - arg2 .......... (int|long): Ook.Fnord.args[1] description
  - arg3 .......... (bool): Ook.Fnord.args[2] description
  - values ........ (str|unicode): Ook.Fnord.args[3] (values) description
**kwargs .......... Ook.Fnord keyword-arguments list description
  - keyword1 ...... (int|long|float, required): Ook.Fnord (method) "keyword1" description
  - keyword2 ...... (None|str|unicode, defaults to None): Ook.Fnord (method) "keyword2" description
  - keyword3 ...... (None|str|unicode): Ook.Fnord (method) "keyword3" description

And that is pretty much exactly what I wanted.

That takes care of all the decorators for function- and method-arguments that Python supports, but there are a few other items that I think are nearly as important. Several posts back, I noted a fairly detailed list of what I think needs to be present (if applicable) in every callable's documentation. In a somewhat cleaner form, that list is:

• A description of the method.
  Done, uses the normal doc-string that is returned as the __doc__ of an element;
• A description of each argument
  Done.
• A description of the argument-list (and any specific meaning of items in that sequence)
  Done.
• A description of the keyword-arguments (and any keywords that have specific meaning or use in that context)
  Done.
• What the callable returns, if anything (with None being the default if nothing is returned); and
• What exceptions might be raised, and the conditions under which they would be raised.

I'd like to have those last two in place before I call the callable documentation done. I'd also like to add a few new items to that list, I think:

ToDo

Used in the documentation to tag items that are going to be worked on, but that should represent little or no risk of introducing breaking changes when they are addressed.

FixMe

Used in the documentation to tag items that are in need of potentially significant re-work, that need attention because of inefficiencies, or that may otherwise be considered less-than-stable, but were part of the current version for some reason nevertheless.

Deprecated

Used in the documentation to tag items that are going to go away in the foreseeable future.

Ideally, a FixMe should never make its way into live code, but we don't live in an ideal world, so having some sort of indicator of functionality that is sketchy, likely to change, or otherwise not yet ready for prime time is probably a good idea. I'd rather have it and not need it than need it and not have it, at any rate. I've not yet worked on a project with a long enough lifespan to need a Deprecated in play either, but again, I'd rather have it and not need it than need it and not have it.

So, my plan is to pick up next time with either the three new items or the remaining two from the original list. I don't know which one I'm going to tackle just yet, though, so come back next time and see for yourself...