Documentation Decorators: Argument Lists

Apologies...

This post turned out to be a lot longer than I'd anticipated it would be when I started writing it. I struggled for a while with trying to find one or more good break-points, in order to split it into two or more smaller posts (easier to digest), but there wasn't anything that felt like a good spot, so this is long...

So, picking up where I left of with my last post, today I'm planning to tackle the documentation-metadata structure for argument lists. Like arguments, arglist documentation should have a description, and should allow but not require the specification of expected value-types. Since only one arglist is allowed on any callable in Python, there's no need to specify the name the way the describe.argument decorator does for arguments, but the name should still be stored as part of the metadata structure, if only for consistency. The data-structure should, then, be somewhat familiar, looking much like the structure for arguments, but would be stored as a separate metadata-item in the data-structure. By way of example, consider the Ook test-class defined last time, with describe.arglist decorators set up for its methods:

class Ook( object ):
    """
Test-class."""
    @describe.argument( 'arg1', 'Ook.Fnord (method) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Fnord (method) arg2 description' )
    @describe.arglist( 'Ook.Fnord (method) arglist description', str, unicode )
    def Fnord( self, arg1, arg2, *args, **kwargs ):
        """Ook.Fnord (method) original doc-string"""
        return None

    @classmethod
    @describe.argument( 'arg1', 'Ook.Bleep (classmethod) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Bleep (classmethod) arg2 description' )
    @describe.arglist( 'Ook.Bleep (classmethod) arglist description' )
    def Bleep( cls, arg1, arg2=None, *args, **kwargs ):
        """Ook.Bleep (classmethod) original doc-string"""
        return None

    @staticmethod
    @describe.argument( 'arg1', 'Ook.Flup (staticmethod) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Flup (staticmethod) arg2 description' )
    @describe.arglist( 'Ook.Flup (staticmethod) arglist description', str, unicode )
    def Flup( arg1, arg2, *args, **kwargs ):
        """Ook.Flup (staticmethod) original doc-string"""
        return None

In its most basic form, the metadata for the args arglist would be very simple. In context, with the metadata for the arguments, the data-structure would look something like this:

{
    'arg1': {
        'description': 'Ook.Fnord (method) arg1 description',
        'expects': ( <type 'int'>, <type 'long'>, <type 'float'>),
        'hasDefault': False,
        'name': 'arg1'
        },
    'arg2': {
        'description': 'Ook.Fnord (method) arg2 description',
        'expects': (<type 'object'>,),
        'hasDefault': False,
        'name': 'arg2'
        },
    'arglist': {
        'description': 'Ook.Fnord (method) arglist description',
        'expects': (<type 'str'>, <type 'unicode'>),
        'name': 'args'
        }
}

That would cover the most basic expected arglist usage pretty nicely. However, there are some additional wrinkles to consider. One is that the generally-accepted way to define functions that have variable but specifically-sequenced arguments is to use arglists and assign values based on the existence and position of the value in the argument-list sequence. That explanation got away from me, so let me provide an example. Say I want a function that might be documented as MyFunction( [spam, [eggs, [beans]] ). That is, MyFunction accepts any of the following:

• MyFunction()
• MyFunction( 1 )
• MyFunction( 1, 2 )
• MyFunction( 1, 2, 3 )

where the first value is spam, the second is eggs and the third beans. If any given value is provided, all of its predecessors in the function's signature must be provided as well. So beans require spam and eggs. That function might look something like this:

def MyFunction( *arguments ):
    """
An example of a function that can accept zero-to-many arguments with 
specific meanings based on their position in the argument-list."""
    print '#' + '-'*78 + '#'
    print 'MyFunction%s called' % ( str( arguments ) )
    # Get the first three items from the arguments list *plus* three 
    # None values so that zero-to-three values are populated (with 
    # None as a default if not provided)
    spam, eggs, beans = tuple( ( list( arguments[ 0:3 ] ) + [ None, None, None ] )[0:3] )
    if arguments:
        if spam:
            print 'spam .... %s' % spam
        if eggs:
            print 'eggs .... %s' % eggs
        if beans:
            print 'beans ... %s' % beans
    else:
        print 'No spam, eggs or beans specified'

If MyFunction is executed in any of several ways:

MyFunction()
MyFunction( 1 )
MyFunction( 1, 2 )
MyFunction( 1, 2, 3 )
MyFunction( 'spam', 'spam' )

then the results, I hope, make the idea more clear:

#------------------------------------------------------------------------------#
MyFunction() called
No spam, eggs or beans specified
#------------------------------------------------------------------------------#
MyFunction(1,) called
spam .... 1
#------------------------------------------------------------------------------#
MyFunction(1, 2) called
spam .... 1
eggs .... 2
#------------------------------------------------------------------------------#
MyFunction(1, 2, 3) called
spam .... 1
eggs .... 2
beans ... 3
#------------------------------------------------------------------------------#
MyFunction('spam', 'spam') called
spam .... spam
eggs .... spam

Given that it's possible (and even probable) that a function or method will be defined using a variable number of specific-purpose arguments, the question that should, hopefully, come to mind is What if those arguments have specific descriptions or type-expectations? The metadata structure shown so far would most emphatically not handle that case. It would require something with more detail, along the lines of:

{
    # arguments removed for brevity...
    'arglist': {
        'description': 'Ook.Fnord (method) arglist description',
        'expects': (<type 'str'>, <type 'unicode'>),
        'name': 'args',
        'sequence':[
                {
                    'name':'spam', 
                    'defaultValue':None,
                    'description':'The "spam" value to use', 
                    'expects':(<type 'str'>, )
                },
                {
                    'name':'eggs', 
                    'defaultValue':None,
                    'description':'The "eggs" value to use', 
                    'expects':(<type 'int'>, <type 'long'>)
                },
                {
                    'name':'beans', 
                    'defaultValue':None,
                    'description':'The "beans" value to use', 
                    'expects':(<type 'bool'>, None)
                },
            ]
        }
}

Ideally, the decoration that generates that structure would look something like this, then:

@describe.argument( 'arg1', 'Ook.Fnord (method) arg1 description', int, long )
@describe.argument( 'arg2', 'Ook.Fnord (method) arg2 description' )
@describe.arglist( 'Ook.Fnord (method) arglist description', str )
@describe.arglistitem( 0, 'spam', None, 'The "spam" value to use', str, unicode )
@describe.arglistitem( 1, 'eggs', None, 'The "eggs" value to use', int, long )
@describe.arglistitem( 2, 'beans', None, 'The "beans" value to use', bool, None )
def Fnord( self, arg1, arg2, *args, **kwargs ):
    """Ook.Fnord (method) original doc-string"""
    return None

The arguments for the describe.arglistitem decorator are:

1. A sequence value: The position of the sequence-item in the argument list's sequence. I'd rather not have to specify a sequence, frankly, but since there really is no way to enforce decorator-ordering (and they fire from last to first anyway, on top of that), I don't see any good way around that.
2. A name. The name is not as formal as an argument's name, since it's really just there for programmer reference;
3. A default value. There are some considerations about providing a default value that I want to discuss, so more on that in a bit;
4. A description; and
5. Zero-to-many expected types for the sequence-item. I expect this to behave pretty much exactly like the expects values in describe.argument.

The end goal for the documentation generated should, then, look something like this (as plain text):

--------------------------------------------------------------------------------
Fnord( self, arg1, arg2, *args, **kwargs ) [instancemethod]

Ook.Fnord (method) original doc-string

ARGUMENTS:

self .............. (instance, required): The object-instance that the method will bind to for execution.
arg1 .............. (int|long|float, required): Ook.Fnord (method) arg1 description
arg2 .............. (any, required): Ook.Fnord (method) arg2 description
args .............. (str, optional): Ook.Fnord (method) arglist description. Specific values accepted, in order, are:
  spam ............ (str|unicode, optional, defaults to None): The "spam" value to use
  eggs ............ (int|long, optional, defaults to None): The "eggs" value to use
  beans ........... (bool|None, optional, defaults to None): The "beans" value to use

--------------------------------------------------------------------------------

There are a few caveats and items of note that I think arise from this set-up. First and foremost is that if sequence-items are provided, they must, eventually, resolve to a contiguous list. That is, if the spam and beans items in the example above are specified, then the sequence-item between them (eggs) must also be specified somewhere along the line for the documentation to be accurate and meaningful. If eggs was not specified as an argument sequence item, and there was no generated display as a result, then the documentation would indicate (inaccurately) that spam and beans were the only meaningful items in the argument-sequence, and anyone relying on the documentation would not expect their beans values to appear as eggs. That would be a problem. To prevent that from becoming an issue, I'm planning to put a default metadata-structure in place in any blank spots in the sequence, probably as part of the process of creating or updating the sequence member of the metadata-structure.

Another consideration stems from the fact that a Python argument-list is, for all practical purposes, allowed to be arbitrarily long. Even if, as in this case, some number of arguments are documented as having specific meanings in the sequence, *args can have additional items after those. So, for example, a call like

Fnord( 'spam', 1, True, 'these', 'are', 'additional', 'values' )

is perfectly valid from the standpoint of the syntax of the language, even if it makes no sense in the context of the function. How should that kind of scenario be handled? Honestly, I don't know. In cases where the function is only expecting those first three (or however-many) sequenced items, I'd implement sequence-length checking and raise an error if too many arguments were supplied. But that wouldn't help in cases where there's a legitimate need for a potentially-infinite remaining set of sequence-items, and there are legitimate uses for that sort of structure. I have in mind a variant of a min or max function that supplies one or more cut-off values – something that is equivalent to give me the minimum/maximum value of all supplied values that is also greater than x and optionally less than y, though that case might be better handled with an optional standard argument for y. In any case, the potential for legitimate, continuous sequences of values after the ones that have specific meaning exists, and should be represented in the documentation somehow. My gut feeling is to allow a special sequence-number, -1, for example, that indicates that the documentation-structure applies to all remaining values. Something like this, maybe:

@describe.arglistitem( -1, 'values', None, 'The values to process', int, long )
def Fnord( self, arg1, arg2, *args, **kwargs ):
    # ...

Another wrinkle is that if there are specific meanings associated with each argument in the sequence and the sequence is limited, it's possible (probable, even, maybe?) that the normal type-specification of the argument-list is meaningless. That is, using the same Fnord function above, but constrained in the function so that it only accepts the spam, eggs, and beans items, the general (str, optional) description of the argument-list as a whole is unnecessary as well as being inaccurate. I'm almost certain that it would be possible to write code that would handle all of the various logically-legitimate variants of an argument-list as far as documentation-decoration were concerned, but it seems to me that doing so would complicate things more than I'd like — the premise of these decorators is Documentation Made Easy, after all, not Documentation Made Complicated. I think, in order to keep things easy, a reasonable balance between simplicity and hard-and-fast structure probably requires a rule-set about like this one:

• Argument-lists with no associated sequence simply use the baseline decorator, with results that look like a decorated argument:

  args .............. (str, optional): Ook.Fnord (method) arglist description.

• Argument-lists that specify one-to-many sequence items, but that do not have an all-remaining-items sequence-item specified do not show type-information in documentation except those associated with the sequence-items:

  args .............. Ook.Fnord (method) arglist description. Specific values accepted, in order, are:
    spam ............ (str|unicode, optional, defaults to None): The "spam" value to use
    eggs ............ (int|long, optional, defaults to None): The "eggs" value to use

• Argument-lists that specify one-to-many sequence-items and an all-remaining-items at the end of the sequence show the type-specification of the argument-list as a whole for those, remaining items, and use the all-remaining-items description. Using the example -1 describe.arglistitem decorator-call above, that would yield:

  args .............. Ook.Fnord (method) arglist description. Specific values accepted, in order, are:
    spam ............ (str|unicode, optional, defaults to None): The "spam" value to use
    eggs ............ (int|long, optional, defaults to None): The "eggs" value to use
    values* ......... (str, optional, defaults to None): The values to process.

The complete decoration for that last case would look like this, then:

@describe.argument( 'arg1', 'Ook.Fnord (method) arg1 description', int, long )
@describe.argument( 'arg2', 'Ook.Fnord (method) arg2 description' )
@describe.arglist( 'Ook.Fnord (method) arglist description' )
@describe.arglistitem( 0, 'spam', None, 'The "spam" value to use', str, unicode )
@describe.arglistitem( 1, 'eggs', None, 'The "eggs" value to use', int, long )
@describe.arglistitem( -1, 'values', None, 'The values to process', str )
def Fnord( self, arg1, arg2, *args, **kwargs ):
    """Ook.Fnord (method) original doc-string"""
    return None

So: Implementation time! There are a few things that need to be done:

• The existing api_documentation class needs to be altered to set up default argument-list metadata structures;
• The describe.arglist method needs to be built;
• The describe.arglistitem method needs to be built;
• The __str__ method of the api_documentation class needs to be altered to output the argument-list metadata;

The changes to api_documentation are pretty straightforward, given the pattern established with the various argument-specific functionality already in place:

class api_documentation( object ):
    """
Provides a common collection-point for all API documentation metadata for a 
programmatic element (class, function, method, whatever)."""

    # ...

    #####################################
    # Instance property-getter methods  #
    #####################################

    # ...

    def _GetArglist( self ):
        """
Gets the dictionary of the argument-list that the instance has been used to 
document/describe:
'arglist': {
    'description':<str|unicode>,
    'expects':<tuple<types>>,
    'name':<str|unicode>,
    'sequence':[
        {
            'name':<str|unicode>,
            'defaultValue':<any>,
            'description':<str|unicode>,
            'expects':<tuple<types>>,
        },
    ]
}"""
        return self._arglist

    # ...

    #####################################
    # Instance Properties               #
    #####################################

    # ...
    arglist = property( _GetArglist, None, None, 
        _GetArglist.__doc__.strip() )

    # ...

    #####################################
    # Instance Methods                  #
    #####################################

    # ...

    def _createArglistMetadata( self, callable, description, *expects ):
        """
Creates and returns a metadata-dictionary for a function or method arglist.

RETURNS
 - dict of metadata values

ARGUMENTS:
callable ......... (callable, required) The function or method whose arglist-
                   metadata is being created
description ...... (str|unicode, required) The description of the arglist.
expects .......... (tuple of types, optional, if not specified, stores 
                   ( object, ) as the value) The expected type(s) of the 
                   arglist.

RAISES:
 - TypeError:
   + If the supplied callable is not a function, method, or generator
   + If a type for an argument is invalid
 - ValueError:
   + If a value for an argument is invalid
"""
        # Type-check callable
        if not inspect.isfunction( callable ) and \
            not inspect.isgenerator( callable ) and \
            not inspect.ismethod( callable ):
            raise TypeError( '%s._createArglistMetadata expects a callable '
                '(function, generator or method) in its "callable" argument, '
                'but was passed "%s" (%s)' % ( self.__name__, callable, 
                    type( callable ).__name__ ) )
        # Get the argspecs of the callable if necessary
        if not self._argSpecs:
            self._argSpecs = inspect.getargspec( callable )
        # Set up the default structure:
        results = {
            'name':self._argSpecs.varargs,
            'description':None,
            'expects':( object, ),
            'sequence':[]
            }
        # Type- and value-check description. Any non-empty value is kosher, 
        # since a description should, in theory, be unrestricted.
        if type( description ) not in ( str, unicode ):
            raise TypeError( '%s._createArgumentMetadata expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( self.__name__, description, 
                    type( description ).__name__ ) )
        if not description.strip():
            raise ValueError( '%s._createArgumentMetadata expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( self.__name__, description, 
                    type( description ).__name__ ) )
        # Description checks out, so add it to the structure.
        results[ 'description' ] = description
        if expects:
            results[ 'expects' ] = expects
        # Return the structure
        return results

    # ...

    def AddArglist( self, callable, description, *expects ):
        """
Adds arglist-metadata for the specified argument to the instance's 
documentation-metadata."""
        self._arglist = self._createArglistMetadata( 
            callable, description, *expects )

As I was working through this chunk of changes, it occurred to me that I've never actually shown the __str__ method of the api_documentation class, which is what is generating the output that I have shown for documentation thus far. I'm going to hold off on going into that in any great detail for a while, though, because while it's functional at present, and sufficient for demonstrating that the metadata-generation is working, it's pretty clunky code and it doesn't generate what I really want it to, ultimately. It's missing hanging indentation and variable-length dot-leaders that I'd like to have in pplace to keep printed documentation well-formatted and with a maximum line-length of 80 characters. I'll plan to hit that after all of the documentation-generation for arguments is complete.

In the meantime, looking at this last chunk of code: The AddArglist method will have to be revisited once the describe.arglistitem decorator is implemented. While it works fine with our Ook test-class (above) so far after adding metadata-decoration for the methods' argument-lists, once argument-list items are added into the mix, it'll overwrite an existing argument-list metadata structure that may well have valid documentation-metadata if the sequence of decorators isn't very specific, and I don't want that.

The decoration on the Ook class' methods, for proving that the decoration works, looks like this now:

class Ook( object ):
    """
Test-class."""
    @describe.argument( 'arg1', 'Ook.Fnord (method) arg1 description', bool, None )
    @describe.argument( 'arg2', 'Ook.Fnord (method) arg2 description' )
    @describe.arglist( 'Ook.Fnord (method) arglist description', int, long, float )
    def Fnord( self, arg1, arg2, *args, **kwargs ):
        """Ook.Fnord (method) original doc-string"""
        return None

    @classmethod
    @describe.argument( 'arg1', 'Ook.Bleep (classmethod) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Bleep (classmethod) arg2 description' )
    @describe.arglist( 'Ook.Bleep (classmethod) arglist description' )
    def Bleep( cls, arg1, arg2=None, *args, **kwargs ):
        """Ook.Bleep (classmethod) original doc-string"""
        return None

    @staticmethod
    @describe.argument( 'arg1', 'Ook.Flup (staticmethod) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Flup (staticmethod) arg2 description' )
    @describe.arglist( 'Ook.Flup (staticmethod) arglist description' )
    def Flup( arg1, arg2, *args, **kwargs ):
        """Ook.Flup (staticmethod) original doc-string"""
        return None

And the related decorator method, describe.arglist, is:

    @classmethod
    def arglist( cls, description='No description provided', *expects ):
        """
Decorates a function or method by attaching documentation-metadata about its 
argument-list, including a description of the argument-list, and an optional 
type-specification set."""
        # Type- and value-check description
        if type( description ) not in ( str, unicode ):
            raise TypeError( '%s.argument expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( cls.__name__, description, 
                    type( description ).__name__ ) )
        if not description.strip():
            raise ValueError( '%s.argument expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( cls.__name__, description, 
                    type( description ).__name__ ) )
        # Type- and value-check expects, if it exists, or set it to ( object, )
        def _arglistDecorator( decoratedItem ):
            """
Performs the actual arglist decoration"""
            # Make sure the decorated item has a _documentation attribute, and 
            # if it doesn't, create and attach one:
            try:
                _documentation = decoratedItem._documentation
            except AttributeError:
                decoratedItem.__dict__[ '_documentation' ] = api_documentation( decoratedItem )
                _documentation = decoratedItem._documentation
            # If we reach this point, then we should add the item to the 
            # metadata, but first we have to create the metadata entry...
            _documentation.AddArglist( decoratedItem, description, *expects )
            # Return the decorated item!
            return decoratedItem
        return _arglistDecorator

This yields the following documentation on Ook, when printed:

--------------------------------------------------------------------------------
Fnord( self, arg1, arg2, *args, **kwargs ) [instancemethod]
Ook.Fnord (method) original doc-string

ARGUMENTS:

self .............. (instance, required): The object-instance that the method will bind to for execution.
arg1 .............. (bool|None, required): Ook.Fnord (method) arg1 description
arg2 .............. (any, required): Ook.Fnord (method) arg2 description
args .............. (int|long|float): Ook.Fnord (method) arglist description

--------------------------------------------------------------------------------
Bleep( cls, arg1, arg2, *args, **kwargs ) [instancemethod]
Ook.Bleep (classmethod) original doc-string

ARGUMENTS:

cls ............... (class, required): The class that the method will bind to for execution.
arg1 .............. (int|long|float, required): Ook.Bleep (classmethod) arg1 description
arg2 .............. (any, optional, defaults to None): Ook.Bleep (classmethod) arg2 description
args .............. (any): Ook.Bleep (classmethod) arglist description

--------------------------------------------------------------------------------
Flup( arg1, arg2, *args, **kwargs ) [instancemethod]
Ook.Flup (staticmethod) original doc-string

ARGUMENTS:

arg1 .............. (int|long|float, required): Ook.Flup (staticmethod) arg1 description
arg2 .............. (any, required): Ook.Flup (staticmethod) arg2 description
args .............. (any): Ook.Flup (staticmethod) arglist description

--------------------------------------------------------------------------------

I mentioned earlier that the logic in the AddArglist method of api_documentation would need to be altered, and I hinted at why that would be necessary. Here's a more detailed breakdown of that issue. Consider the following alterations to Ook.Fnord in the test-class:

class Ook( object ):
    """
Test-class."""
    @describe.argument( 'arg1', 'Ook.Fnord (method) arg1 description', bool, None )
    @describe.argument( 'arg2', 'Ook.Fnord (method) arg2 description' )
    @describe.arglist( 'Ook.Fnord (method) arglist description', int, long, float )
    @describe.arglistitem( 0, 'arg1', 'Ook.Fnord.args[0] description', float )
    @describe.arglistitem( 1, 'arg2', 'Ook.Fnord.args[1] description', int, long )
    @describe.arglistitem( 2, 'arg3', 'Ook.Fnord.args[2] description', str, unicode )
    @describe.arglistitem( -1, 'values', 'Ook.Fnord.args[3] (values) description', str, unicode )
    def Fnord( self, arg1, arg2, *args, **kwargs ):
        """Ook.Fnord (method) original doc-string"""
        return None

The decorator-sequence, I hope, makes good, solid sense to a developer. Document the argument list, then document the expected items in that list, in the order they are going to appear in a call. The documentation decorators also include an all remaining values item, as discussed before. The problem is that while the describe decorator-methods are called in the order specified, the underlying real decorator methods are executed in the opposite order. After dropping some print statements into a bare-bones/stub arglistitem method and the current version of arglist, this becomes pretty apparent:

 - Calling describe.arglist( Ook.Fnord (method) arglist description, (<type 'int'>, <type 'long'>, <type 'float'>) )
 - Calling describe.arglistitem( 0, arg1, Ook.Fnord.args[0] description, (<type 'float'>,) )
 - Calling describe.arglistitem( 1, arg2, Ook.Fnord.args[1] description, (<type 'int'>, <type 'long'>) )
 - Calling describe.arglistitem( 2, arg3, Ook.Fnord.args[2] description, (<type 'str'>, <type 'unicode'>) )
 - Calling describe.arglistitem( -1, values, Ook.Fnord.args[3] (values) description, (<type 'str'>, <type 'unicode'>) )
   + Calling _arglistitemDecorator( <function Fnord at 0x7f3a58c0ec08> )
     * sequence ...... -1
     * name .......... values
     * description ... Ook.Fnord.args[3] (values) description
     * expects ....... (<type 'str'>, <type 'unicode'>)
   + Calling _arglistitemDecorator( <function Fnord at 0x7f3a58c0ec08> )
     * sequence ...... 2
     * name .......... arg3
     * description ... Ook.Fnord.args[2] description
     * expects ....... (<type 'str'>, <type 'unicode'>)
   + Calling _arglistitemDecorator( <function Fnord at 0x7f3a58c0ec08> )
     * sequence ...... 1
     * name .......... arg2
     * description ... Ook.Fnord.args[1] description
     * expects ....... (<type 'int'>, <type 'long'>)
   + Calling _arglistitemDecorator( <function Fnord at 0x7f3a58c0ec08> )
     * sequence ...... 0
     * name .......... arg1
     * description ... Ook.Fnord.args[0] description
     * expects ....... (<type 'float'>,)
   + Calling _arglistDecorator( <function Fnord at 0x7f3a58c0ec08> )
     * description ... Ook.Fnord (method) arglist description
     * expects ....... (<type 'int'>, <type 'long'>, <type 'float'>)

Since the arglistitem decorators execute before the arglist decorator that they relate to, the method needs to establish the underlying bare-minimum argument-list metadata structure in order to be able to do what it needs to do. Following the pattern that's been established thus far, that means that AddArglistItem needs to create it:

    def AddArglistItem( self, callable, description, *expects ):
        """
Adds arglist-metadata for the specified argument to the instance's 
documentation-metadata."""
        if not self._arglist:
            self._arglist = self._createArglistMetadata( 
                callable )

That also means that AddArglist needs to check for an existing metadata-structure, and modify it if it already exists, rather than just creating a new one every single time:

    def AddArglist( self, callable, description, *expects ):
        """
Adds arglist-metadata for the specified argument to the instance's 
documentation-metadata."""
        if not self._arglist:
            # If it doesn't exist, create it
            self._arglist = self._createArglistMetadata( 
                callable, description, *expects )
        else:
            # If it DOES exist, update it (partially)
            self._arglist[ 'description' ] = description
            self._arglist[ 'expects' ] = expects

That should take care of the potential collisions, and yields documentation results for Ook.Fnord, Ook.Bleep and Ook.Flup methods that are identical to those before the modification, even with the arglistitem decorations in place.

It occurs to me that arglistitem documentation would probably be at least slightly easier to implement if the sequence were only a list of non-final argument-list items, rather than allowing (or requiring) a final list-item to be tracked in the same list-structure. The process of building the list of argument-list items would not have to contend with a length that fluctuates based on more than one condition in that case. The decoration-call wouldn't need to change at all, still using -1 as an indicator-value for the final items in the argument-list, just the handling of that indicator during decoration. That would require a minor change to the data-structure, though, looking something like so:

{
    # arguments removed for brevity...
    'arglist': {
        'description':<str|unicode>,
        'expects':<tuple<types>>,
        'name':<str|unicode>,
        'sequence':[
            {
                'name':<str|unicode>,
                'defaultValue':<any>,
                'description':<str|unicode>,
                'expects':<tuple<types>>,
            },
            # ...
        ],
        'final': {
            'name':<str|unicode>,
            'defaultValue':<any>,
            'description':<str|unicode>,
            'expects':<tuple<types>>,
        }
    }
}

Structurally, a final item is identical to any one of the sequence items, so the process for generating the metadata should be easily adaptable, if not actually identical across the board.

Another possibility that occurs to me is that maybe the sequence items (possibly including the final item) could be stored in the metadata-structure not as a list, but as a dictionary. The advantage to that approach is that there would be less need, perhaps no need at all, for rebuilding the sequence items list every time a new item is added. Looking at the Ook.Fnord method with the sequence and final items above, and the processing-sequence that occurred, the steps that would need to happen during execution are (in the order they were reported above):

• _arglistitemDecorator( -1, values, ... is called:
  arglist: does not exist, so create it with minimal data;
  final: does not exist, so create it and populate it;
  sequence: does not exist, but isn't being populated, so leave it be;
• _arglistitemDecorator( 2, arg3, ... is called:
  arglist: exists, use as-is;
  final: exists, but isn't being populated, so leave it be;
  sequence: does not exist, so create it and add the item to it;
• _arglistitemDecorator( 1, arg2, ... is called:
  arglist: exists, use as-is;
  final: exists, but isn't being populated, so leave it be;
  sequence: exists, so add the item to it;
• _arglistitemDecorator( 0, arg1, ... is called:
  arglist: exists, use as-is;
  final: exists, but isn't being populated, so leave it be;
  sequence: exists, so add the item to it;
• _arglistDecorator( ... is called:
  arglist: exists, populate with description and expects as needed;
  final: exists, but isn't being populated, so leave it be;
  sequence: exists, but isn't being populated, so leave it be;

If sequence is a list, then each time that it is being touched, it would, at least potentially, need to be analyzed and rebuilt from its current state with the addition of the new item. The same sequence of events, then, from the perspective of the sequence list, would look like so:

• _arglistitemDecorator( -1, values, ... is called:
  sequence: does not exist, but isn't being populated, so leave it be;
  Nothing to do here
• _arglistitemDecorator( 2, arg3, ... is called:
  sequence: does not exist, so create it and add the item to it;
  At this point, we know that there will eventually be three items [0..2], so we should create a list 3 elements in length, and insert the new element at index 2;
• _arglistitemDecorator( 1, arg2, ... is called:
  sequence: exists, so add the item to it;
  At this point, we know that the list exists, and that the index of the item is within its range of indexed values, so we can just insert it.
• _arglistitemDecorator( 0, arg1, ... is called:
  sequence: exists, so add the item to it;
  At this point, we know that the list exists, and that the index of the item is within its range of indexed values, so we can just insert it.
• _arglistDecorator( ... is called:
  sequence: exists, but isn't being populated, so leave it be;
  Nothing to do here

If, however, the decorator sequence is out of kilter, so that the sequence of sequences is not in order (say, 1, 0, 2), each call has to check and reconcile the current sequence against the incoming index:

• _arglistitemDecorator( -1, values, ... is called:
  Nothing to do here
• _arglistitemDecorator( 1, arg2, ... is called:
  sequence: exists, so add the item to it;
  At this point, we know that there will eventually be two items [0..1], so we should create a list 2 elements in length, and insert the new element at index 1;
• _arglistitemDecorator( 0, arg1, ... is called:
  sequence: exists, so add the item to it;
  At this point, we know that the list exists, and that the index of the item is within its range of indexed values, so we can just insert it.
• _arglistitemDecorator( 2, arg3, ... is called:
  sequence: does not exist, so create it and add the item to it;
  At this point, we know that there will eventually be three items [0..2], so we should create a list 3 elements in length, copy the existing sequence into the new sequence, and insert the new element appropriately.
• _arglistDecorator( ... is called:
  sequence: exists, but isn't being populated, so leave it be;
  Nothing to do here

It would get even more complicated to implement to allow for gaps in the provided decorator-sequence values. Imagine the process-steps for (2, 0, 1, 3, 5, 4) for example: The first three items in the sequence would be fine, then sequence-items 3, 5 and 4 would each require the creation of a new list to replace the old one, population with old values and insertion/replacement of the new missing value at the appropriate index-position.

That feels messy and brittle to me, but it's worth confirming or disproving that feeling, so I'm going to write some code that uses that basic process, and feed it some simple values using the sequence I had concerns about (2, 0, 1, 3, 5, 4):

def addToList( theList, theIndex, theItem ):
    # First, check the index to see if the current list is long enough
    if len( theList ) -1 < theIndex:
        # The current list is too short to accommodate the supplied 
        # index-value, so create a new list that's long enough and 
        # populate it with the elements of the original list.
        theList = [ 
            theList[ index ]
            if len( theList ) > index and theList[ index ]
            else None
            for index in range( 0, theIndex + 1 )
        ]
        # THEN set the appropriate index-location to the value
        theList[ theIndex ] = theItem
    else:
        # Otherwise, just set the appropriate index-location to the 
        # value
        theList[ theIndex ] = theItem
    # Return the list, since we cannot pass it by reference and modify 
    # the original list in place.
    return theList

myList = []

print '#' + '-'*78 + '#'
print myList
myList = addToList( myList, 2, 'two' )

print '#' + '-'*78 + '#'
print myList
myList = addToList( myList, 0, 'zero' )

print '#' + '-'*78 + '#'
print myList
myList = addToList( myList, 1, 'one' )

print '#' + '-'*78 + '#'
print myList
myList = addToList( myList, 3, 'three' )

print '#' + '-'*78 + '#'
print myList
myList = addToList( myList, 5, 'five' )

print '#' + '-'*78 + '#'
print myList
myList = addToList( myList, 4, 'four' )

print '#' + '-'*78 + '#'
print myList

I'll admit that I'm surprised that it's not as bad as I'd expected. This code may not be as optimal as it could be (though I was able to use a conditional list-comprehension to generate the new list, and that feels pretty good). I haven't run across anything in the standard Python libraries (yet) that does exactly what's needed: simultaneously growing and populating a list where the sequence is important. And it actually works, pretty much first try out of the gate, which I'm very happy about:

#------------------------------------------------------------------------------#
[]
#------------------------------------------------------------------------------#
[None, None, 'two']
#------------------------------------------------------------------------------#
['zero', None, 'two']
#------------------------------------------------------------------------------#
['zero', 'one', 'two']
#------------------------------------------------------------------------------#
['zero', 'one', 'two', 'three']
#------------------------------------------------------------------------------#
['zero', 'one', 'two', 'three', None, 'five']
#------------------------------------------------------------------------------#
['zero', 'one', 'two', 'three', 'four', 'five']

Using this proof-of-concept function as a baseline, and dropping default undefined argument-list items into undefined positions in the sequence (instead of None as used in the example) actually feels pretty good, the more I think on it.

With all of these thought out, the relevant implementations are:

class api_documentation( object ):
    """
Provides a common collection-point for all API documentation metadata for a 
programmatic element (class, function, method, whatever)."""
    #####################################
    # Class attributes (and instance-   #
    # attribute default values)         #
    #####################################

    # ...

    #####################################
    # Instance Methods                  #
    #####################################

    # ...

    def AddArglistItem( self, callable, sequence, name, description, *expects ):
        """
Adds arglist-metadata for the specified argument to the instance's 
documentation-metadata."""
        if not self._arglist:
            self._arglist = self._createArglistMetadata( 
                callable )
        # First, check the index to see if the current list is long enough
        if sequence >= 0:
            if len( self._arglist[ 'sequence' ] ) -1 < sequence:
                # The current list is too short to accommodate the supplied 
                # index-value, so create a new list that's long enough and 
                # populate it with the elements of the original list.
                self._arglist[ 'sequence' ] = [ 
                    self._arglist[ 'sequence' ][ index ]
                    if len( self._arglist[ 'sequence' ] ) > index and self._arglist[ 'sequence' ][ index ]
                    else {
                        'name':'unnamed',
                        'description':'No description provided',
                        'expects':( object, ),
                    }
                    for index in range( 0, sequence + 1 )
                ]
                # THEN set the appropriate index-location to the value
                self._arglist[ 'sequence' ][ sequence ] = {
                    'name':name,
                    'description':description,
                }
                if expects:
                    self._arglist[ 'sequence' ][ sequence ][ 'expects' ] = expects
                else:
                    self._arglist[ 'sequence' ][ sequence ][ 'expects' ] = ( object, )
            else:
                # Otherwise, just set the appropriate index-location to the 
                # value
                self._arglist[ 'sequence' ][ sequence ] = {
                    'name':name,
                    'description':description,
                }
                if expects:
                    self._arglist[ 'sequence' ][ sequence ][ 'expects' ] = expects
                else:
                    self._arglist[ 'sequence' ][ sequence ][ 'expects' ] = ( object, )
        elif sequence == -1:
                self._arglist[ 'final' ] = {
                    'name':name,
                    'description':description,
                }
                if expects:
                    self._arglist[ 'final' ][ 'expects' ] = expects
                else:
                    self._arglist[ 'final' ][ 'expects' ] = ( object, )
        else:
            raise RuntimeError( 'AddArglistItem expects a sequence value of -1 or greater.' )

class describe( object ):
    """
Nominally-static class (not intended to be instantiated) that provides the 
actual functionality for generating documentation-metadata structures."""

    # ...

    @classmethod
    def arglistitem( cls, sequence, name, description='No description provided', *expects ):
        """
Decorates a function or method by attaching documentation-metadata about an 
argument-list item at a specific position/index in the sequence of values 
expected/accepted by the argument-list."""
        # Type-check sequence it must be a long or int value
        if type( sequence ) not in ( int, long ):
            raise TypeError( 'describe.argument expects an integer or long '
                'numeric value containing an index-location for the item '
                'being decorated, greater than or equal to -1, but was passed '
                '"%s" (%s)' % ( sequence, type( sequence ).__name__ ) )
        # Value-check sequence it must be >= -1
        if sequence < -1:
            raise ValueError( 'describe.argument expects an integer or long '
                'numeric value containing an index-location for the item '
                'being decorated, greater than or equal to -1, but was passed '
                '"%s" (%s)' % ( sequence, type( sequence ).__name__ ) )
        # Type-check name it must be a string or unicode value
        if type( name ) not in ( str, unicode ):
            raise TypeError( 'describe.argument expects a non-empty string or '
                'unicode text-value containing an argument-name for the item '
                'being decorated, but was passed "%s" (%s)' % ( 
                    name, type( name ).__name__ ) )
        # Type- and value-check description
        if type( description ) not in ( str, unicode ):
            raise TypeError( '%s.argument expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( cls.__name__, description, 
                    type( description ).__name__ ) )
        if not description.strip():
            raise ValueError( '%s.argument expects a non-empty '
                'string or unicode text-value for its "description" argument, '
                'but was passed "%s" (%s)' % ( cls.__name__, description, 
                    type( description ).__name__ ) )
        def _arglistitemDecorator( decoratedItem ):
            # Make sure the decorated item has a _documentation attribute, and 
            # if it doesn't, create and attach one:
            try:
                _documentation = decoratedItem._documentation
            except AttributeError:
                decoratedItem.__dict__[ '_documentation' ] = api_documentation( decoratedItem )
                _documentation = decoratedItem._documentation
            # If we reach this point, then we should add the item to the 
            # metadata, but first we have to create the metadata entry...
            _documentation.AddArglistItem( decoratedItem, sequence, name, description, *expects )
            # Return the decorated item!
            return decoratedItem
        return _arglistitemDecorator

So, with the updated Ook class:

class Ook( object ):
    """
Test-class."""
    @describe.argument( 'arg1', 'Ook.Fnord (method) arg1 description', bool, None )
    @describe.argument( 'arg2', 'Ook.Fnord (method) arg2 description' )
    @describe.arglist( 'Ook.Fnord (method) arglist description', int, long, float )
    @describe.arglistitem( 0, 'arg1', 'Ook.Fnord.args[0] description', float )
    @describe.arglistitem( 1, 'arg2', 'Ook.Fnord.args[1] description', int, long )
    @describe.arglistitem( 2, 'arg3', 'Ook.Fnord.args[2] description', bool )
    @describe.arglistitem( -1, 'values', 'Ook.Fnord.args[3] (values) description', str, unicode )
    def Fnord( self, arg1, arg2, *args, **kwargs ):
        """Ook.Fnord (method) original doc-string"""
        return None

    @classmethod
    @describe.argument( 'arg1', 'Ook.Bleep (classmethod) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Bleep (classmethod) arg2 description' )
    @describe.arglist( 'Ook.Bleep (classmethod) arglist description' )
    def Bleep( cls, arg1, arg2=None, *args, **kwargs ):
        """Ook.Bleep (classmethod) original doc-string"""
        return None

    @staticmethod
    @describe.argument( 'arg1', 'Ook.Flup (staticmethod) arg1 description', int, long, float )
    @describe.argument( 'arg2', 'Ook.Flup (staticmethod) arg2 description' )
    @describe.arglist( 'Ook.Flup (staticmethod) arglist description' )
    def Flup( arg1, arg2, *args, **kwargs ):
        """Ook.Flup (staticmethod) original doc-string"""
        return None

the resulting documentation output is:

--------------------------------------------------------------------------------
Fnord( self, arg1, arg2, *args, **kwargs ) [instancemethod]
Ook.Fnord (method) original doc-string

ARGUMENTS:

self .............. (instance, required): The object-instance that the method 
                    will bind to for execution.
arg1 .............. (bool|None, required): Ook.Fnord (method) arg1 description
arg2 .............. (any, required): Ook.Fnord (method) arg2 description
*args ............. (int|long|float): Ook.Fnord (method) arglist description
  - arg1 .......... (float): Ook.Fnord.args[0] description
  - arg2 .......... (int|long): Ook.Fnord.args[1] description
  - arg3 .......... (bool): Ook.Fnord.args[2] description
  - values ........ (str|unicode): Ook.Fnord.args[3] (values) description

--------------------------------------------------------------------------------
Bleep( cls, arg1, arg2, *args, **kwargs ) [instancemethod]
Ook.Bleep (classmethod) original doc-string

ARGUMENTS:

cls ............... (class, required): The class that the method will bind to 
                    for execution.
arg1 .............. (int|long|float, required): Ook.Bleep (classmethod) arg1 
                    description
arg2 .............. (any, optional, defaults to None): Ook.Bleep (classmethod) 
                    arg2 description
*args ............. (any): Ook.Bleep (classmethod) arglist description

--------------------------------------------------------------------------------
Flup( arg1, arg2, *args, **kwargs ) [instancemethod]
Ook.Flup (staticmethod) original doc-string

ARGUMENTS:

arg1 .............. (int|long|float, required): Ook.Flup (staticmethod) arg1 
                    description
arg2 .............. (any, required): Ook.Flup (staticmethod) arg2 description
*args ............. (any): Ook.Flup (staticmethod) arglist description

--------------------------------------------------------------------------------

Argument-lists and their related (child?) items are probably the most complicated item to document. Certainly they involved a substantial volume of code, as well as the discussion of the implementation needs and the strategies and tactics that arose from them. The next (and final) argument-documentation item is keyword arguments, which I expect to be less complex, and to follow a pattern similar to the argument-list items covered in this post.

So come back next time and I'll start digging into that... Oh, wait. I said earlier that I'd discuss the odd decoration-structure that I used in decorating the decorators themselves, didn't I? OK, then: As things stand right now, the decoration of all the decorators written so far looks like this:

######################################################
# Because the actual decorators themselves aren't    # 
# available to decorate themselves until this point, #
# decorate the decorators. O.o :-)                   #
######################################################
# describe.arglist
describe.argument( 'description', 
    'The description of the argument-list', 
    str, unicode )( describe.arglist )
describe.arglist( 'The types expected by the argument-list.', 
    type )( describe.arglist )

# describe.arglistitem
describe.argument( 'sequence', 
    'The sequence of the named item in the argument-list (zero-index)', 
    int, long )( describe.arglistitem )
describe.argument( 'name', 
    'The name of the argument-list item', 
    str, unicode )( describe.arglistitem )
describe.argument( 'description', 
    'The description of the argument-list item', 
    str, unicode )( describe.arglistitem )
describe.arglist( 'The types expected by the argument-list sequence-item.', 
    type )( describe.arglistitem )

# describe.argument
describe.argument( 'name', 
    'The name of the argument', 
    str, unicode )( describe.argument )
describe.argument( 'description', 
    'The description of the argument', 
    str, unicode )( describe.argument )
describe.arglist( 'The types expected in the argument.', 
    type )( describe.argument )

Remember that the decorator methods themselves return a function. In normal" usage, where a @decoratorName( args )-style is set up before a decorated item, Python knows to apply the resulting function to the next item. The automatic chaining of those decorator calls ultimately applies every decorator to the final decorated item — a function or method in all the cases so far. But those functions in need of decoration don't actually exist at the point where a normal decorator-call would be placed to decorate them. That is, looking at arglist, for example, the argument decorator hasn't even been defined by the point where it would be applied as a normal decorator for arglist. That, then, means that we have to wait until all of the decorators' definitions are complete, and their functionality ready to use, before we can decorate the decorators themselves.

Ultimately, that's all the free-standing decoration is doing. Stepping through what happens during the execution of describe.argument( ... ) ( describe.arglist ):

1. describe.argument is called, returning a function;
2. that function is then called with ( describe.arglist ), which passes describe.arglist as the decoratedItem;
3. The decoration happens, returning the now-decorated describe.arglist

It's really that simple. Admittedly, it looks kind of strange, but that's all that's happening under the hood.

OK, now I'll sign off...