Showing posts with label download. Show all posts
Showing posts with label download. Show all posts

Tuesday, May 9, 2017

Generating and Parsing Markup in Python [6]

Though there are fewer methods in the Tag class than properties, there are still a good number of them that are, in my opinion, worth taking a detailed look at, so I'll just dive right in.

Method Implementations

There are three groups of methods that have a common theme between them, and that share a fair amount of logic as a result:

  • Methods relating to manipulating a Tag's childNodes (variations of adding children to and removing children from a Tag, ultimately);
  • Methods for getting various logical groupings or sets of child Tags from a Tag instance; and
  • Methods for working with a the attributes of a Tag
The majority of the other methods of Tag are either already implemented in BaseNode, are very simple implementations (in my opinion), or need to be deferred until some other class in the markup module is implemented.

Methods Relating to Manipulating Child Nodes

There are five different methods that add child nodes to a Tag instance in some fashion, three that involve removing a child, and two that relate to replacing a child (methods that I'm adding are indicated like this):

appendChild:
Adds a child to the instance's childNodes at the end of the collection
insertAfter:
Adds a child to the instance's childNodes after the position of a specified existing child
insertBefore:
Adds a child to the instance's childNodes before the position of a specified existing child
insertChildAt:
Adds a child to the instance's childNodes at a specified index/position in the colelction
prependChild:
Adds a child to the instance's childNodes at the beginning of the collection
removeChild:
Removes the specified child from the childNodes collection
removeChildAt:
Removes the child at a specified index/position from the childNodes collection
removeSelf:
Removes the instance from its parent's childNodes collection
replaceChild:
Replaces the specified child in the instance's childNodes collection with a new child
replaceChildAt:
Replaces the child at a specified index/position in the instance's childNodes collection with a new child
Experience with a previous incarnation of this module (the one I am re-creating from the ground up) led me to the conclusion that any method that alters a Tag's childNodes should return a relevant node-value. That is, for example:
# Creating a <table> to display values in a list of dicts:
listOfDicts = [
    { 'name':'row1 - name', 'value':'row1 - value' },
    { 'name':'row2 - name', 'value':'row2 - value' },
    { 'name':'row3 - name', 'value':'row3 - value' },
    { 'name':'row4 - name', 'value':'row4 - value' },
    ]

table = Tag( 'table', border='1' )
thead = table.appendChild( Tag( 'thead' ) )
tbody = table.appendChild( Tag( 'tbody' ) )
tr = thead.appendChild( Tag( 'tr' ) )
for key in sorted( listOfDicts[ 0 ] ):
    th = tr.appendChild( Tag( 'th' ) )
    th.appendChild( Text( key ) )
for row in listOfDicts:
    tr = tbody.appendChild( Tag( 'tr' ) )
    for key in sorted( row ):
        td = tr.appendChild( Tag( 'td' ) )
        td.appendChild( Text( str( row[ key ] ) ) )
generates the following <table> quickly and easily because, in part, each appendChild returns the Tag appended, which can then be used to append other nodes to the new element:
name value
row1 - name row1 - value
row2 - name row2 - value
row3 - name row3 - value
row4 - name row4 - value

It didn't occur to me to check what the JavaScript methods I'm copying did until I wrote this post, but a cursory check indicates that they do the same thing — returning an appended child — so in that respect, at least, I feel this decision is solid. Similarly, anything that removes a child should return the child being removed, which is also what the JavaScript equivalents do. The lone JavaScript replace* method (replaceChild) returns the node being replaced, so I followed that convention in all of the remove* methods of Tag in the interests of consistency.

All of these methods are also responsible for making certain that the nodes being manipulated have their parent updated as part of the process. That is:

  • Any method that adds a child to a Tag assures that those child nodes' parent doesn't already exist, and that they are set to the Tag that they are being added to; and
  • Any method that removes a child also clears that child's parent (which makes it available to be added to a different Tag if needed);
Since the replace* methods are, essentially, just a removal of an existing child and the addition of a new one, they are responsible for performing both of those tasks if they aren't handled by calling some other methods.

The carry-through of nodes added to a Tag is shown in the implementation of appendChild, as is the checking of a child's parent before executing the addition to the Tag:

@describe.AttachDocumentation()
@describe.argument( 'child', 
    'the child to append to the instance\'s childNodes', 
    BaseNode
)
@describe.raises( TypeError, 
    'if the specified child is not an instance of BaseNode'
)
@describe.raises( MarkupError, 
    'if the child to be appended is already a child of another element'
)
@describe.returns( 'the child that was appended to the instance\'s '
    'childNodes' )
def appendChild( self, child ):
    if not isinstance( child, BaseNode ):
        raise TypeError( '%s.appendChild expects an instance of BaseNode, '
            'but was passed "%s" (%s)' % ( 
                self.__class__.__name__, child, type( child ).__name__ )
            )
    # Checking for the child's current parent
    if child.parent:
        raise MarkupError( '%s.appendChild cannot append "%s" to "%s" '
            'because the node to be appended is already a child of '
            'another element (%s)' % ( 
                self.__class__.__name__, child, self, child.parent )
            )
    # Append the child
    self._childNodes.append( child )
    # Set the child's new parent
    child._SetParent( self )
    # Return the child
    return child
The return of a removed item is shown in removeChild:
@describe.AttachDocumentation()
@describe.argument( 'child', 
    'the child node to remove', 
    BaseNode
)
@describe.raises( TypeError, 
    'if the specified child is not a BaseNode instance'
)
@describe.raises( MarkupError, 
    'if the specified child is not a child of the instance'
)
@describe.returns( 'the child node removed' )
def removeChild( self, child ):
    """
Removes the specified child from the instance's childNodes"""
    if not isinstance( child, BaseNode ):
        raise TypeError( '%s.replaceChild expects a BaseNode-derived object '
            'for its child argument, but was passed "%s" (%s)' % ( 
                self.__class__.__name__, child, 
                type( child ).__name__
            )
        )
    try:
        del self._childNodes[ self._childNodes.index( child ) ]
    except ValueError:
        raise MarkupError( '%s.removeChild could not remove %s because it '
            'is not a childNode of %s' % ( self.__class__.__name__, 
                child, self )
            )
    return child
Finally, the return of the old child is shown in replaceChildAt (which is called by replaceChild):
@describe.AttachDocumentation()
@describe.argument( 'index', 
    'the position of the child to be replaced in the instance\'s childNodes', 
    int, long
)
@describe.argument( 'newChild', 
    'the new child to replace the specified child with', 
    BaseNode
)
@describe.raises( TypeError, 
    'if the newChild specified is not a BaseNode-derived object'
)
@describe.returns( 'the child being replaced in the instance\'s '
    'childNodes' )
def replaceChildAt( self, index, newChild ):
    """
Replaces the child at the specified index/position in the instance's childNodes 
with a new child"""
    if not isinstance( newChild, BaseNode ):
        raise TypeError( '%s.replaceChild expects a BaseNode-derived object '
            'for its newChild argument, but was passed "%s" (%s)' % ( 
                self.__class__.__name__, newChild, 
                type( newChild ).__name__
            )
        )
    oldChild = self._childNodes[ index ]
    self._childNodes[ index ] = newChild
    oldChild._DelParent()
    newChild._SetParent( self )
    return oldChild

The Collected getElement* Methods

JavaScript provides a number of methods that can be used to retrieve zero-to-many tag-elements based on various criteria, and I've added three more to the mix in Tag:

getElementById:
Returns the first Tag child with an id-attribute whose value matches the id supplied
getElementsByAttributeValue:
Returns a list of Tag children that have a specific attribute whose value exactly matches the value supplied
getElementsByClassName:
Returns a list of Tag children that have a class-attribute (using the classList property) containing the value supplied
getElementsByNamespace:
Returns a list of Tag children whose namespaces match the one provided, including children of children that inherit their parent's namespace
getElementsByPath:
Returns a list of Tag children whose DOM-paths relative to the instance match the path specified
getElementsByTagName:
Returns a list of Tag children whose tag-names match the tag-name specified
Many of these methods rely on being able to start with a list of all of an instance's children, so the implementation of getElementsByTagName (which can fulfil that need) is important enough to show in some detail before discussing the remainder.

getElementsByTagName in JavaScript allows * to be provided as a wild-card tag-name. If that wild-card is provided, then the method returns all child tags. That behavior is mirrored in Tag.getElementsByTagName, as is the return of a null (None) value if there are no matches.

@describe.AttachDocumentation()
@describe.argument( 'tagName',
    'the tag-name to search for in child elements. Using "*" will return '
    'all children',
    str, unicode
)
@describe.returns( 'list of child IsElement objects whose tag-name matches '
    'the tagName provided, or None' )
@describe.raises( TypeError, 
    'if the tagName provided is not a str or unicode value'
)
def getElementsByTagName( self, tagName ):
    """
Gets all child tags whose name matches the tag-name provided"""
    if type( tagName ) not in ( str, unicode ):
        raise TypeError( '%s.getElementsByTagName expects a string or '
            'unicode value for the tag-name it\'s to search for but was '
            'passed "%s" (%s)' % ( 
                self.__class__.__name__, tagName, type( tagName ).__name__ )
            )
    results = []
    for child in self.children:
        if child.tagName == tagName or tagName == '*':
            results.append( child )
        subResults = child.getElementsByTagName( tagName )
        if subResults:
            results += subResults
    if results:
        return results
    return None
getElementsByTagName makes use of recursion, by calling itself again for each child found in the current execution and appending the results of that recursive call to the results at the current level of execution. I suspect that there may be a better way of implementing this method, perhaps using a generator in some fashion, but as of this writing, I simply haven't dug into the idea enough to see if it would be worth pursuing.

With getElementsByTagName available, several of the remaining getElement* methods become fairly simple candidate-filtering problems in their implementation:

getElementsByAttributeValue:
Each result is a candidate that has the specified attribute with the specified value
getElementsByClassName:
Each result is a candidate that has the specified value in its classList
getElementsByNamespace:
Each result is a candidate whose namespace matches the one provided
Each, then, follows the same pattern as getElementsByAttributeValue:
@describe.AttachDocumentation()
@describe.argument( 'name',
    'the name of the attribute whose value is to be checked',
    str, unicode
)
@describe.argument( 'value',
    'the value in the specified attribute that must be matched',
    str, unicode
)
@describe.raises( TypeError, 
    'if the specified name is not a str or unicode value' )
@describe.raises( TypeError, 
    'if the specified value is not a str or unicode value' )
@describe.returns( 'a list of Tag instance matching the '
    'attribute-name/-value criteria' )
def getElementsByAttributeValue( self, name, value ):
    """
Gets the child elements that have the attribute specified containg the value 
specified"""
    if type( name ) not in ( str, unicode ):
        raise TypeError( '%s.getElementsByAttributeValue expects a str or '
            'unicode value for its name, but was passed "%s" (%s)' % 
                ( self.__class__.__name__, name, 
                    type( name ).__name__
                )
            )
    if type( value ) not in ( str, unicode ):
        raise TypeError( '%s.getElementsByAttributeValue expects a str or '
            'unicode value for its value, but was passed "%s" (%s)' % 
                ( self.__class__.__name__, value, 
                    type( value ).__name__
                )
            )
    results = [ 
            c for c in self.getElementsByTagName( '*' ) 
            if c.attributes.get( name ) == value
        ]
    if results: 
        return results
    return None
The variations of the other two are, ultimately, just in the generation of the results being returned:
# getElementsByClassName
results = [ 
            c for c in self.getElementsByTagName( '*' ) 
            if className in c.classList 
        ]
# getElementsByNamespace
    results = [ 
        c for c in self.getElementsByTagName( '*' ) 
        if c.namespace == namespace
    ]

getElementById uses getElementsByAttributeValue as a helper-method, but also provides an optional strict argument (defaulting to False) that allows it to raise a MarkupError if more than one result is found:

# ...
results = self.getElementsByAttributeValue( 'id', value )
if strict and len( results ) > 1:
    raise MarkupError( '%s.getElementById, with strict enforcement, '
        'found more than one child with the specified id' % ( 
            self.__class__.__name__, value )
        )
if results:
    return results[ 0 ]
return None

The last remaining method in this group, getElementsByPath, may take some explanation. Consider a web-page that has a fair amount of content, including a lot contained in <div> tags. The page also has two <form>s in it, and within one form are a number of rows, constructed with <div>s. As part of the application's requirements, there is a need to apply some CSS classes to every form row <div>, without altering any of the others, and it has to be done, for whatever reason, server-side in the code. The form to be altered can be identified by a specific DOM path relative to the document, as can the <div>s that need to be altered. That path, to each of those <div>s, might look something like /div/form/fieldset/div from the <body> of the page.

That is what getElementsByPath is built to do. Like getElementsByTagName it uses recursion, but in this case it uses it to drill down through the DOM tree, matching tag-names (and allowing the same wild-card capabilities) in order to find all of the children in the right position relative to the Tag that the method was called from. Its implementation is simpler than might be expected, given the complexity of what it's doing:

@describe.AttachDocumentation()
@describe.argument( 'path', 
    'the path to find matching elements for, delimited by "/", and allowing '
    'wild-cards ("*")',
    str, unicode
)
@describe.raises( TypeError, 'if the supplied path is not a str or '
    'unicode value' )
@describe.returns( 'list of elements whose dom-path from the instance '
    'matched the one specified, or None' )
def getElementsByPath( self, path ):
    """
Gets all child tags that can be identified by following matching tag-names 
down the tree"""
    if type( path ) not in ( str, unicode ):
        raise TypeError( '%s.getElementsByPath expects a string or '
            'unicode value for the path it\'s to search for but was '
            'passed "%s" (%s)' % ( 
                self.__class__.__name__, path, type( path ).__name__ )
            )
    results = []
    try:
        tagName, subPath = path.split( '/', 1 )
    except:
        tagName = path
        subPath = None
    for child in self.children:
        if tagName == '*' or child.tagName == tagName:
            if subPath != None:
                subPathResults = child.getElementsByPath( subPath )
                if subPathResults:
                    results += subPathResults
            else:
                results.append( child )
    if results:
        return results
    return None

Attribute-Related Methods

getAttribute:
Returns the value of the named attribute of the instance, or None if it doesn't exist
hasAttribute:
Checks for the existence of a specific attribute in the instance's attributes collection
hasAttributes:
Checks forthe existence of any attributes in the instance's attributes collection
removeAttribute:
Removes the specified attribute from the instance's attributes collection
setAttribute:
Sets the value of an attribute in the instance's attributes collection

Minus the type-checking of the name argument, getAttribute is really nothing more than:

return self.attributes.get( name )

Similarly, hasAttribute is:

if self.attributes.get( name ):
        return True
    return False
and hasAttributes is:
if self.attributes:
        return True
    return False

Setting attributes is a bit more detailed, but only because the name and value of the inbound attribute are both checked before the set actually occur, and the special handling for data-* attributes:

@describe.AttachDocumentation()
@describe.argument( 'name', 
    'the name of the attribute to set', 
    str, unicode
)
@describe.argument( 'value', 
    'the value of the attribute to set', 
    str, unicode, None
)
@describe.raises( TypeError, 
    'if the supplied name is not a str or unicode value'
)
@describe.raises( TypeError, 
    'if the supplied value is not a str or unicode value or None'
)
@describe.raises( ValueError, 
    'if the supplied name is not a valid attribute-name'
)
@describe.raises( ValueError, 
    'if the supplied name is not a valid attribute-value'
)
def setAttribute( self, name, value ):
    if type( name ) not in ( str, unicode ):
        raise TypeError( '%s.setAttribute expects a str or unicode value '
            'that is a valid attribute-name for the name of the attribute '
            'to be set, but was passed "%s" (%s)' % ( 
                self.__class__.__name__, name, type( name ).__name__
            )
        )
    if not self.attributes.IsValidName( name ):
        raise ValueError( '%s.setAttribute expects a str or unicode value '
            'that is a valid attribute-name for the name of the attribute '
            'to be set, but was passed "%s" which is not valid' % ( 
                self.__class__.__name__, name
            )
        )
    if value == None:
        try:
            del self._attributes[ name ]
        except KeyError:
            pass
        return
    if type( value ) not in ( str, unicode ):
        raise TypeError( '%s.setAttribute expects a str or unicode value '
            'that is a valid attribute value for the value of the attribute '
            'to be set, but was passed "%s" (%s)' % ( 
                self.__class__.__name__, name, type( name ).__name__
            )
        )
    if not self.attributes.IsValidValue( value ):
        raise ValueError( '%s.setAttribute expects a str or unicode value '
            'that is a valid attribute value for the value of the attribute '
            'to be set, but was passed "%s" (%s)' % ( 
                self.__class__.__name__, name, type( name ).__name__
            )
        )
    if name[0:5] == 'data_':
        name = name.replace( 'data_', 'data-' )
    self._attributes[ name ] = value
Attribute removal, though, is also very simple — barring the type-checking of the name, it's basically just a fail-safe deletion of an item from the attributes collection:
try:
    del self.attributes[ name ]
except KeyError:
    pass

The balance of Tag's methods are pretty straightforward, and I won't go into any depth on them:

cloneNode:
Returns a copy of the Tag with the option of returning copies of all of the its childNodes as well
contains:
Determines if a Tag contains another specified node
hasChildNodes:
Determines if a Tag has any childNodes members
The cloneNode method, like the innerHTML property, had to wait until I've got MarkupParser implemented. My plan for implementing it centers around either creating a new Tag instance for shallow copies, or using the MarkupParser class to generate complete copies of a markup-tree from the __str__ and/or __unicode__ methods of Tag, since nodes in general, and tags in particular, cannot have multiple parents (as noted earlier).

In light of how much code there actually is behind the implementation of Tag, and how much of it I didn't do any sort of deep dive into in this post or the last one, I figured I'd share the Tag class code, as well as its unit-test code before I signed off for the day. These are not the complete markup or test_markup modules, so they won't actually execute for lack of various dependencies, but all the code for both (as of this post) is there:

92.9kB
120.1kB

Thursday, May 4, 2017

Generating and Parsing Markup in Python [5]

The Tag class turned out to be something of a beast, partly because of the sheer scope of it, and partly for reasons that had nothing to do with the code involved. The non-code reasons I'll discuss in my next post after Tag is complete, because I think some interesting points surfaced that bear some discussion, but today, I'm going to stick to telling the story of how Tag's implementation unfolded.

Long Post and More to Come

I had really hoped that I'd be able to get all of the implementation of Tag covered in a single post, but by the time I got to the end of the properties (this post), this was already the longest post I've written to date, so I'll pick up next time with the methods implementations.

Tag is the Workhorse of the Module

It should hopefully come as no great surprise that Tag is a pretty large class — the markup-construct that it represents is the foundation for the structure of web-pages and other document-types in other languages. Given the relationships it has with other classes in the markup module:

there were 33 properties and 28 methods that I originally expected to have to implement, some of which were required by IsElement, BaseNode or IsNode. I took some time to gather all of these together into one coherent list in an effort to make sure that I could just progress down that list, implementing as I went, without missing anything. It's a pretty substantial list, despite the occasional items I decided to remove (usually because they served no real purpose in a server-side context). There were also a few members that I decided I wanted to add to the class, and a few relatively minor concerns about name-conflicts that required some thought about altering the member-names. Here's where the final member-list landed, with the additions and alterations noted:
  Tag Members  
Member Name Impl. Req. By Notes
Property Members
accessKey Tag   Is attribute (accesskey)
attributes Tag    
childElementCount Tag IsElement  
childNamespaces Tag    
childNodes Tag IsElement  
children Tag IsElement  
classList Tag   Relates to attribute (class)
className Tag   Relates to attribute (class)
dir Tag   Is attribute (dir),
Name-conflict
firstChild Tag IsElement  
firstElementChild Tag IsElement  
id Tag   Is attribute (id),
Name-conflict
innerHTML Tag    
lang Tag   Is attribute (lang)
lastChild Tag IsElement  
lastElementChild Tag IsElement  
namespace Tag   Relates to Namespace class
namespaceURI Tag   Relates to Namespace class
nextElementSibling BaseNode    
nextSibling BaseNode    
nodeName Tag IsNode  
nodeType Tag IsNode  
ownerDocument BaseNode IsNode  
parent BaseNode    
parentElement BaseNode    
parentNode BaseNode    
previousElementSibling BaseNode    
previousSibling BaseNode    
style Tag   Is attribute (style)
styleList Tag   Relates to attribute (style)
tabIndex Tag   Is attribute (tabindex)
tagName Tag    
title Tag   Is attribute (title)
Method Members
appendChild Tag IsElement  
cloneNode Tag IsElement  
contains Tag IsElement  
getAttribute Tag    
getElementById Tag    
getElementsByAttributeValue Tag    
getElementsByClassName Tag    
getElementsByNamespace Tag    
getElementsByPath Tag    
getElementsByTagName Tag    
hasAttribute Tag    
hasAttributes Tag    
hasChildNodes Tag IsElement  
insertBefore Tag IsElement  
insertChildAt Tag IsElement  
isDefaultNamespace Tag    
isEqualNode Tag IsNode  
isSameNode BaseNode IsNode  
normalize Tag    
prependChild Tag    
removeAttribute Tag    
removeChild Tag IsElement  
removeChildAt Tag IsElement  
removeSelf Tag IsElement  
replaceChild Tag IsElement  
replaceChildAt Tag IsElement  
setAttribute Tag    
toString Tag IsNode  
As before, these members are derived from the w3schools' HTML DOM Element Object page, with some additions from their list of HTML Global Attributes.

Property Implementations

There were a total of five basic patterns that cropped up while I was working through the implementation of Tag's properties, each with their own particular aspects that I found interesting. I've grouped them accordingly in the discussion below.

Storing Attribute Values

When I realized that several of the properties of Tag also had to be expressed as attributes in the rendered markup, I had to give some serious thought to how I wanted to implement the storage of attributes in general, as well as how I was going to link those properties to the attributes they were related to. There are seven properties that are, in a typical HTML/JavaScript environment, both DOM-object properties and attributes that can be set in the text of the markup:

  • accesskey
  • dir
  • id
  • lang
  • style
  • tabindex
  • title
Those do not include the other eight that were added in HTML 5 (see the list noted earlier for details on those).

To further complicate matters, two of them, dir and id are also the names of built-in functions in Python. Setting the naming-conflict aside for the moment, these properties were a potential concern because as attributes, changes to their values as properties should also be reflected in the markup generated and rendered for Tag-instances that use them. That is, given a Tag instance myTag:

# myTag is a Tag instance
myTag.accessKey = 'X'
myTag.style += 'padding:6px;'
myTag.setAttribute( 'name', 'tagname' )
# or myTag.attributes[ 'name' ] = 'tagname'
should eventually render markup that looks something like this:
<myTag accesskey="X" name="tagname" style="padding:6px;">
Given that I expected to implement at least two different ways to set attribute-values, using the setAttribute method and setting the values directly in an attributes dict, my first thought was to simply use an internal dict as the underlying data-storage mechansim for a Tag's attributes. The next potential concern is that as a dict, the attributes property would be both mutable and unconstrained, which felt like a point of some concern. Specifically, because a dict would be mutable, it'd possible to alter an attribute's value to something that isn't legitimate (a non-text value). It'd also be possible to set an attribute with a non-text name (key), because while the keys of a dict can't be any type of value, they can be any of a lot of types that didn't make sense as an attribute-name. For example:
tagInstance.attributes[ True ] = 'value'
shouldn't be valid as an attribute-name, but wouldn't raise an error when it was attempted, which I was concerned about on a longer-term basis. The mutability concern felt like it would become moot if the underlying dict that stores the attributes were able to perform type- and/or value-checking when setting keys and values.

I's not seen any way to accomplish that sort of key- or value-constraint on a standard dict, which more or less required that I create a custom dict-equivalent or -subclass to handle that. I called it AttributesDict.

AttributesDict is a pretty sparse class — it's got an __init__, mostly to assure that the parent dict.__init__ is called, a couple of instance methods for checking the validity of attribute names and values, and an override of the __setitem__ method of the base dict that performs the type- and value-checking:

#-----------------------------------#
# Class attributes (and instance-   #
# attribute default values)         #
#-----------------------------------#

validNameRE = re.compile( '[_A-Za-z][-_A-Za-z0-9]*' )

# ...

#-----------------------------------#
# Instance Methods                  #
#-----------------------------------#

@describe.AttachDocumentation()
@describe.argument( 'name', 
    'the name to check as being valid as an attribute name'
)
@describe.returns( 'True if valid, False otherwise' )
@describe.raises( TypeError, 
    'if the supplied value is not a str or unicode value'
)
def IsValidName( self, name ):
    """
Determines whether the supplied name is valid as an attribute name"""
    if type( name ) not in ( str, unicode ):
        raise TypeError( '%s._IsValidAttributeValue expects a str or '
            'unicode value, but was passed "%s" (%s)' % ( 
                self.__class__.__name__, value, type( value ).__name__ ) )
    if '\n' in name or '\r' in name:
        return False
    if self.validNameRE.sub( '', name ) != '':
        return False
    # TODO: Other checks for validity of the name?
    return True

@describe.AttachDocumentation()
@describe.argument( 'value', 
    'the value to check as being valid in an attribute', 
    str, unicode
)
@describe.returns( 'True if valid, False otherwise' )
@describe.raises( TypeError, 
    'if the supplied value is not a str or unicode value'
)
def IsValidValue( self, value ):
    """
Determines whether the supplied value is valid as an attribute value"""
    if type( value ) not in ( str, unicode ):
        raise TypeError( '%s._IsValidAttributeValue expects a str or '
            'unicode value, but was passed "%s" (%s)' % ( 
                self.__class__.__name__, value, type( value ).__name__ ) )
    if '\n' in value or '\r' in value:
        return False
    # TODO: Other checks for validity of the name?
    return True

@describe.AttachDocumentation()
@describe.argument( 'key', 
    'the key-name to set the value to',
    str, unicode
)
@describe.argument( 'value', 
    'the value to set in the key-name',
    str, unicode
)
@describe.raises( TypeError, 
    'if passed a key-name value that is not a str or unicode type'
)
@describe.raises( TypeError, 
    'if passed a member-value that is not a str or unicode type'
)
@describe.raises( MarkupError, 
    'if passed an invalid key-name'
)
@describe.raises( MarkupError, 
    'if passed an invalid member-value'
)
def __setitem__( self, key, value ):
    """
Override of standard dict.__setitem__ that checks the types and values of key 
and value arguments both before allowing the itemn to be set"""
    if not isinstance( key, ( str, unicode ) ):
        raise TypeError( '%s cannot accept key-names that are not str or '
            'unicode values, or a type derived from one of them. "%s" (%s) '
            'is not allowed' % ( 
                self.__class__.__name__, key, type( key ).__name__
            )
        )
    if not isinstance( value, ( str, unicode ) ):
        raise TypeError( '%s cannot accept member values that are not str '
            'or unicode values, or a type derived from one of them. '
            '"%s" (%s) is not allowed' % ( 
                self.__class__.__name__, value, type( value ).__name__
            )
        )
    if not self.IsValidName( key ):
        raise AttributeError( '%s is not a valid attribute-name in a %s' 
            % ( key, self.__class__.__name__ )
        )
    if not self.IsValidValue( value ):
        raise AttributeError( '%s is not a valid attribute-value in a %s' 
            % ( key, self.__class__.__name__ )
        )
    dict.__setitem__( self, key, value )
The _Delattributes method of Tag then uses an instance of AttributesDict instead of a normal dict:
@describe.AttachDocumentation()
def _Delattributes( self ):
    """
"Deletes" the attributes of the instance by setting it to a new, empty 
AttributesDict instance"""
    self._attributes = AttributesDict()
and the constraint-concern is taken care of. Whether that's enough to resolve the mutability concern remains to be seen.

Implementing Attribute-Properties

Five of the seven Tag-properties that were also attributes all followed a very similar implementation-pattern. Those five properties were:

  • accesskey
  • lang
  • style
  • tabindex
  • title
Here's what accesskey's methods look like, in detail:
#-----------------------------------#
# Instance property-getter methods  #
#-----------------------------------#

@describe.AttachDocumentation()
@describe.returns( 'str or unicode character, or None' )
def _GetaccessKey( self ):
    """
Returns the value of the instance's "accesskey" attribute"""
    return self._attributes.get( 'accesskey' )

# ...

#-----------------------------------#
# Instance property-setter methods  #
#-----------------------------------#

@describe.AttachDocumentation()
@describe.argument( 'value', 
    'the value to set the instance\'s "accesskey" attribute to',
    str, unicode
)
@describe.raises( TypeError, 
    'if passed a value that is not a str or unicode value'
)
@describe.raises( ValueError, 
    'if passed a value that is more than one character in length'
)
@describe.raises( MarkupError, 
    'if passed a value that is not valid as an attribute value'
)
def _SetaccessKey( self, value ):
    """
Sets the value of the instance's "accesskey" attribute"""
    if not value:
        self._DelaccessKey()
    else:
        if not self._IsValidAttributeValue( value ):
            raise MarkupError( '%s.accessKey could not be set to "%s" '
                '- That value is not a valid attribute-value' % ( 
                    self.__class__.__name__, value
                )
            )
        if type( value ) not in ( str, unicode ):
            raise TypeError( '%s.accessKey expects a single-character '
                'str or unicode value, but was passed "%s" (%s)' % ( 
                    self.__class__.__name__, value, 
                    type( value ).__name__
                )
            )
        if len( value ) > 1:
            raise ValueError( '%s.accessKey expects a single-character '
                'str or unicode value, but was passed "%s" (%s)' % ( 
                    self.__class__.__name__, value, 
                    type( value ).__name__
                )
            )
        self._attributes[ 'accesskey' ] = value

# ...

#-----------------------------------#
# Instance property-deleter methods #
#-----------------------------------#

@describe.AttachDocumentation()
def _DelaccessKey( self ):
    """
"Deletes" the value of the instance's "accesskey" attribute by removing 
it from the instance's attributes"""
    try:
        del self._attributes[ 'accesskey' ]
    except KeyError:
        # No such attribute available to delete; ignore
        pass
There was a bit more underneath the getter-, setter- and deleter-methods for lang and tabIndex and less for title's methods. The variations across those three properties were:
lang
Could potentially be validated against any of the various standard ISO language-codes, but I decided to leave that for later, if I even go that far. At present it only raises a TypeError.
tabIndex
The tab-index of a tag is a string representation of a non-negative integer value, so the setter checks to see if the value can be converted into an int, and raises a ValueError if it can't.
title
Has no value-checking, since it should be able to accept anything that is a valid attribute-value.

Implementing Name-Conflicted Attribute-Properties

The last two properties that are also attributes are dir and id. The concern that I had with using those as names as-is was that my plan for creating a Tag instance was to build out an __init__ that looked like this:

def __init__( self, tagName, namespace, **attributes):
    # ...
that allowed attributes to be specified in the code using the attributes keyword-argument. That felt like it made the prospect of instantiating new Tags pretty simple and straightforward. However, since dir and id are defined as Python built-in functions, allowing those to be used as keywords felt... sketchy. By way of example, consider:
class Example( object ):
    def __init__( self, **kwargs ):
        print kwargs

testExample = Example( id='id-value', dir='dir-value' )
actually executes successfully (for now):
{'id': 'id-value', 'dir': 'dir-value'}
There's no guarantee that this would always be the case, though — and even if it never raises any errors in the future because of using a potentially-reserved word, it would still make the built-in dir and id functions unavailable in the body of the function. While I couldn't think of any use-case where that would've been a concern, I also couldn't say with any certainty that it wouldn't be a problem down the line either.

I gave some serious consideration to the idea of establishing a pattern where any attributes specified that began with html would have the html stripped, and the rest reduced to lower-case before being stored as attributes. That would've allowed, for example, htmlId to be used as a keyword for the id attribute, which felt pretty good. Then I thought through what that would mean for an htmlClass attribute-specification. htmlClass would set and read the className property, and would tie to a class attribute. That felt awkward to me. Very awkward. I spent a lot of time going back and forth on various ways of implementing that before deciding that I couldn't really decide how I wanted things to work. Ultimately, in order to keep development moving, I ended up settling on a more brute-force approach, but one that I felt would be easier to refactor later if I could ever escape the analysis paralysis I was encountering about the different approaches. I ended up with a Tag.__init__ looking like this:

#-----------------------------------#
# Instance Initializer              #
#-----------------------------------#
@describe.AttachDocumentation()
def __init__( self, tagName, namespace=None, **attributes ):
    """
Instance initializer"""
    # Call parent initializers, if applicable.
    BaseNode.__init__( self )
    IsElement.__init__( self )
    # Set default instance property-values with _Del... methods as needed.
    # - Attributes first, since many of the rest use that
    self._Delattributes()
    # - Then the rest
    self._DelaccessKey()
    self._DelchildNodes()
    self._Delclass()
    self._DelhtmlDir()
    self._DelhtmlId()
    self._DelinnerHTML()
    self._Dellang()
    self._Delnamespace()
    self._Delstyle()
    # Set instance property values from arguments if applicable.
    self._SettagName( tagName )
    # Various attribute-setters that collide with "reserved" words 
    # in Python
    if 'className' in attributes:
        self._SetclassName( attributes[ 'className' ] )
        del attributes[ 'className' ]
    if 'htmlDir' in attributes:
        self._SethtmlDir( attributes[ 'htmlDir' ] )
        del attributes[ 'htmlDir' ]
    if 'htmlId' in attributes:
        self._SethtmlId( attributes[ 'htmlId' ] )
        del attributes[ 'htmlId' ]
    if 'htmlFor' in attributes:
        self.setAttribute( 'for', attributes[ 'htmlFor' ] )
        del attributes[ 'htmlFor' ]
    # The remaining (normal) attributes:
    if attributes:
        self._Setattributes( attributes )
    # The namespace
    if namespace:
        self._Setnamespace( namespace )
    # Other set-up
and htmlDir and htmlId properties (className has special considerations that I'll go into in a bit, and htmlFor is really only a convenience item for creating <label> tags, so I didn't feel the need to set up an htmlFor property).

The implementation of the getter-/setter-/deleter-mthods for htmlId and htmlDir are very similar, though it seemed prudent to put some value-checks in htmlDir, since the attribute was not supposed to allow completely free-form values. htmlDir's related methods ended up looking like this:

#-----------------------------------#
    # Instance property-getter methods  #
    #-----------------------------------#

    # ...

    @describe.AttachDocumentation()
    def _GethtmlDir( self ):
        """
Returns the value of the instance's "dir" attribute"""
        return self._attributes.get( 'dir' )

    # ...

    #-----------------------------------#
    # Instance property-setter methods  #
    #-----------------------------------#

    # ...

    @describe.AttachDocumentation()
    @describe.argument( 'value', 
        'the value to set the instance\'s "dir" attribute to',
        str, unicode
    )
    @describe.raises( TypeError, 
        'if passed a value that is not a str or unicode value'
    )
    @describe.raises( MarkupError, 
        'if passed a value that is not valid as an attribute value'
    )
    def _SethtmlDir( self, value ):
        """
Sets the value of the instance's "dir" attribute"""
        if value == None or value == '':
            self._DelhtmlDir()
        else:
            validValues = ( 'auto', 'ltr', 'rtl' )
            if not self.attributes.IsValidValue( value ):
                raise MarkupError( '%s.dir could not be set to "%s" '
                    '- That value is not a valid attribute-value' % ( 
                        self.__class__.__name__, value
                    )
                )
            if type( value ) not in ( str, unicode ):
                raise TypeError( '%s.htmlDir expects a str or unicode '
                    'value, one of %s, but was passed "%s" (%s)' % ( 
                        self.__class__.__name__, str( validValues ), 
                        value, type( value ).__name__
                    )
                )
            if value.lower() not in validValues:
                raise ValueError( '%s.htmlDir expects a str or unicode '
                    'value, one of %s, but was passed "%s" (%s)' % ( 
                        self.__class__.__name__, str( validValues ), 
                        value, type( value ).__name__
                    )
                )
            self._attributes[ 'dir' ] = value

    # ...

    #-----------------------------------#
    # Instance property-deleter methods #
    #-----------------------------------#

    # ...

    @describe.AttachDocumentation()
    def _DelhtmlDir( self ):
        """
Deletes the value of the instance's "dir" attribute by removing 
it from the instance's attributes"""
        try:
            del self._attributes[ 'dir' ]
        except KeyError:
            # No such attribute available to delete; ignore
            pass

    # ...

    #-----------------------------------#
    # Instance Properties               #
    #-----------------------------------#

    # ...

    htmlDir = describe.makeProperty(
        _GethtmlDir, _SethtmlDir, _DelhtmlDir, 
        'the value of the instance\'s dir attribute',
        str, unicode
    )

    # ...

Properties that Relate to Attributes

The className and classList properties have an interesting relationship on the browser side: Altering one affects the other, which means that it's possible to use array-based operations on classList, and those changes will carry through to className. Consider:

<div id="example" class="class1 class2 class3">Example div</div>
<script>
    example = document.getElementById( 'example' );
    console.log( 'example.className ... ' + example.className );
    console.log( 'example.classList ... ' + example.classList );
    console.dir( example.classList );
    console.log( 'Removing class2' );
    example.classList.remove( 'class2' );
    console.log( 'example.classList ... ' + example.classList );
    console.dir( example.classList );
    console.log( 'Adding class4 to className' );
    example.className += ' class4'
    console.log( 'example.className ... ' + example.className );
    console.log( 'example.classList ... ' + example.classList );
    console.dir( example.classList );
</script>
If this is executed in a browser (Chromium in my case), the console shows:
example.className ... class1 class2 class3
example.classList ... class1 class2 class3
   [DOMTokenList] ... [ 'class1', 'class2', 'class3' ]
Removing class2
example.classList ... class1 class3
   [DOMTokenList] ... [ 'class1', 'class3' ]
Adding class4 to className
example.className ... class1 class3 class4
   [DOMTokenList] ... [ 'class1', 'class3', 'class4' ]
That's actually kind of neat, I think.

I first discovered that when I was building out the first big list of properties and methods at the beginning of the markup module posts, and it got me thinking about doing the same sort of thing with the style attributes and a styleList attribute — having the ability to use list-operations against individual class-names and inline style specifications seemed like a potentially powerful tool to add. The real challenge felt like it would be in how to actually implement that sort of functionality, because of all the varied interactions needed:

  • The underlying storage still needs to live in an attribute-value, as a flat text-value if possible, so that special considerations don't have to be made for rendering the class and style attributes;
  • The getter-methods for the *List properties need to return a list-structure from the flat-text attribute-value;
  • The setter-methods for the *List properties need to accept a list, and generate the flat text-value in the applicable attribute;
  • The deleter-methods for the *List properties needs to not destroy the interaction between the *List and non-*List properties;
I determined that all of this could be managed by creating a class that either derived from the built-in list, and overrode the functions that allow the mutation of members (the full list of properties and methods is published on the Python site, or creating a completely custom class that does all the list-emulation needed. In either case, any change to the members of the object would have to be able to call the appropriate setter-method of the instance, and the rest would take care of itself. The only other consideration is that CSS classes and inline styles have different member-separators: Classes use a space, and style-declarations use a semicolon.

At first, I wasn't sure if that would be too complex for what I needed — Since I'd just encountered the classList property in the last couple of weeks, I'd obviously never used it, so it wasn't a big concern for me to not include it. At the same time, it definitely felt like it could be of a lot of use, so I preferred to implement it if I could. As it turned out, though, it wasn't as complex as I'd feared. The implementation proof-of-concept code is too long for me to cover in great detail if I want to keep this post to anything close to a reasonable length, but I'll make it downloadable at the end of the post. Here's a quick summary of what I ended up with:

  • I defined a class (AttributeValueList) that derives from list;
  • I added pointers to the getter-, setter- and deleter-methods for the flat-text attributes to the __init__ of the new class, as well as a separator value that would be used elsewhere to fetch a new list from the flat-text value, or to join the instance's list as a new flat-text value:
    def __init__( self, getter, setter, deleter, separator, iterable=[] ):
        self._getter = getter
        self._setter = setter
        self._deleter = deleter
        self._separator = separator
        list.__init__( self, iterable )
  • I defined two helper-methods (_pullFromGetter() and _pushToSetter()) that would refresh the instance's list-values from the flat-text attribute and re-set the flat-text value from the current list-values, respectively:
    def _pullFromGetter( self ):
        print '### Calling %s._SetclassName' % self.__class__.__name__
        # remove all current members
        while len( self ):
            self.remove( 0 )
        # get the new values
        values = self._getter()
        # append each of them to self
        for value in values:
            self.append( value )
    
    def _pushToSetter( self ):
        print '### Calling %s._pushToSetter' % self.__class__.__name__
        self._setter( self._separator.join( self ) )
  • I overrode all of the methods of list that could affect the members if the base list, following a pattern like:
    def __some_list_method__( self, [args] ):
        # Call the original list-method against the instance, 
        # with the arguments passed to the method
        self.__some_list_method__( self, [args] )
        # Call a protected helper-method to update the 
        # "flat-text" value in the object that the 
        # instance relates to
        self._pushToSetter()
  • Finally, in a very stripped-down copy of Tag, I created basic property getter-, setter- and deleter-methods and the corresponding properties, wiring things up so that:
    • The deleter-methods for the *List properties created a new, empty instance of AttributeValueList;
    • The setter-methods for the *List properties removed all members from the current AttributeValueList storage-object, then added in the new values;
    • The setter-methods for the flat-text attributes would call the _pullFromGetter method of their AttributeValueList equivalent;
    The bare-bones implementation for the classList/classNameclassList property-set in the POC shows all of that:
    def _GetclassName( self ):
        return self._attributes.get( 'class' )
    
    def _GetclassList( self ):
        return self._classList
    
    def _SetclassName( self, value ):
        self._attributes[ 'class' ] = value
        self._classList._pullFromGetter()
    
    def _SetclassList( self, value ):
        self._classList = AttributeValueList( 
            self._GetclassName, 
            self._SetclassName, 
            self._DelclassName, 
            ' ', value
        )
    
    def _DelclassName( self ):
        try:
            del self._attributes[ 'class' ]
        except KeyError:
            pass
    
    def _DelclassList( self ):
        self._classList = AttributeValueList( 
            self._GetclassName, 
            self._SetclassName, 
            self._DelclassName, 
            ' '
        )
    
    className = property( _GetclassName, _SetclassName, _DelclassName )
    classList = property( _GetclassList, _SetclassList, _DelclassList )
That proved out enough of the concept that I could run with it behind the scenes. The quick-and-nasty testing from the POC script performed a few typical/basic manipulations:
def printItem( item ):
    print '+- className .............. %s (%s)' % ( 
        item.className, type( item.className ).__name__ )
    print '+- classList .............. %s (%s)' % ( 
        item.classList, type( item.classList ).__name__ )

example = Tag()
print 'example Tag: %s' % example
print '+- classList._getter ...... %s' % ( example.classList._getter.__name__ )
print '+- classList._setter ...... %s' % ( example.classList._setter.__name__ )
print '+- classList._deleter ..... %s' % ( example.classList._deleter.__name__ )
print '+- classList._separator ... "%s"' % ( example.classList._separator )
print '#' + '-'*38 + '#'

print 'example'
printItem( example )

print '| == example.classList += [ \'addedClass\' ]'
example.classList += [ 'addedClass' ]
printItem( example )

print '| == example.className = \'class1 class2\''
example.className = 'class1 class2'
printItem( example )

print '| == example.classList.remove( \'class1\' )'
example.classList.remove( 'class1' )
printItem( example )

print '| == example.classList.append( \'class4\' )'
example.classList.append( 'class4' )
printItem( example )

print '| == example.classList.insert( 1, \'class1\' )'
example.classList.insert( 1, 'class1' )
printItem( example )

print '| == example.classList += [ \'addedClass\' ]'
example.classList += [ 'addedClass' ]
printItem( example )
and yielded expected results for those actions/operations:
example Tag: <__main__.Tag object at 0x7f996ac85e10>
+- classList._getter ...... _GetclassName
+- classList._setter ...... _SetclassName
+- classList._deleter ..... _DelclassName
+- classList._separator ... " "
#--------------------------------------#
example
+- className .............. None (NoneType)
+- classList .............. []
| == example.classList += [ 'addedClass' ]
+- className .............. addedClass (str)
+- classList .............. ['addedClass']
| == example.className = 'class1 class2'
+- className .............. class1 class2 (str)
+- classList .............. ['class1', 'class2']
| == example.classList.remove( 'class1' )
+- className .............. class2 (str)
+- classList .............. ['class2']
| == example.classList.append( 'class4' )
+- className .............. class2 class4 (str)
+- classList .............. ['class2', 'class4']
| == example.classList.insert( 1, 'class1' )
+- className .............. class2 class1 class4 (str)
+- classList .............. ['class2', 'class1', 'class4']
| == example.classList += [ 'addedClass' ]
+- className .............. class2 class1 class4 addedClass (str)
+- classList .............. ['class2', 'class1', 'class4', 'addedClass']

As a side-note: There were other ways to accommodate the list- and non-list versions of both attributes. One that I contemplated for a while was to simply store the actual list-of-string values, and just collapse those down during the rendering process. The problem that I ended up having with that approach was a combination of the discrepancy between the attribute- and property-names for class/className and a strong desire to be able to just dump the attribute keys and values during rendering. For style that wasn't a concern — the property- and attribute-names are identical. Trying to come up with a process that would handle rendering the class from a className that was, in turn calculated from classList started giving me a headache pretty quickly, though I believe I found a way to make it workable. The trade-off, though, was the addition of what might be called special handling for just that one attribute. I'm not a big fan of hard-coding exceptions into code if there's any viable way around it. Ultimately, that preference on my part was why I took the path I did.

Parents and Children, Nodes and Elements, and Their Related Properties

In the JavaScript world that I'm modeling the proeprties and methods of Tag after, there is a distinction between nodes and elements. An element is a type of node, as are text-nodes, comments, and (presumably) CDATA sections. What sets an element apart from a node, as far as my analysis seemed to indicate, was that elements can have children, which are also nodes, and may be elements.

It seems likely that's why JavaScript elements have both children and childNodes properties, and why there are members like lastChild and lastElementChild — to allow retrieval of either all children, or only children that are elements.

While I didn't see a whole lot of use for that distinction while working on the baseline notes and ideas for the markup module, providing as close a parallel functionality-set as I could more or less required that I implement all of those members as well. The foundation of all of them was the childNodes property, and the storage of child BaseNode objects therein.

When push comes to shove, BaseNode children of a Tag are a sequence of objects, so I started with a basic Python list to store them as a proof of concept, but it shared a lot of the concerns that I had that led to the creation of AttributesDict: The underlying list was mutable and unconstrained, so it would be possible to directly insert a member that wasn't valid as a child. Fundamenally, the ElementList class that I built to handle the constraint wasn't all that different from the AttributeValueList class I mentioned earlier. The main difference was really in that there was no reason to care when items were removed, so the method-overrides that related to that could be stripped out, leaving:

#-----------------------------------#
# Instance Methods                  #
#-----------------------------------#

@describe.AttachDocumentation()
def __add__( self, y ):
    """
Override of the base method from list that performs type-checking on the item 
to be added"""
    if not isinstance( y, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) cannot be added.' % ( 
                self.__class__.__name__, y, type( y ).__name__ )
            )
    list.__add__( self, y )

def __iadd__( self, y ):
    """
Override of the base method from list that performs type-checking on the item 
to be added"""
    if not isinstance( y, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) cannot be added.' % ( 
                self.__class__.__name__, y, type( y ).__name__ )
            )
    for item in y:
        list.append( self, item )
    return self

def __imul__( self, y ):
    """
Override of the base method from list that performs type-checking on the item 
to be multiplied"""
    if not isinstance( y, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) cannot be multiplied.' % ( 
                self.__class__.__name__, y, type( y ).__name__ )
            )
    list.__imul__( self, y )
    return self

def __mul__( self, y ):
    """
Override of the base method from list that performs type-checking on the item 
to be multiplied"""
    if not isinstance( y, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) cannot be multiplied.' % ( 
                self.__class__.__name__, y, type( y ).__name__ )
            )
    list.__mul__( self, y )

def __rmul__( self, y ):
    """
Override of the base method from list that performs type-checking on the item 
to be multiplied"""
    if not isinstance( y, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) cannot be multiplied.' % ( 
                self.__class__.__name__, y, type( y ).__name__ )
            )
    list.__rmul__( self, y )

def __setitem__( self, i, y ):
    """
Override of the base method from list that performs type-checking on the item 
to be set"""
    if not isinstance( y, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) is not allowed.' % ( 
                self.__class__.__name__, y, type( y ).__name__ )
            )
    list.__setitem__( self, i, y )

def __setslice__( self, i, j, y ):
    """
Override of the base method from list that performs type-checking on the item 
to be set"""
    if not isinstance( y, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) is not allowed.' % ( 
                self.__class__.__name__, y, type( y ).__name__ )
            )
    list.__setslice__( self, i, j, y )

def append( self, y ):
    """
Override of the base method from list that performs type-checking on the item 
to be appended"""
    if not isinstance( y, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) is not allowed.' % ( 
                self.__class__.__name__, y, type( y ).__name__ )
            )
    list.append( self, y )

def extend( self, iterable ):
    """
Override of the base method from list that performs type-checking on the items 
to be extended"""
    badItems = [ i for i in iterable if not isinstance( i, BaseNode ) ]
    if badItems:
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members, but included %s whish are not allowed.' 
            % ( self.__class__.__name__, badItems )
        )
    list.extend( self, iterable )

def insert( self, index, obj ):
    """
Override of the base method from list that performs type-checking on the item 
to be inserted"""
    if not isinstance( obj, BaseNode ):
        raise TypeError( '%s is only allowed to have BaseNode-'
            'derived members: "%s" (%s) is not allowed.' % ( 
                self.__class__.__name__, obj, type( obj ).__name__ )
            )
    list.insert( self, index, obj )

The childNodes property was set up to use an instance of ElementList for its storage, and is read-only. The children property, also read-only, was built using a list comprehension that filtered childNodes down to only those members that were instances of IsElement:

@describe.AttachDocumentation()
def _Getchildren( self ):
    """
Gets the the sequence of all children of the instance that are elements"""
    return [ 
        c for c in self._childNodes 
        if isinstance( c, IsElement )
    ]
With those two properties in place, a lot of the remaining properties were easily implemented:
childElementCount
The number of members of children
firstChild
The first member of childNodes
firstElementChild
The first member of children
lastChild
The last member of childNodes
lastElementChild
The last member of children
nextElementSibling
The first member of childNodes after the index of the instance itself that is an IsElement instance
nextSibling
The first member of childNodes after the index of the instance itself
previousElementSibling
The first member of childNodes before the index of the instance itself that is an IsElement instance
previousSibling
The first member of childNodes before the index of the instance itself
The parentElement and parentNode properties seemed to me to be needlessly confusing — As far as I've been able to tell, there is no way for a non-element node to be a parent to another node. I looked around for a while to see if I was missing anything, but couldn't find anything that led me to think otherwise. Ultimately I decided to collapse the two of them down into the parent property.

The Remaining Properties

The majority of the remaining properties' implementations either follow some simple variation of my normal property-getter, -setter and -deleter structure, storing the value in an underlying protected local attribute, or are calculated in some fashion from another property or one of the underlying local attributes. The namespace and namespaceURI properties are a perhaps-typical example of that structure as it applies to a non-simple underlying-attribute type/value.

#-----------------------------------#
# Instance property-getter methods  #
#-----------------------------------#

# ...

@describe.AttachDocumentation()
def _Getnamespace( self ):
    """
Gets the Namespace associated with the instance"""
    if self._namespace:
        return self._namespace
    else:
        if parent:
            return parent.namespace
        else:
            return None

@describe.AttachDocumentation()
def _GetnamespaceURI( self ):
    """
Gets the URI of the Namespace associated with the instance"""
    if self.namespace:
        return self.namespace.namespaceURI
    return None

# ...

#-----------------------------------#
# Instance property-setter methods  #
#-----------------------------------#

# ...

@describe.AttachDocumentation()
@describe.argument( 'value', 
    'The Namespace, or the URI/unique identifier of the Namespace to '
    'associate with the instance', 
    Namespace, str, unicode
)
@describe.raises( TypeError, 
    ''
)
def _Setnamespace( self, value ):
    """
Sets the Namespace association for the instance"""
    if not value:
        self._Delnamespace()
    if type( value ) in ( str, unicode ):
        try:
            value = Namespace.GetNamespace( value )
        except MarkupError:
            value = None
    if not isinstance( value, Namespace ):
        raise TypeError( '%s.namespace expects a Namespace instance, '
            'or a str or unicode URI value of a registered Namespace, '
            'but was passed "%s" (%s)' % ( 
                self.__class__.__name__, value, 
                type( value ).__name__
            )
        )
    self._namespace = value

# ...

#-----------------------------------#
# Instance property-deleter methods #
#-----------------------------------#

@describe.AttachDocumentation()
def _Delnamespace( self ):
    """
"Deletes" the namespace-association of the instance by setting it to None"""
    self._namespace = None

There is one remaining property that I can't implement just yet: innerHTML. The getter-method side of it is pretty straightforward, since it really just boils down to rendering the children of the instance. While that's in the method-members that I haven't touched on yet (__str__ and/or __unicode), I don't expect the getter functionality of innerHTML to be much more than a call to one or the other. On the setter side of the property, though, I need the ability to parse markup to be functional before I can work that out. That means that innerHTML is waiting on the implementation of the MarkupParser class.

That wraps up the properties of Tag. I promised earlier to make the proof-of-concept code for AttributeValueList available for download, so before I stop, here it is:

Tuesday, May 2, 2017

Generating and Parsing Markup in Python [4]

Before tackling the Tag class, the last major concrete class in the markup module that I'll need to be able to generate, well... markup, there are a few items that contribute to it that need attention. The reasons for needing the attention are, perhaps, not obvious, so in today's post I'll take a step back, and explain/examine aspects of my end-goal and show how those items fit into meeting that goal.

Keeping Markup and Logic Separate

Back in the post where I decided to work on markup-generation first, I mentioned:

I firmly believe the idea of separation of markup/structure from functionality/logic has merit — to the point that one of my goals for this framework is to make it as easy as possible to keep that separation, while still allowing as much designer-level control as possible over the structure and appearance of pages.
I didn't really go into any details about how I wanted that to work, just that it was a priority. I'm not going to get deeply into the details today, but I can at least shed some light on what I'm going to do, and what the implications of that are as they relate, here and now, to the markup module.

Separation of Markup/Design from Logic/Function

Consider the following tentative page-template:

<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="XML_NS_FOR_SOME_HTML_TYPE" 
  xmlns:idic="idic.page.component.path"
  xmlns:app="app.page.component.path">
  <head>
    <title>
      <idic:IfLoggedIn>
        <idic:Placeholder value="Page.Title">Page.Title</idic:Placeholder>
      </idic:IfLoggedIn>
      <idic:IfNotLoggedIn>
        <idic:Placeholder value="Page.Title">Page.Title</idic:Placeholder>
        Log-in Required:
      </idic:IfNotLoggedIn>
    </title>
    <idic:ScriptManager role="HeadScripts">
      <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
    </idic:ScriptManager>
    <idic:StyleManager>
      <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" />
    </idic:StyleManager>
  </head>
  <body>
    <idic:IfLoggedIn>
      <div id="main" class="container">
        <h1>
          <idic:Placeholder value="Page.Title">Page.Title</idic:Placeholder>
        </h1>
        <app:SomeComponent>
          <!-- etc., etc. -->
        </app:SomeComponent>
      </div>
    </idic:IfLoggedIn>
    <idic:IfNotLoggedIn>
      <div id="main" class="container">
        <h1>
          <idic:Placeholder value="Page.Title">Page.Title</idic:Placeholder>
          Log-in Required:
        </h1>
        <form action="" method="post" 
          app:component="LogInForm"
          app:user="username" app:passwd="userpass">
          <idic:Template>
            <div class="form-group">
              <label for="username">Name:</label>
              <input type="text" id="username" name="username"
              class="form-control" />
            </div>
            <div class="form-group">
              <label for="userpass">Password:</label>
              <input type="password" id="userpass" name="userpass"
              class="form-control" />
            </div>
            <div>
              <button type="submit" class="btn btn-default">Log In</button>
            </div>
          </idic:Template>
        </app:LogInForm>
      </div>
    </idic:IfNotLoggedIn>
    <idic:ScriptManager role="DefinitionScripts">
      <script src="AnotherExternalScript.js"></script>
    </idic:ScriptManager>
    <idic:ScriptManager role="RuntimeScripts" />
  </body>
</html>
This is a very bare-bones example of the sort of templating that I'm trying to accomplish with the framework. In order to illustrate the ability to add third-party frameworks, I've added links to the Bootstrap core CSS and JavaScript. If it were passed off to a client browser (removing the initial XML declaration if necessary) it would render reasonably well, though it would show two major page-sections, one in the <idic:IfLoggedIn> element in the markup and one in the <idic:IfNotLoggedIn>. Still, it would render:

On top of that, though this example might be a little odd to work with because it's got displays for logged-in and not-logged-in states, there's not much here that even a still-in-school intern wouldn't be able to understand just by looking at it, and the new stuff can pretty much be ignored:
  • Any page-components — that is, any tag with an XML namespace (idic or app) — can be ignored or left alone;
  • All the markup inside the page-components uses standard HTML tag-names and reasonably-normal tag structure, barring the XML structure;
    That even holds true for:
    • The external/third-party style-sheet references; and
    • The external/third-party script-references
  • Any attributes with a page-component namespace can also be ignored or left alone.
Granted, it will take at least some getting used to, particularly for anyone who isn't familiar with XML's rules, but I'd guess that most of the tools that are out there in the wild will recognize XML and provide assistance with it if/as needed while authoring in a page-template document.

Without worrying too much about the details or implementation behind the page-component tags in the example template, here's at least a rough approximation of what they'd probably do:

<idic:IfLoggedIn>:
Would render any child markup to the final page-output if the user is logged in.
<idic:Placeholder>:
Would replace its child markup with a value identified by its value attribute.
<idic:IfNotLoggedIn>:
Would render any child markup to the final page-output if the user is not logged in.
<idic:ScriptManager>:
Would gather any number of external script-references or inline script-code, and keep track of them so that components can require specific scripts without having to worry about multiple instance of those scripts being present in the final rendered page markup.
<idic:StyleManager>:
Performs much the same task as a ScriptManager, but for external style-sheet references and inline stylesheets.
<app:SomeComponent>:
Some application component tag — Something that the underlying application renders, with or without user interaction.
<idic:Template>:
Defines a block of markup that will be used by a parent page-component to define some part of what its rendered markup will look like.
Each of these component-tags would need to be able to map back to a Python class (something that is derived from a common base page-component class that I'll define after I've completed the markup module and the next two major topics after that). That mapping, I think, can eventually be handled by a combination of one or more properties in a Namespace instance, and probably some sort of component-registration process that I'll figure out in detail later.

I'm also planning, at least tentatively, to make page-component equivalents of all of the standard HTML form-tags — <form>, all of the <input> variations, <select> and <textarea>, with an eye towards allowing in-template specification of server-side (and maybe client-side) validation. That, too, will rely at least to some degree on the same sort of Namespace-based mapping and/or component-registration process.

If all of this seems like a lead-in to defining the Namespace class, well... Yes, really. But before that, there's one other item to consider...

Rendering Models

In order for an XML-based page-template to render out to a non-XML-based markup language like HTML 5, and to do so without strict XML rendering rules, there needs to be some way to determine how any given tag in a document should be rendered. Take a

<link rel="stylesheet" ... >
tag as an example. In XML, that would be constructed as a self-closing tag:
<link rel="stylesheet" ... />
but in HTML 5 it's not closed at all. A <script> tag that only references an external source, having no internal content, is perfectly legitimate in XML as
<script src="..." />
but if that markup gets issued to a browser, there's a good chance that it'll make the page puke in odd and unexpected ways. I've seen similar things happen with self-closed <div> tags, and I'd expect similar issues to arise from any HTML tag that's supposed to have content inside it.

Then there are the component-tags listed above. Any one of them might generate child markup (or not), have a wrapping tag (or not) or be represented themselves by a tag in the rendered markup (or not), in any combination of those three possibilities.

All of these represent what I think of as a rendering model — some indication of how a given tag must or should be rendered to a client browser, and whose rendering rules might vary from one markup-dialect to another, even if the tags themselves are identical.

By the time I get to a point where I can define actual document types, I should have a pretty good idea of what the rendering rules are for all the tags within the markup language that the document is for. Those individual tag-level rendering-models, then, can be defined as a set of properties for each tag for a given document-type, and those document-types can be identified by a Namespace that can actually be identified, in turn, by a real namespace, though the official namespace for HTML 5 is not distinct, so it'll require some workaround:

XML
http://www.w3.org/XML/1998/namespace
XHTML
http://www.w3.org/1999/xhtml
HTML5
http://www.w3.org/2015/html
or, maybe:
idic.markup.HTML5Document
(Because the official namespace for HTML 5, even as late as the HTML 5.2 specification is http://www.w3.org/1999/xhtml, which is the same as for XHTML...)
But I digress...

I could think of the following rendering-model variations:

NoChildren
A tag that should never have children, and so shouldn't render any, even if it does
Example: [HTML 5] <link ... >
Example: [XHTML] <link ... />
Mixed
A tag that might or might not have children, and should render with a closing-tag if children are present, or as if it were a NoChildren tag if it doesn't.
Example: [XML] Any tag that doesn't have required children in its schema or DTD definition.
RequireEndTag
A tag that should always render with a closing tag, even if it has no child content.
Example: [HTML 5, XHTML] <div></div>, <script></script> and most other tags
ChildrenOnly
A tag that renders only any child markup.
Example: There will likely be at least a few page-components that will use this model, though at present I don't have any defined that I can point to. The idic:ScriptManager and idic:StyleManager tags might fit into this model, depending on how their managed scripts and styles get stored, though.
These feel to me like they could fit well into another pseudo-enumeration, the same sort of structure/construct that was created for managing node-types:
renderingModels = namedtuple(
    'enumRenderingModels', 
    [ 'NoChildren', 'Mixed', 'RequireEndTag', 'ChildrenOnly' ]
    )(
        NoChildren=0,
        Mixed=1,
        RequireEndTag=2,
        ChildrenOnly=3,
    )

__all__.append( 'renderingModels' )

My earlier comparison of XML vs. HTML 5 rendering of a link tag was, now that I look at it again, somewhat misleading. There is a (subtle?) distinction between these rendering models and the XML-style vs. HTML-style self-closing/unclosed tags (<link ...> vs. <link ... /> as an example again): Whether a given language's handling of a NoChildren object uses the XML-style unary-tag syntax (<link />) or just leaves it hanging like HTML 5 does (<link>) is really more a function of the markup language than the tags within that language. That, too, feels like something that could be stored as a Namespace property and used when the final rendered output is generated, but I'm going to ponder on that until I get to the point where I'm actually defining how documents work.

Another perhaps-odd consideration: This structure would allow the generation of tags with child markup, while also allowing the rendering of the final markup of such tags to prohibit rendering of those children. On the surface that probably sounds odd. I thought so too. I'm leaving that implied capability in place in the framework, though, because while I can't think of any real-world case where a tag in one markup-language allows children, but the same tag in another doesn't, I can't guarantee that it can't happen. I'll probably think more on that in the future, but for the time being, it doesn't feel like a major consideration, so I'll leave it in place, as weird as it feels to me.

That, I think, is all that need be done to define rendering-models. I'll dig in to the application of them for tag-rendering purposes when I get to the Tag class, but I've got enough now to define Namespace, I think.

The Namespace class

Namespace is built with my standard final class template as a starting-point. The rationale for making it nominally-final is about as weak as I consider to still be valid: I cannot think of an actual need for it to ever be extended. That said, if a reason surfaces, I'll drop the nominally-final check-code out of its definition. Apart from that, it's a very straightforward class, I think: a few properties, a class-level registration-process of specific namespaces to facilitate creation of some commonly-used variants as constants in the markup module... Not much else to it.

The properties of Namespace are:

DefaultRenderingModel
A value from the renderingModels enumeration, defining the rendering-model to use for tags that don't have a specific one identified;
TagRenderingModels
A dictionary of tag-names to rendering-model values that indicate the rendering-model to be used for specific tags — e.g.:
{
    # ...
    'br':renderingModels.NoChildren,
    'ing':renderingModels.NoChildren,
    'link':renderingModels.NoChildren,
    # ...
}
for an HTML dialect.
namespaceURI
The unique identifier of the namespace instance, used as a name to register the instance with the Namespace class, and to retrieve a namespace by that URI if needed.

As is typical for me, I'm type- and value-checking the values going into these properties in their various _Set* methods. For the most part, those checks are pretty simple, but the check- and set-process for setting TagRenderingModels is a bit more complex than anything I think I've shown so far, so I'll show and discuss it briefly:

@describe.AttachDocumentation()
@describe.raises( TypeError, 
    'if passed a value that is not a dict, or not derived from one'
)
@describe.raises( ValueError, 
    'if passed a dict with one or more invalid keys (that are not '
    'valid tag-names)'
)
@describe.raises( ValueError, 
    'if passed a dict with one or more invalid values (not members '
    'of renderingModels)'
)
def _SetTagRenderingModels( self, value ):
    """
Sets the dictionary of tag-names:rendering-models to use for tags that don't 
use the default rendering model of the namespace."""
    if not isinstance( value, dict ):
        raise TypeError( '%s.TagRenderingModels expects a dict of '
            'str or unicode values that are valid tag-names as keys, '
            'and members of renderingModels as values, but was passed '
            '"%s" (%s)' % ( 
                self.__class__.__name__, value, type( value ).__name__
            )
        )
    # TODO: Figure out a better way to validate tag-names
    badKeys = [
        k for k in sorted( value ) 
        if type( k ) not in ( str, unicode )
        or ' ' in k
        or '\n' in k
        or '\t' in k
        or '\r' in k
    ]
    if badKeys:
        raise ValueError( '%s.TagRenderingModels expects a dict of '
            'str or unicode values that are valid tag-names as keys, '
            'and members of renderingModels as values, but was passed '
            'a dict with invalid key-values %s' % ( 
                self.__class__.__name__, badKeys
            )
        )
    badValues = dict(
            [
                ( k, value[ k ] ) for k in sorted( value ) 
                if value[ k ] not in renderingModels
            ]
        )
    if badValues:
        raise ValueError( '%s.TagRenderingModels expects a dict of '
            'str or unicode values that are valid tag-names as keys, '
            'and members of renderingModels as values, but was passed '
            'a dict with invalid values %s' % ( 
                self.__class__.__name__, badValues
            )
        )
    self._tagRenderingModels = value
All of this code is intended to accomplish a few key checks:
  • The initial incoming value is expected to be a dict, or a subclass of one — the process needs the keys and values in order to support multiple different values for multiple tags, after all.
  • Any tag-name specified must be a valid tag-name. I'm going to look for a better way to make that determination while I'm working out the Tag class, I'm sure, but for now the simple checks I have in place will suffice — though I won't really be able to fully unit-test the property until I get that resolved.
  • Any tag-name specified must have a valid rendering-model value — it must be a member of the renderingModels enumeration defined earlier.
If this seems like an awful lot of code to write when it should be possible to just set self._tagRenderingModels = value, I'd point to
Raise errors as close to their ultimate source as possible
and
If specific types are expected, test for those where they're expected
from my coding standards. This, I think, is a really good example of why I think those are important — Without those checks, it would be possible to dump pretty much any value as a rendering-model for any tag in a namespace. It would almost certainly be possible to have the rendering-process for Tag instances check for valid values, but I'm pretty sure that those processes are going to be complex enough without adding that sort of checking to them. Even if those checks were made during rendering, and were to raise an error of some sort, that wouldn't necessarily help identify where the source of the error was.

I'm not going to take the time to fully populate the Namespace constants in markup just yet — doing so would require digging through all the tag-level documentation for HTML 5 and XHTML, at a minimum, and though it needs to be done, it doesn't need to be done right now, I think. I am, however, going to stub those constants out so that they're available (using a bogus URI for HTML 5, as noted earlier):

#-----------------------------------#
# Default Namespace constants       #
# provided by the module.           #
#-----------------------------------#

# HTML 5 namespace
HTML5Namespace = Namespace(
    'http://www.w3.org/2015/html', 
    br=renderingModels.NoChildren,
    img=renderingModels.NoChildren,
    link=renderingModels.NoChildren,
    )
__all__.append( 'HTML5Namespace' )

# XHTML namespace
XHTMLNamespace = Namespace(
    'http://www.w3.org/1999/xhtml', 
    br=renderingModels.NoChildren,
    img=renderingModels.NoChildren,
    link=renderingModels.NoChildren,
    )
__all__.append( 'XHTMLNamespace' )

I'm also going to defer finishing out the unit-tests for Namespace until I have Tag implemented, if only so that I can use the tag-name validation that it'll require to also check tag-names in the _SetTagRenderingModels method. I'm accumulating testing tech-debt by doing that, but I'm at the point where I'd rather get all of the unit-testing resolved at once after Tag is done, and the markup module is more complete. Unit-test stubs for Namespace and other recent items have added nearly 40 new test-methods to the mix, and six new failures:

########################################
Unit-test results
########################################
Tests were successful ... False
Number of tests run ..... 104
 + Tests ran in ......... 0.01 seconds
Number of errors ........ 0
Number of failures ...... 21
########################################

At this point, the next logical item to tackle is the Tag class. That has the potential to be a really long post (Tag has to implement all of the DOM-element functionality listed in the first markup-module post, and there's a lot there). That's way more than I feel comfortable with tackling in today's post.

On the off chance that there's any interest in the XML page-template that I started with today, I'm making that available for download: