Where to Go From Here?

Having spent the last several weeks now on stuff that, while necessary, just isn't fun or sexy from a coding perspective, I'm looking to start getting into some parts of the framework that are actually usable for writing web-applications. As I see it, there are several moving parts that are common to most web-apps, even setting aside the integration with a web-server. I'm not certain which I'll pick up next, though, so I figured I'd write about some of them and see if anything jumped to mind as a logical next step.

Generation (and Parsing) of Markup

Web-applications are all about pages, ultimately, even if those pages are very nearly empty to start with and populated with an AJAX call or whatever other process might surface in the future. In turn, pages are all about markup, whether it's HTML5, XHTML, or XML.

Having spent most of the last several years writing application-code for a creative agency, I firmly believe the idea of separation of markup/structure from functionality/logic has merit — to the point that one of my goals for this framework is to make it as easy as possible to keep that separation, while still allowing as much designer-level control as possible over the structure and appearance of pages. If you're thinking that sounds like a tall order... well, you may be right. Still, I've worked out ways and means of getting a fair part of the way to that goal, and I think it's quite do-able, though it might take some work.

If I'm brutally honest about it, I suspect that my preference for that sort of separation probably stems, at least in part, from several years of working with web CMSs, where the separation of content from markup was a key concept. There have been other influences that push me in that direction, though.

Setting that aside for the moment, I have to say also: I've looked at several Python markup libraries over the years, and have yet to find one that I really like. My perfect solution would include:

• The ability to represent all of the different and varied node-types in any of the markup languages that I'm interested in or expect to use. Right now, that would include:
  • Tags (of course);
  • Documents (which are, more or less, just extensions of Tags);
  • Text-nodes (also of course);
  • Comments; and
  • CDATA nodes (mostly for the sake of a complete set of node-types).
• The ability to extend Tags out into specific types of tags in specific languages. This would include things like HTML <form>-elements and all the various <input>-tags within those forms, at a minimum. Part of the reasoning behind that desire is the potential for being able to attach server-side validation to individual user interaction elements on a page, so that, for example, submission of a bad date-value through an <input type="date"> field will automatically flag the submission as invalid.
• That ability to extend Tags is also something that I hope to use to generate page components — Items that can be included in a page through markup and by non-programmers to allow folks with more of a graphic-design kind of focus to still be able to work with pages that use the framework with little or no technical assistance.
• A reasonably complete set of DOM functions. More specifically, I'd like programmatic manipulation of markup-objects to be do-able and I'd like it to mirror the set of capabilities that a lot of web-developers already know through JavaScript. There are some additions that I'd like to make as well (being able to tell an element to remove itself from its parent, for example), but on the whole, if it's do-able with JavaScript on the client side of the exchange, I want it to also be do-able server-side, and I want it to use the same property, function- or method-names unless there's a compelling reason that it can't.
• The ability to parse static markup, whether in the form of a template for an entire document, or some chunk of markup that's used for templating at a smaller scale, into a server-side object-set. That capability allows the server-side application-logic to make changes to some or all of a document before it gets rendered and sent back to the browser client. That capability opens up a lot of flexibility in creating web-applications, I believe.

There may be other considerations that surface later, but these are the ones that, at present, are my high-priority criteria — must-haves, mostly, with a few should-haves sprinkled here and there through the hypothetical stack.

I expect that to be a pretty daunting effort. Without planning out any internal detail or functionality, and before dealing with any of the variations between different markup languages, I believe that all of this requires a pretty substantial class-structure:

Then, too, in order for this to actually be usable, I'd also need to start digging in to the process(es) of connecting a web-server request to a markup-generation response. There are a couple of options to pick from, as I mentioned some time ago, but pursuing any of them will take some time as well.

But the results would be visible. Not a small consideration, perhaps.

A DAL and Data-Objects Implementation

Most web-applications also have some sort of data-access layer too. That's what makes them applications, in many cases — the ability to interact with data that persists across users and sessions. While Python's got a solid set of database-connection modules: there are several for MySQL and PostgreSQL, at least one for ODBC connections, and I'm sure there are several for the various NoSQL data-stores. All of them that I'm aware of conform to the Python Database API Specification, which makes data-access across those different engines pretty consistent.

What I'd like to see on top of that is some consistency in how database calls are actually made. In particular, I feel that if there are objects in an application's structure whose state-data is persisted in a back-end data-store, I'd very much like for those data-objects to be able to manage their own state, without the code having to explicitly check to see if they need to be created, updated or deleted in the data-store. There's also some common access-functionality that I believe would be useful to attach to all instances of data-objects, or, in some cases, to the classes that define those data-objects in the context of an application.

Data-access can be relatively expensive in an application — a connection to the database has to be made, queries executed, and results gathered and returned, and maybe formatted before the data is really ready for the application to make use of it. The data-object strategy may help streamline those data-access processes, at least somewhat, but something else that could help would be allowing the data-access layer to be able to cache queries' results, at least for the duration of a page request/response cycle. Another possible gain would center around a more lazy loading sort of data-access — one where queries aren't actually executed until their results are needed. I'm not sure that is viable (or even desirable — I'll want to think on that more), but it's something that I'd like to at least consider in a DAL.

Since data-stores are going to have different properties from one another (server and credentials at the minimum), but different instances of data-stores that connect to the same server with the same credentials is possible, that strikes me as a good justification to make them configurable. If I pursue that line of thinking, that means that I'll have to start thinking about how I want to set up configuration, then.

Even with that, I think it'll be less complex, structurally, than the markup module, though — the number of classes is about the same, but there's more items that derive from a single, common base class than in the markup structure, and I'm not expecting the base classes in the DAL space to have nearly the same method-member count as I think will exist in the markup base classes.

Though that ignores the configuration aspects of individual data-stores.

So, Where Next?

After all this, I think I've arrived at a path to continue with:

• Creating the markup module — There's at least the potential that writing all of that out will occupy a lot of time, if only because there's a lot of functionality that will surface, but it will lead nicely, I think, into...
• Working out details on how to connect parsed-, generated- and application-manipulated markup to a web-server, so that it can be displayed.
• Creating the data_access module feels like a solid next step after those — with this in place, the beginnings of real application functionality can, I think, start to take shape, leading to...
• Design and implementation of a page-component model that will facilitate the sort of content/logic separation that I'm aiming for.
• That, in turn, will allow me to take a serious look at creation of the interaction elements I mentioned earlier, though I may have to think out details on server-side validation before that, or at least enough of the fundamentals of validation to be able to work on validation of interaction elements concurrently.