Project Structure

With the documentation-decorators ready to be included in my code-templates, it's time to take a look at those. There's a circular-reference kind of thing that I have to resolve first, though — In order to actually use the decorators, they have to import the module that the describe class lives in:

# Make sure the documentation-decorators are available!
from doc_metadata import describe

In order to do that, the Python interpreter has to know where the doc_metadata.py file lives. That, in turn, means that I need to decide where I'm going to put it, which means that I need to drop it into a project.

I'll start a new project for the framwork-level code that doc_metadata.py (arguably) belongs to. That project will be named idic, and it will follow the structure shown below, with any of the duplicated project-directories removed.

A Basic Project-Structure

I'm expecting all of the projects on my current list to share a common folder structure. This structure is based on the final deployed/installed location for project-files in an Ubuntu Linux (and presumably POSIX-compliant) file-system, and looks something like this:

File-system Path
[Project Root Directory]
	etc
		apache2
			sites-available
	usr
		local
			bin
				[project-name]
			etc
				idic
					datastores
			lib
				idic
					[project-name]
		share
			doc
				[project-name]
			icons
				[project-name]
			[project-name]
	var
		cache
			[project-name]
		www
			[project-name]
				media
				scripts
				styles

The directories in this structure have the following roles a the project once it is deployed:

etc

Items to be deployed to the global (root-access-only) configuration-file directory

etc/apache2/sites-available

Configuration-files for Apache websites.

usr

Items to be deployed to the global (root-access-only) usr directory

usr/local/bin/[project-name]

Executable files associated with the project (command-line applications and scripts).

usr/local/etc

Standard local-machine configuration-files directory

usr/local/etc/idic

Configuration-files directory for idic applications

usr/local/etc/idic/datastores

Configuration-files for machine-resident idic data-stores (database-connection credentials)

usr/local/lib

Standard local-machine shared-libraries directory

usr/local/lib/idic

Shared-library directory for all idic-project codebases

usr/local/lib/idic/[project-name]

Shared-library directory for all project-specific shared-library codebases (the top-level package directory for a project's namespace)

usr/share

Standard machine-specific shared-resources directory

usr/share/doc/[project-name]

Documentation-directory for the project

usr/share/icons/[project-name]

Icon-directory for the project

usr/share/[project-name]

Directory for other shared resource-files originating with the project

var

Items to be deployed to the global (root-access-only) var directory

var/cache/[project-name]

Space for storing (reading and writing) cache-files for the project.

var/www/[project-name]

The document-root directory for a website associated with the project

var/www/[project-name]/media

Image (and other media) files for the project website

var/www/[project-name]/scripts

Client-side scripts for the project website

var/www/[project-name]/styles

Style-sheets for the project website

Some projects may not have the entire directory-tree shown — projects that do not include any websites, for example, will not have the var/www/* portion of the tree, nor the site-configuration items in etc/apache2/*. The build-process that I'm planning to create for the projects I cover here will allow variations of files to be defined in the project workspace for each of several distinct environments.

Once I start digging in to the packaging- and deployment-processes part of the build-process I'll be creating, I expect that many of the top-level directories (/etc, /usr and /var) will need some conditional installation/set-up — This structure will work fine if the installer has root access, but not so much for, say, installers wanting to use the codebase on a hosting-provider machine that they don't have root access on (and cannot convince the provider to install the project[s]). Since I don't know yet what the packaging mechanisms are going to be, I'm going to shelve that for the time being, knowing that I'll have to look at it again.

All of the directories shown as [project-name] would be renamed to reflect the actual project-name. If multiple project-name directories need to exist (for example, if a project provides both a client-facing and administrative website), they should be named accordingly, with the project's name in the directory-name somewhere.

I'm planning to provide a structure that would accommodate a perhaps-typical development-and-migration set-up, and will keep track of files that belong to each specific environment by prefixing them with the environment's name. These environments, their purpose/role, and an example environment-specific file name (I picked a website configuration-file as an example) are:

LOCAL-*

The LOCAL environment is one that exists on the developer's machine, that is built and deployed to frequently as part of the code-test-debug cycle of ongoing/daily development.

Example: /etc/apache2/sites-available/LOCAL-site.conf

DEV-*

A DEV environment is the first shared environment accessible to all project-developers. The stability (or perhaps even existence) of a deployed project-copy should not be taken as a given — developers may well destroy and rebuild some or all of a project on its dev-environment over and over again every few minutes while working out integration of code from multiple developers.

Example: /etc/apache2/sites-available/DEV-site.conf

TEST-*

A TEST environment is where integrated and hopefully production-ready code is deployed to for purposes of quality assurance and maybe user acceptance. It may or may not have a complete application data-set available, but if it does, that data is probably a mirrored copy of the live environment's data, not the live data-set itself.

Example: /etc/apache2/sites-available/TEST-site.conf

STAGE-*

A STAGE (or STAGING) environment is the final stop before a project is deployed to a live environment. Ideally, the staging system(s) will be running on identical hardware, and may have direct access to live-environment application data-sets. A staging environment is often where load-testing is undertaken.

Example: /etc/apache2/sites-available/STAGE-site.conf

LIVE-*

The final live/production environment, where an application is available to all of the users it's intended to be available to.

Example: /etc/apache2/sites-available/LIVE-site.conf

So, to be clear, the example files noted above would exist side by side in a project's directory-tree like so:

File-system Path						Purpose
etc
	apache2
		sites-available
			DEV-site.conf			Dev-environment's site-configuration.
			LIVE-site.conf			Live-environment's site-configuration.
			LOCAL-site.conf			Local-environment's site-configuration.
			STAGE-site.conf			Stage-environment's site-configuration.
			TEST-site.conf			Test-environment's site-configuration.

The build-process for a given environment would then remove all of the files whose names indicate they are not part of the build for that environment (e.g., a LOCAL build would remove DEV-*, TEST-*, STAGE-* and LIVE-* files), and rename the LOCAL-* files to remove their LOCAL- prefixes, leaving, for example, a site.conf file that is specifically aimed at the environment the build is being created for. This is a pretty brute-force process, but it's simple, easily understood (I think), and relatively easily managed in Makefile code.

Each project would also have its own Makefile (not shown) in the top-level project-directory. I'm also expecting a few installation-scripts at a minimum to reside at the same level, but that will depend on the packaging-mechanism decision I mentioned in my previous post, so I'm not showing (or committed to) any specific structure or approach just yet.

I'm not completely sure, but I suspect that his project-structure would work for any project-type (e.g., for projects that aren't centering around a Python codebase). Since I don't really have anything in my project-queue that isn't a Python project, I'm happy enough to use this structure until/unless I find that I need to vary it for some other project.

Oh... And since doc_metadata.py belongs to the idic project (and eventually namespace), it will live at

[project-root]/usr/local/lib/idic/doc_metadata.py

That feels like enough for this post, particularly after all of the longer ones of late. The next few posts will delve into module- and package-file templates, then I'll probably generate and share templates or snippets for classes and other language-level elements that it makes sense to have them for.