A Repeatable Build Process [2]

So now that I know what I want my Makefile to actually do, it's time to actually write one for the current project, as well as create a template Makefile for use with future projects (though I expect it will be a while until I have need of it elsewhere).

Common Header Variables

There were four common variables across the two build-targets that I noted in my previous post:

build_directory:

The temporary build-directory (e.g., /tmp/build-process);

current_builds:

The local user-directory where final build-results will be dropped (e.g., ~/Current_Builds);

project_copy:

The copy of the original project_directory (below), created inside the build_directory, where deployable files will be gathered in order to be archived into the final package-tarball (e.g., /tmp/build-process/{project_name});

project_directory:

The original project-directory (e.g., ~/IDreamInCode/{project_name});

project_name:

The name of the project (e.g., idic);

Having given them some thought (and done a bit of research), I'm going to drop those into the top of the Makefile. I'm also going to add a project-root variable that'll eventually be used for dependent-project build-processing:

#############################
# PROJECT SETTINGS          #
#############################

# The root directory for all projects that use this build-
# process -- MUST be the same if project-dependencies are 
# going to be preserved!
PROJECT_ROOT=~/IDreamInCode

# The final destination directory for all project-build 
# results. This is local to the user!
CURRENT_BUILDS=~/Current_Builds

# The name of the project being built. It'd be nice to 
# have this auto-detected in some way, but for now, I'm 
# OK with it being a "magic" string value
PROJECT_NAME=idic

# The directory that will be created to house all of the 
# temporary files for the build-process
BUILD_DIRECTORY=/tmp/$(PROJECT_NAME)-build

# The directory that will be created to house all of the 
# temporary files for the build-process
PROJECT_COPY=$(BUILD_DIRECTORY)/$(PROJECT_NAME)

# The root directory of the PROJECT - assumes that all 
# projects reside in the $(PROJECT_ROOT) directory, 
# in a directory named after the project.
PROJECT_DIRECTORY=$(PROJECT_ROOT)/$(PROJECT_NAME)

#############################
# BUILD SETTINGS            #
#############################

#############################
# MAIN BUILD TARGETS        #
#############################

#############################
# COMMON BUILD-TASK TARGETS #
#############################

The values of these project-settings variables, defined as they are in the head of the Makefile and outside (before) any of the targets allows the to be set and available inside the targets. For example, adding this target to the current Makefile:

show_vars:
    @echo "- show_vars ----------------------------"
    @echo "BUILD_DIRECTORY ..... $(BUILD_DIRECTORY)"
    @echo "CURRENT_BUILDS ...... $(CURRENT_BUILDS)"
    @echo "PROJECT_COPY ........ $(PROJECT_COPY)"
    @echo "PROJECT_DIRECTORY ... $(PROJECT_DIRECTORY)"
    @echo "PROJECT_NAME ........ $(PROJECT_NAME)"
    @echo "PROJECT_ROOT ........ $(PROJECT_ROOT)"
    @echo "- /show_vars ---------------------------"

and running make (with the new target as the default target), yields:

- show_vars ----------------------------
BUILD_DIRECTORY ..... /tmp/idic-build
CURRENT_BUILDS ...... ~/Current_Builds
PROJECT_COPY ........ /tmp/idic-build/idic
PROJECT_DIRECTORY ... ~/IDreamInCode/idic
PROJECT_NAME ........ idic
PROJECT_ROOT ........ ~/IDreamInCode
- /show_vars ---------------------------

I'll change and re-use the show_vars target as I work through the other examples here... if output shown here has

- show_vars ----------------------------

in it, it originated with show_vars.

Those same variables can also be used as parts or all of target-level variables. By way of example, consider the starting-point for the local_all target:

local_all: ENV_PREFIX=LOCAL
local_all: BUILD_PATHS=$(BUILD_PATHS_ALL)
local_all: BUILD_OUTPUT=$(CURRENT_BUILDS)/$(ENV_PREFIX).$(PROJECT_NAME)*
local_all:
    # TODO: Implement recursive build for projects that 
    #       this project uses: 
    #       cd $(PROJECT_ROOT)/[project];$(MAKE) local_all
    # TODO: Determine if there is anything else that needs
    #       to be done for a $(ENV_PREFIX) build here.
    # Build complete for $(ENV_PREFIX) version of $(PROJECT_NAME) project:
    # + Source ... $(PROJECT_DIRECTORY)/[$(BUILD_PATHS)]
    # + Output ... $(BUILD_OUTPUT)

If make local_all is executed, it yields:

# TODO: Implement recursive build for projects that 
#       this project uses: 
#       cd ~/IDreamInCode/[project];make local_all
# TODO: Determine if there is anything else that needs
#       to be done for a LOCAL build here.
# Build complete for LOCAL version of idic project:
# + Source ... ~/IDreamInCode/idic/[etc usr var]
# + Output ... ~/Current_Builds/LOCAL.idic*

This also shows that the target-level variables can be used to build other variables-values for the target — note the results of $(BUILD_OUTPUT) in the output, whose value originated from the global $(CURRENT_BUILDS) and $(PROJECT_NAME) and the target-level $(ENV_PREFIX) values... The target-level variables also carry through to any targets called afer they are defined. That is, if the local_all target calls show_vars (and it's modified to display the current value of $(ENV_PREFIX)), it's set to LOCAL inside show_vars, just like it is in local_all:

- show_vars ----------------------------
BUILD_DATETIME ...... Mon Jan  9 14:18:48 MST 2017
BUILDER_NAME ........ ballbee
BUILD_PATHS_ALL ..... etc usr var
BUILD_PATHS_APP ..... usr var/cache
BUILD_PATHS_SITE .... etc/apache2/sites-available var/www
BUILD_PATHS_SNAP .... etc test_idic usr var
ENV_PREFIX .......... LOCAL
- /show_vars ---------------------------

Stubbing Out the COMMON BUILD-TASK TARGETS

At this point, my expectation is that the local_all target will actually get most of its work done by calling other targets, so I'm going to sub out those targets first, include them in the chain for local_all, run local_all to see what happens, tweak things if/as needed, and keep iterating over that process until all of those targets are showing up when local_all is executed, and displaying what I need to see.
The first step, then, is generating targets that at least display that they've been called. That doesn't require anything fancy, just the collection of targets and a comment in each of them to the effect of what target they are:

#############################
# COMMON BUILD-TASK TARGETS #
#############################

add_snapshot_files:
    # Calling add_snapshot_files

base_all:
    # Calling base_all

base_app:
    # Calling base_app

base_site:
    # Calling base_site

build_directory:
    # Calling build_directory

clean_build_directory:
    # Calling clean_build_directory

clean_executables:
    # Calling clean_executables

copy_to_current_builds:
    # Calling copy_to_current_builds

create_documentation:
    # Calling create_documentation

create_install_scripts:
    # Calling create_install_scripts

create_snapshot_zip:
    # Calling create_snapshot_zip

create_tarball:
    # Calling create_tarball

current_builds:
    # Calling current_builds

fix_environment:
    # Calling fix_environment

test:
    # Calling test

With that in place, calling all of the relevant targets for each of the two main targets (snapshot and local_all) is also pretty straightforward, if a bit long:

#############################
# MAIN BUILD TARGETS        #
#############################

snapshot: ZIP_FILE_NAME=$(PROJECT_NAME).zip
snapshot: BUILD_OUTPUT=$(CURRENT_BUILDS)/$(ZIP_FILE_NAME)
snapshot: BUILD_PATHS=$(BUILD_PATHS_SNAP)
snapshot: BUILD_ZIP=$(BUILD_DIRECTORY)/$(ZIP_FILE_NAME)
snapshot: current_builds build_directory base_all add_snapshot_files clean_build_directory create_snapshot_zip
    # Build complete for SNAPSHOT of $(PROJECT_NAME) project:
    # + Source ... $(PROJECT_DIRECTORY)/[$(BUILD_PATHS)]
    # + Output ... $(BUILD_OUTPUT)

local_all: ENV_PREFIX=LOCAL
local_all: BUILD_PATHS=$(BUILD_PATHS_ALL)
local_all: BUILD_OUTPUT=$(CURRENT_BUILDS)/$(ENV_PREFIX).$(PROJECT_NAME)*
local_all: current_builds build_directory base_all clean_build_directory clean_executables fix_environment create_tarball create_install_scripts copy_to_current_builds
    # TODO: Implement recursive build for projects that 
    #       this project uses: cd $(PROJECT_ROOT)/[project];$(MAKE) local_all
    # TODO: Determine if there is anything else that needs
    #       to be done for a $(ENV_PREFIX) build here.
    # Build complete for $(ENV_PREFIX) version of $(PROJECT_NAME) project:
    # + Source ... $(PROJECT_DIRECTORY)/[$(BUILD_PATHS)]
    # + Output ... $(BUILD_OUTPUT)

Calling make snapshot and make local_all then return, respectively:

# Calling current_builds
# Calling build_directory
# Calling base_all
# Calling add_snapshot_files
# Calling clean_build_directory
# Calling create_snapshot_zip
# Build complete for SNAPSHOT of idic project:
# + Source ... ~/IDreamInCode/idic/[etc test_idic usr var]
# + Output ... ~/IDIC_Builds/idic.zip

# Calling current_builds
# Calling build_directory
# Calling base_all
# Calling clean_build_directory
# Calling clean_executables
# Calling fix_environment
# Calling create_tarball
# Calling create_install_scripts
# Calling copy_to_current_builds
# TODO: Implement recursive build for projects that 
#       this project uses: cd ~/IDreamInCode/[project];make local_all
# TODO: Determine if there is anything else that needs
#       to be done for a LOCAL build here.
# Build complete for LOCAL version of idic project:
# + Source ... ~/IDreamInCode/idic/[etc usr var]
# + Output ... ~/IDIC_Builds/LOCAL.idic*

which at least demonstrate that the required sub-targets are firing as expected.

Putting some Meat in the Sub-Targets

So now it's time to actually implement the sub-targets. Since I'm not currently concerned about the items in local_all, I won't address any of the sub-targets here that are only part of that in any real detail — I'll need to finish those out at some point, but until I've figured out what an installation-script needs to do and look like, no environment-oriented build is really relevant, not even the local one. That leaves, in order of execution in the Makefile for a make snapshot build that I'll discuss in this post:

• current_builds;
• build_directory;
• base_all;
• add_snapshot_files;
• clean_build_directory;
• create_snapshot_zip; and
• any changes or additions to the main snapshot target that arise as a result of any of the earlier targets in execution order.
• target_name

To be clear: I'm planning on implementing the other targets, if only so I have a (hopefully) complete local-build-ready Makefile, but I'm not going to show or discuss them in any great detail. However far I do get with them, they'll be in the template Makefile, and that'll be downloadable at the end of this post.

The current_builds Target

Ultimately, the current_builds target is really only responsible for creating the final destination-directory that all of the top-level build-processes will drop their final output in. It should hopefully be no great surprise, then, that it's not a terribly complicated target. It does need the CURRENT_BUILDS variable set, but that happens at the top of the Makefile, so that's already accounted for. The complete target is:

current_builds:
    # Creating local current-builds directory
    #   at $(CURRENT_BUILDS)
    @mkdir -p $(CURRENT_BUILDS)

One item that's worth mentioning here because it'll show up elsewhere later: One of the things that I find I don't like about make is that all of the commands that get executed in a target (like the mkdir -p $(CURRENT_BUILDS) here) get echoed out to the console during a make run by default. I'm guessing that in a more normal build-process, say compiling and assembling a C or C++ program, there may be some advantage to seeing the commands that were run as well as any output from them. I don't expect that most of what I'm going to be doing in any of these targets really needs that degree of visibility, though, so I almost always hide the commands being executed (and their output) by default. That's what the @ in the @mkdir -p $(CURRENT_BUILDS) line does. I usually figure that if I need (or want) output, it's easy enough to generate it where or if I determine I do want it.

The build_directory Target

build_directory is just as simple a target as current_builds, because it does much the same thing, just with a different variable (BUILD_DIRECTORY, also defined in the head of the Makefile) controlling the path of the build-directory to be created:

build_directory:
    # Creating build-directory
    #   at $(BUILD_DIRECTORY)
    @mkdir -p $(BUILD_DIRECTORY)

The base_all Target

This is where things start to get interesting, I think...
One of the main tasks that this build-process has to do is to assemble the project's code into a safe location for further processing. In order to do that, it has to know several things about the project, all of which are based around the project-structure:

• Where the project itself lives (the PROJECT_DIRECTORY variable);
• Where the safe code-copy is to be made (the PROJECT_COPY variable); and
• What directories in the project need to be copied (the BUILD_PATHS variable)

PROJECT_DIRECTORY and PROJECT_COPY are set in the head of the Makefile, but BUILD_PATHS is created by the various top-level targets:

#############################
# MAIN BUILD TARGETS        #
#############################
# ...
snapshot: BUILD_PATHS=$(BUILD_PATHS_SNAP)

# ...

local_all: BUILD_PATHS=$(BUILD_PATHS_ALL)

# ...

They, in turn, refer to other variables (BUILD_PATHS_SNAP and BUILD_PATHS_ALL) that are also defined in the Makefile's head.
base_all also marks the first time that I'm writing a build-target that uses programs available in the external shell. I ran into some oddities while I was working this target out, probably because of either something new in my version of make (unlikely), or some misunderstanding on my part about what a target has access to in a Makefile. Discussion of that will make more sense after I show the target, though, so here it is:

base_all:
    # Creating project-copy directory: 
    #   at $(PROJECT_COPY)
    @mkdir -p $(PROJECT_COPY)
    # Copying $(BUILD_PATHS) 
    #      to $(PROJECT_COPY):
    @for PATH in $(BUILD_PATHS); do \
        /bin/cp -r $(PROJECT_DIRECTORY)/$$PATH \
            $(PROJECT_COPY)/$$PATH; \
        echo "# + $(PROJECT_NAME)/$$PATH copied to $(PROJECT_COPY)/$$PATH"; \
    done

Like the previous two targets, base_all creates a directory — this one being the place to copy all of the project-code, defined by the PROJECT_COPY variable, which is defined in the head of the Makefile. That's nothing too new, since it's been done before.
The next step is where things got a bit odd. The target then uses a shell-loop to iterate over the BUILD_PATHS (also defined in the head), and makes a copy of the directory, with all of its children, in the PROJECT_COPY directory, before displaying messaging indicating that the copy succeeded. The odd thing, I thought, was that the cp shell-utility couldn't be found within the target while it was running. That is, with

base_all:
    # ...
    
    @for PATH in $(BUILD_PATHS); do \
        cp -r $(PROJECT_DIRECTORY)/$$PATH \
            $(PROJECT_COPY)/$$PATH; \
        echo "# + $(PROJECT_NAME)/$$PATH copied to $(PROJECT_COPY)/$$PATH"; \
    done

in the target-code, execution died in the middle of that target, reporting

/bin/sh: 2: cp: not found

I have no idea why. I've used similar command-structures (though usually calling rsync instead of cp) before and never run into this issue. Nor, frankly, does there appear to be any mention of this sort of bugaboo anywhere that I could find with a Google search. To be fair, I may not have hit on the right search-terms in my efforts, but even so — no mention of this at all leads me to think that there's something odd going on.
Still, fortunately, I could access the cp command by providing the full shell-path to it (found with which cp in a terminal). I'd like to figure out why a normal cp call is barfing, especially since the echo and mkdir commands weren't having issues, but the workaround that's in place now will do for me, even if it does make some assumptions about the system that builds are being generated on.
With this target operational, the build-output is starting to look close to complete:

# Creating local current-builds directory at ~/IDIC_Builds
# Creating build-directory at /tmp/idic-build
# Creating project-copy directory: 
#   at /tmp/idic-build/idic
# Copying etc test_idic usr var 
#      to /tmp/idic-build/idic:
# + idic/etc copied to /tmp/idic-build/idic/etc
# + idic/test_idic copied to /tmp/idic-build/idic/test_idic
# + idic/usr copied to /tmp/idic-build/idic/usr
# + idic/var copied to /tmp/idic-build/idic/var
# Adding snapshot files and directories to /tmp/idic-build/idic

and the build-copy directory is starting to show the source-files:

So far, so good...

The add_snapshot_files Target

As things stand right now, only one file of three (or more) that I expect to be part of any given project actually exist in my current project-structure template: the runTests.py file that will be called by the test target. Since the purpose of a snapshot build is to copy the entire active codebase for dissemination here on the blog, it needs to include all the files that might be considered formal members of the project, which will, eventually, include the Makefile that I'm working on now, any installation-scripts for the project, and maybe a text-file report of the output of the last unit-test run. Not all of these files will exist at any given time, though at least two can be specified right now:

#############################
# PROJECT SETTINGS          #
#############################

# ..

# The list of all additional files that need to be added 
# to a snapshot build's file-set
SNAPSHOT_PATHS=runTests.py $(PROJECT_NAME)-test-results.txt
# TODO: Once the other files that belong to all projects 
#       have been defined, add them to the list here, 
#       including:
#       - Makefile
#       - install-scripts
#       - Others?

Having the SNAPSHOT_PATHS defined allows the add_snapshot_files target to be fully defined:

add_snapshot_files:
    # Adding snapshot files and directories to $(PROJECT_COPY)
    @for PATH in $(SNAPSHOT_PATHS); do \
        if [ -f "$(PROJECT_DIRECTORY)/$$PATH" ]; then \
            /bin/cp -r $(PROJECT_DIRECTORY)/$$PATH $(PROJECT_COPY); \
            echo "# + $(PROJECT_NAME)/$$PATH copied to $(PROJECT_COPY)"; \
        else \
            echo "# + $(PROJECT_NAME)/$$PATH skipped"; \
        fi; \
    done

The output of the add_snapshot_files target specifically lists each file in the SNAPSHOT_PATHS list, so that it can report on the finsl disposition of each one individually:

# Adding snapshot files and directories to /tmp/idic-build/idic
# + idic/runTests.py copied to /tmp/idic-build/idic
# + idic/idic-test-results.txt skipped

The clean_build_directory Target

Deploying or distributing compiled .pyc files (and presumably .pyo files as well) can be troublesome. If the original .py file that a .pyc file was compiled from hasn't been modified since the last time the .pyc was compiled, it may well contain references to paths on the original system that it was compiled on. While I've never seen that cause any actual errors, I have seen cases where an unrelated error in the code raises an exception, and the traceback that accompanies it displays those original-system paths. That may not be a big deal. It wasn't anything more than an annoyance to me the few times I encountered it, but then I knew the codebase that the error resided in pretty intimately, and had no great difficulty translating from the live error module-paths to the stored paths that originated on my local development machine. Someone less familiar with a codebase than I was might well get more than a little frustrated trying to track down the source of a bug when they could be spending the time fixing it. Since I'm writing Python code, and there's no need for me to distribute .pyc or .pyo files, there's no point in keeping them in the file-set for any build at present. That is what the clean_build_directory target is all about:

clean_build_directory:
    # Removing files that shouldn't be part of a build
    #  - Removing all files matching: $(REMOVE_FILE_TYPES)
    @for FILE_SPEC in $(REMOVE_FILE_TYPES); do \
        for FILE_NAME in `find $(PROJECT_COPY) -name "$$FILE_SPEC"`; do \
            rm $$FILE_NAME; \
            echo "#    + Removed $$FILE_NAME"; \
        done; \
    done
    # - Removing individual files: $(REMOVE_FILES)
    @for FILE_SPEC in $(REMOVE_FILES); do \
        for FILE_NAME in `find $(PROJECT_COPY) -name "$$FILE_SPEC"`; do \
            if [ -f "$$FILE_NAME" ]; then \
                rm $$FILE_NAME; \
                echo "#    + Removed $$FILE_NAME"; \
            fi; \
        done; \
    done

Structurally, at least, clean_build_directory isn't much different from add_snapshot_files. The same sort of iteration over a list of files is present (twice, admittedly). The main differences are in how those files are identified, and what happens to the individual files during the iteration. Instead of copying a file into the project-copy directory from the original project-directory, it's removing files from the project-copy. Identification of the files to be removed is through the system's find utility, allowing patterns to be specified in REMOVE_FILE_TYPES and specific files in the REMOVE_FILES list.

The create_snapshot_zip Target

At this point, all that's remaining to do is to generate the final snapshot ZIP file, and maybe do a little clean-up (removing the build-directory, since there's nothing left to do with it):

#create_snapshot_zip:
    # Creating snapshot ZIP file:
    @cd $(BUILD_DIRECTORY);zip -qr $(BUILD_OUTPUT) $(PROJECT_NAME)
    # Removing the build-directory
    @rm -fR $(BUILD_DIRECTORY)

Final Check of the snapshot Target

Since the create_snapshot_zip target creates the final ZIP-file in it's final location, the BUILD_ZIP value that was originally in the snapshot target was no longer needed. It was originally going to be used to generate an intermediary ZIP-file in the build-directory, but the intermediate step wasn't necessary, so I've removed it:

snapshot: ZIP_FILE_NAME=$(PROJECT_NAME).zip
snapshot: BUILD_OUTPUT=$(CURRENT_BUILDS)/$(ZIP_FILE_NAME)
snapshot: BUILD_PATHS=$(BUILD_PATHS_SNAP)
snapshot: current_builds build_directory base_all add_snapshot_files clean_build_directory create_snapshot_zip
    # Build complete for SNAPSHOT of $(PROJECT_NAME) project:
    # + Source ... $(PROJECT_DIRECTORY)/[$(BUILD_PATHS)]
    # + Output ... $(BUILD_OUTPUT)

In the interests of showing where things have landed with being able to make snapshots from a build-process, I'm saving a copy of the current Makefile template, before I start tackling any of the local_* build-targets, that can be downloaded and examined as you see fit. I've also got a Makefile in the idic project now, one that's slightly further along than what I've shown in this post. It's also available for download below, as an item in the snapshot that it built.
I don't think I'll pursue the local_* builds much further than I have for now — until I get to a point where I need to be able to generate a local installation of some sort, I'd have no way to meaningfully test the accuracy of the process. With where I am in my efforts on the idic framework so far, that's likely to be a while, a point that I'll think over and discuss in my next post.

Makefile template

7.9kB

idic.zip

43.4kB

A Repeatable Build Process [1]

In one of my earliest posts, I said that...

... I think that even a bare-bones build-process should:

• Run automated tests and stop if any tests fail;
• Generate notifications if the tests fail;
• Package the build(s) in some fashion so that it's ready to be deployed;
• Generate notifications if a build fails for reasons other than test-failures; and
• Deploy to an environment where the current build can be executed.

Now that there's more than one file in the codebase that I'm building, I have the beginnings of a repeatable test-process thought out, and I'm generating the snapshots that appear in the right sidebar from time to time, I think it's about time for me to start digging in to a repeatable build-process that can, at a minimum, run tests (and generate test-results reports), and package up those snapshots. Eventually, once I have time to give some thought to the various packaging-solution options, and pick one or more out, this build-process will have to be revisited. I won't get any further down that list than the first three items for now — and only part way through the third, if I'm honest about it — but that will get me far enough for the time being. And, at least for now, I'll be building the codebase out with GNU Make.

Why make? Why not {insert-tool-name}?

As I noted in that same earlier post, it seems to me that there are a truly ridiculous number of build-processes available these days. So why am I using something as old and primitive as make? Why not use grunt, or ant, or puppet or salt or any of the other fifty- or sixty-odd possibilities that can be found quickly and easily?

A good part of that decision is based around various needs that I know I'll eventually have to cater to:

• Multiple build-output formats (more or less in order of priority for me):
  • A snapshot ZIP-file for inclusion here, whose build-process allows a date-specification to be provided during the build — I keep separate snapshot files for each snapshot I post here in order to keep them relevant to the post they were created for;
  • A gzipped tarball with installation scripts (for now, at least);
  • (Eventually) A Python egg file;
  • (Eventually) A deb package-file for Debian-based Linux distros; and
  • (Eventually) An rpm package-file for RPM-based Linux distros;
  (There is always the possibility of a new output-format becoming available as well)
• The ability to run arbitrary command-line programs:
  • Execution of module unit-tests (using the unit-testing structure I just spent time on);
  • Generation of final API documentation;
  • Deployment routines, such as rsync over ssh;
• Recursive and/or dependent-project sub-builds;

There may be other tasks or goals that I haven't thought of yet that would be added to this list...

I already know that make can handle most of these, even the recursive calls that would be required to have one project require another. The tasks that I'm not sure about are the ones that I've noted as eventual goals in the list above, but I'd be very surprised if they weren't feasible with a Makefile.

Additionally, I already get how to work with a Makefile, at least for the most part, and what I don't understand I'm confident I can find references or even sample code for.

To be fair, I'd also be surprised if most of the other available tools couldn't handle all of my requirements, but I don't know that any of them could. Since I'm more interested in writing code and discussing it than in learning some new-to-me build-process tool-chain, I'll stick to what I know, at least for the time being.

What I Want To Accomplish

For now, the two build-results that I'm concerned with are:

• Creating a snapshot ZIP-file; and
• Creating a locally-installable package (which need not be anything more than a tarball and an installation script).

If, as I'm looking at packaging structures and processes, I happen to come across a quick and easy recipe for deb, rpm or egg files, I may flesh those out as well, but I'm not going to make it a priority.

So, the things that need to be done to build a snapshot are (in order):

1. Make a copy of all of the project's files;
2. Clean out any pyc files. Leaving them in the ZIP-file wouldn't cause any harm, but is they're compiled from my local source, they'll have file-path information relevant to my local system, which is annoying (in my opinion) if those appear in an error traceback, and could be a (minor?) security risk in a real-world dev-shop scenario;
3. Remove any irrelevant directories or files from the copy. It may actually be a lot easier to copy only the relevant directories and files than to remove those that aren't relevant — Only including items that are specifically identified for inclusion is also a better security-policy, at least in my opinion.
4. Ask for input for a snapshot name (I've been using a date-string of the format YYYY-MMM-DD, so 2017-Apr-06 for today), and create a folder in some predetermined location using that name;
5. Create a {project_name}.zip archive-file in the named folder;

A tarball-and-install-script result isn't much more complex:

1. Make a copy of all of the project's files;
2. Clean out any pyc files. Leaving them in the ZIP-file wouldn't cause any harm, but is they're compiled from my local source, they'll have file-path information relevant to my local system, which is annoying (in my opinion) if those appear in an error traceback, and could be a (minor?) security risk in a real-world dev-shop scenario;
3. Remove any irrelevant directories or files from the copy. It may actually be a lot easier to copy only the relevant directories and files than to remove those that aren't relevant — Only including items that are specifically identified for inclusion is also a better security-policy, at least in my opinion.
4. Create a {project_name}.tgz archive-file in a common current-builds directory;
5. Create the installation-script and any needed support-files in the same common current-builds directory;

The major chunks of functionality in a Makefile are probably its targets. A target wraps a series of commands that need to be executed in order to achieve a desired outcome, and have a very specific structure:

# An example target - target_name:
target_name: # target variable-settings and/or dependent targets
    # Target process-steps

Each target process-step line is indented with a tab though I'm showing them here with spaces, since tabs are quite a bit longer than the four spaces that the code I've been showing have used because the code-formatting script I have in place doesn't change those to something more spacing-friendly on the site). Each line in each Makefile target will be executed in sequence. By default, unless something is done to store data from any given line, no one line has any awareness of anything done by any other line.

Each target can have any number of variables associated with it, one per line (as far as I'm aware) and one at a time. Each target can also have a list of other targets that need to be executed before that target is executed:

# An example target - target_name:
target_name: BUILD_VAR_1=value1
target_name: BUILD_VAR_2=value2
target_name: BUILD_VAR_3=value3
target_name: dependent_target_1 dependent_target_2 dependent_target_3
    # Target process-step 1 
    #    (uses $(BUILD_VAR_1), maybe...)
    # Target process-step 2 (uses $(BUILD_VAR_2) 
    #    and $(BUILD_VAR_3), maybe...)
    # Target process-step 3

If this target were executed, the various BUILD_VAR_* variables would be set first, then the dependent_target_* targets would be executed, with access to the values set in the BUILD_VAR_* variables, then the target_name target would execute.

This sort of make-based build may be very brute-force in its approach — it may well be that there are more... elegant... ways of structuring a Makefile than what I'm showing, too. Nevertheless, as I noted earlier, I know that this structure, as brute-force and inelegant as it might be, will work for what I want to accomplish.

The Starting-Point for my Makefile Template

Without any functionality or variables set up, the basic structure of what will eventually become my template Makefile is going to start as:

# Makefile template

#############################
# PROJECT SETTINGS          #
#############################
# Eventually, I'll want to set up some "global" variables for 
# keeping track of values that are common across the *entire* 
# build-process --- more on that later

#############################
# BUILD SETTINGS            #
#############################
# Settings that are specific to the project will go here

#############################
# MAIN BUILD TARGETS        #
#############################
# These are the "main" targets -- The ones that will actually 
# be used to build a project for a given environment or end.

snapshot:
    # Gathers all of the project files, ZIPs them, and copies 
    # them to a directory where I can easily share the current 
    # snapshot through my Dropbox account -- which is how I'm 
    # sharing them now...

local_all:
    # Gathers ALL of the project's files needed for a 
    # deployable copy of the project (plus files from dependent 
    # projects!) performs any necessary clean-up (removal of 
    # pyc files, files that are tagged for *other* deployment 
    # destinations, etc.), renames any files that will reside 
    # in standard executable-file locations (/usr/local/bin) 
    # when deployed, etc., etc.

#############################
# COMMON BUILD-TASK TARGETS #
#############################
# These are targets that perform tasks that are common to 
# two or more of the main build targets -- Anything that 
# *can* be done in one place *should* be done in one place 
# if only so there's only one copy of the process-code...

There's a pretty substantial amount of organizational and structural thought that I need to go through before actually creating the Makefile template, let alone putting one in play, even just for the current project, with the very few files it contains so far. I'm going to deep-dive into those details inthe balance of this post, and do the corresponding deep-dive into the guts of the Makefile next time.

Planning the Build Targets

Although the snapshot target is, I think, going to be simpler, I'm going to start with the local_all target because it sets up the pattern that will be in play for all of the other environment-specific targets, at least for the most part. I expect that it will also provide several common dependency-targets that I'll also want/need for the snapshot target. Maybe for several others, too — time will tell.

The local_all Target

The tasks that any environment-centric build needs to perform include:

• Gathering all of the project's deployable files into a single archive-file (a tgz file in this case). As I see it, that involves:
  • Copying all of the deployable directory-structure into a temporary build-location, which, in turn requires:
    • Creating the temporary build-location;
  • Performing any clean-up of files that do not need to be part of the build in that build location (pyc files, maybe others);
  • Renaming any files in standard executable-file locations (/usr/local/bin/*/*.*, for example) to remove their file-extensions (/usr/local/bin/*/*), and making sure they are executable;
  • Reconciling any environment-specific file-names so that (in this case), all files that have been copied into the build-directory whose names start with LOCAL. are renamed without LOCAL. in them, and any files whose names start with any other environments are removed. I'm currently expecting five variants (at most), including LOCAL. files:
    • LOCAL.
    • DEV.
    • TEST.
    • STAGE.
    • LIVE.
    Given that I'm not going to have access to DEV-, TEST- or STAGE-environments in the foreseeable future, and I may not have aLIVE environment for a while, I'm not going to worry too much about planning for those, but I will set things up so that each environment-specific build can follow the same pattern. I'm also planning on putting as much of the environment-specific information into variables in the Makefile as makes sense.
• Generating the final tgz archive-file;
• Copying any install-scripts into the build-directory, next to the archive-file, and modifying it/them as needed to make them viable for the target environment the deployment is for, which involves:
  • Finding the applicable scripts and copying them;
  • Replacing any environment-specific variables/items in them according to the environment the build is for;
  • Making sure they are executable;
• Copying all of the files generated to a common, local current builds directory, in order to have a copy of the most recent build for any environment available locally, just in case.

A deployable package for a local build should be installable by running the install-script from its final location, so arguably there's no need to worry about generating logic in the Makefile to copy the final package-file and install-script anywhere else. When I start looking at builds for targets that aren't local to the machine I'm building on, I'll start thinking about ways to copy the package off-site, but that's still some ways away now, given where I am with the codebase today.

Also further off, but something I can plan for now, is the idea of having builds for a given project be aware of dependencies on other projects, and generating build-packages for those projects as well. My current gut feeling is that builds for dependent projects should include all the files that would be built for each of those dependent projects in the same package. That's without any consideration for real packages (egg, deb and/or rpm files) down the line, though, so that may well change, and change drastically once I have time to do more research and thinking about those.

There are a couple of other targets that should probably be included in any build for a non-local target that I didn't list, but that I think I'll probably create now anyway: targets for unit-testing the project and for generation of API-level documentation (at a minimum). Though only one is explicitly noted in my personal coding standards, they are both important in a real-world development scenario. And, realistically, having a unit-testing target that can be called from the command-line isn't a bad thing at a local-environment level.

Finally, and also for future consideration, so not something I'll plan on implementing now: A lot of my future projects are web-applications, and I may well want to be able to deploy files and code that are only relevant to the website portion of an application, without deploying any of the application-logic. A similar argument could also be made for a logic-only build. Eventually, then, I'm expecting to have, for example, local_all, local_app and local_site builds, and, logically, similar build-targets for other environments (so, live_all, live_app and live_site builds for the eventual live projects).

Each of the bullet-points in the list above should, I think, be represented in the Makefile by its own target — that way, as other targets are needed, the existing sub-targets can be used to accomplish the same tasks. That capability, particularly for items shared across builds for different deployment environments, will save time down the line, and keep the build-process as consistent across environments as I can manage. Those sub-targets, then, are (in order by their name):

base_all:

Copies the current deployable project-directories to the temporary build-directory (into {build_directory}/{project_copy}).

base_app:

Copies the current deployable project-directories for application-logic only to the temporary build-directory (into {build_directory}/{project_copy}).

base_site:

Copies the current deployable project-directories for website items only to the temporary build-directory (into {build_directory}/{project_copy}).

build_directory:

Creates the temporary build-directory that the project-directory and deployment-scripts will be copied to.

clean_build_directory:

Removes any files that should not be part of a deployment (.pyc files, etc.)

clean_executables:

Renames any files in executable paths ({build_directory}/{project_copy}/usr/local/bin, etc.) to remove their file-extensions, makes sure that all such files are executable.

copy_to_current_builds:

Copies the package and install-script(s) ({build_directory}/*.* to the {current_builds} directory

create_documentation:

Creates project-documentation (structure TBD) at {build_directory}/{project_copy}/usr/share/doc.

create_install_scripts:

Makes copies of standard install-scripts in {build_directory}, changes all environment-references within them to match the target environment the build is for, and renames them according to the build's target environment.

create_tarball:

Creates a tgz file of the {build_directory}/{project_copy} at {build_directory}/{project_name}.tgz

current_builds:

Creates the final common current-build directory ({current_builds}) that all results will end up in.

fix_environment:

Removes any files in the build-directory that start with an environment name other than the one that the build is for (e.g., removes {build_directory}[*/[*/[...]]]LIVE.* files for a LOCAL build).

test:

Runs the unit-test code (at {project_directory}/runUnitTests.py) and dies if the tests fail.

Looking ahead at the idea of having *_app and *_site targets, I'm going to try to set things up so that implementation of base_app and base_site targets can use as much of the same structure as the base_all target listed. Those targets could be described similarly, with:

base_app:

Copies the current deployable project-directories for application-logic only to the temporary build-directory (into {build_directory}/{project_copy}).

base_site:

Copies the current deployable project-directories for website items only to the temporary build-directory (into {build_directory}/{project_copy}).

There are several items that show up in those targets that would be worth thinking about setting up as process-variables or global variables within the Makefile:

build_directory:

The temporary build-directory (e.g., /tmp/build-process);

current_builds:

The local user-directory where final build-results will be dropped (e.g., ~/Current_Builds);

project_copy:

The copy of the original project_directory (below), created inside the build_directory, where deployable files will be gathered in order to be archived into the final package-tarball (e.g., /tmp/build-process/{project_name});

project_directory:

The original project-directory (e.g., ~/IDreamInCode/{project_name});

project_name:

The name of the project (e.g., idic);

These feel, to me, like they are all viable global values within the context of the Makefile. That is, I believe that they all should be items that could be set once inthe header of the Makefile and passed along as needed to the individual targets where they are relevant. I'll contemplate that before the next post, and see how well that idea works out once I actually start writing the Makefile code.

The snapshot Target

The snapshot target is likely going to be the simplest build-target I'll create. Since all I'm intending to do with it is make a copy of the current project-directory — keeping all of the files that are part of the project proper — and creating a ZIP file archive of them in a predetermined location, there's not a whole lot involved with it, at least in comparison to the targets that are intended to generate deployable builds... Whether it will (eventually) generate snapshots of project-dependencies is something I'll have to contemplate later, but since I'm focusing on one project right now — the idic core codebase — I don't have to make that decision for a while yet...

The main difference between a snapshot build and a build that's intended to generate an installable package is in the files and directories that the final result contains. Looking back at what I expected a project-structure to look like almost two months ago, there are a few new items that need to be added, I think:

File-system Path
[Project Root Directory]
	etc
		apache2
			sites-available
	test_project_name
	usr
		local
			bin
				[project-name]
			etc
				idic
					datastores
			lib
				idic
					[project-name]
		share
			doc
				[project-name]
			icons
				[project-name]
			[project-name]
	var
		cache
			[project-name]
		www
			[project-name]
				media
				scripts
				styles
	Makefile
	runUnitTests.py
	[install-script]

where:

[Project Root Directory]/test_project_name

is a directory containing the unit-test modules (that I now know how I want to set up);

[Project Root Directory]/Makefile

is the Makefile for the project;

[Project Root Directory]/runUnitTests

is the script that the Makefile will call in its test target, specifically set up to return a standard system error-code if any test fails; and

[Project Root Directory]/[install-script]

is the base install-script that the create_install_scripts target in the Makefile will use as it's starting-point for generating installation-scripts for build for specific environments.

Procedurally, the snapshot target has the following tasks it needs to perform:

• Gathering up all the files needed in the snapshot, which might be able to leverage a target already defined:
  • Gathering all of the project's deployable files into a single archive-file (a zip file in this case). This is already available through the base_all target.
  • Adding the snapshot-specific items noted above into the project-copy directory.
  Like the local_all equivalent, that requires:
  • Creation of the temporary build-location (build_directory);
  Since the snapshot is intended to provide source-code files, it should probably also:
  • Clean out files that do not need to be part of the build in that build location (pyc files, maybe others (clean_build_directory if the types of files to be removed are the same across snapshot/non-snapshot builds);
• Generating the final zip archive-file; and
• Moving the final zip archive-file to a predetermined location.

The predetermined location that the final zip-file gets moved to can be the current-builds directory that an environment-targeted build ends with. That's not quite optimal for my purposes — ideally it'd drop the final file into a directory-structure in my Dropbox directory — but not having to write code in the Makefile that asks for a final destination isn't really necessary just yet. On top of that, I'd have to work out how to check the submitted path-value to ensure that it jives with the existing structure I have already established in Dropbox. That, I think, would add a degree of complexity that I don't need to worry about for now — and it's not like creating a directory and moving a file is all that time-consuming...

The snapshot build-process doesn't use any potential variables that weren't already noted in the environment-specific build-process earlier, so there's nothing new to consider there, for whatever that might be worth. It does add a couple of new targets, though:

add_snapshot_files:

Adds the files and directories (in {project_directory}) to the project-copy directory ({build_directory}/{project_copy})

create_snapshot_zip:

Creates a zip file of the {build_directory}/{project_copy} at {build_directory}/{project_name}.zip

I'm pretty sure that covers all of the variations I need to actually write in the Makefile template, and from a post-length standpoint, today's doesn't feel like it's too long, so I'll break for now, and spend the next post actually writing the Makefile template, and implementing it for the idic project — at least for purposes of generating snapshots.