I Dream In Code: A Repeatable Build Process [2]

So now that I know what I want my Makefile to actually do, it's time to actually write one for the current project, as well as create a template Makefile for use with future projects (though I expect it will be a while until I have need of it elsewhere).

Common Header Variables

There were four common variables across the two build-targets that I noted in my previous post:

build_directory:: The temporary build-directory (e.g., /tmp/build-process);
current_builds:: The local user-directory where final build-results will be dropped (e.g., ~/Current_Builds);
project_copy:: The copy of the original project_directory (below), created inside the build_directory, where deployable files will be gathered in order to be archived into the final package-tarball (e.g., /tmp/build-process/{project_name});
project_directory:: The original project-directory (e.g., ~/IDreamInCode/{project_name});
project_name:: The name of the project (e.g., idic);

Having given them some thought (and done a bit of research), I'm going to drop those into the top of the Makefile. I'm also going to add a project-root variable that'll eventually be used for dependent-project build-processing:

#############################
# PROJECT SETTINGS          #
#############################

# The root directory for all projects that use this build-
# process -- MUST be the same if project-dependencies are 
# going to be preserved!
PROJECT_ROOT=~/IDreamInCode

# The final destination directory for all project-build 
# results. This is local to the user!
CURRENT_BUILDS=~/Current_Builds

# The name of the project being built. It'd be nice to 
# have this auto-detected in some way, but for now, I'm 
# OK with it being a "magic" string value
PROJECT_NAME=idic

# The directory that will be created to house all of the 
# temporary files for the build-process
BUILD_DIRECTORY=/tmp/$(PROJECT_NAME)-build

# The directory that will be created to house all of the 
# temporary files for the build-process
PROJECT_COPY=$(BUILD_DIRECTORY)/$(PROJECT_NAME)

# The root directory of the PROJECT - assumes that all 
# projects reside in the $(PROJECT_ROOT) directory, 
# in a directory named after the project.
PROJECT_DIRECTORY=$(PROJECT_ROOT)/$(PROJECT_NAME)

#############################
# BUILD SETTINGS            #
#############################

#############################
# MAIN BUILD TARGETS        #
#############################

#############################
# COMMON BUILD-TASK TARGETS #
#############################

The values of these project-settings variables, defined as they are in the head of the Makefile and outside (before) any of the targets allows the to be set and available inside the targets. For example, adding this target to the current Makefile:

show_vars:
    @echo "- show_vars ----------------------------"
    @echo "BUILD_DIRECTORY ..... $(BUILD_DIRECTORY)"
    @echo "CURRENT_BUILDS ...... $(CURRENT_BUILDS)"
    @echo "PROJECT_COPY ........ $(PROJECT_COPY)"
    @echo "PROJECT_DIRECTORY ... $(PROJECT_DIRECTORY)"
    @echo "PROJECT_NAME ........ $(PROJECT_NAME)"
    @echo "PROJECT_ROOT ........ $(PROJECT_ROOT)"
    @echo "- /show_vars ---------------------------"

and running make (with the new target as the default target), yields:

- show_vars ----------------------------
BUILD_DIRECTORY ..... /tmp/idic-build
CURRENT_BUILDS ...... ~/Current_Builds
PROJECT_COPY ........ /tmp/idic-build/idic
PROJECT_DIRECTORY ... ~/IDreamInCode/idic
PROJECT_NAME ........ idic
PROJECT_ROOT ........ ~/IDreamInCode
- /show_vars ---------------------------

I'll change and re-use the show_vars target as I work through the other examples here... if output shown here has
- show_vars ----------------------------
in it, it originated with show_vars.

Those same variables can also be used as parts or all of target-level variables. By way of example, consider the starting-point for the local_all target:

local_all: ENV_PREFIX=LOCAL
local_all: BUILD_PATHS=$(BUILD_PATHS_ALL)
local_all: BUILD_OUTPUT=$(CURRENT_BUILDS)/$(ENV_PREFIX).$(PROJECT_NAME)*
local_all:
    # TODO: Implement recursive build for projects that 
    #       this project uses: 
    #       cd $(PROJECT_ROOT)/[project];$(MAKE) local_all
    # TODO: Determine if there is anything else that needs
    #       to be done for a $(ENV_PREFIX) build here.
    # Build complete for $(ENV_PREFIX) version of $(PROJECT_NAME) project:
    # + Source ... $(PROJECT_DIRECTORY)/[$(BUILD_PATHS)]
    # + Output ... $(BUILD_OUTPUT)

If make local_all is executed, it yields:

# TODO: Implement recursive build for projects that 
#       this project uses: 
#       cd ~/IDreamInCode/[project];make local_all
# TODO: Determine if there is anything else that needs
#       to be done for a LOCAL build here.
# Build complete for LOCAL version of idic project:
# + Source ... ~/IDreamInCode/idic/[etc usr var]
# + Output ... ~/Current_Builds/LOCAL.idic*

This also shows that the target-level variables can be used to build other variables-values for the target — note the results of $(BUILD_OUTPUT) in the output, whose value originated from the global $(CURRENT_BUILDS) and $(PROJECT_NAME) and the target-level $(ENV_PREFIX) values... The target-level variables also carry through to any targets called afer they are defined. That is, if the local_all target calls show_vars (and it's modified to display the current value of $(ENV_PREFIX)), it's set to LOCAL inside show_vars, just like it is in local_all:

- show_vars ----------------------------
BUILD_DATETIME ...... Mon Jan  9 14:18:48 MST 2017
BUILDER_NAME ........ ballbee
BUILD_PATHS_ALL ..... etc usr var
BUILD_PATHS_APP ..... usr var/cache
BUILD_PATHS_SITE .... etc/apache2/sites-available var/www
BUILD_PATHS_SNAP .... etc test_idic usr var
ENV_PREFIX .......... LOCAL
- /show_vars ---------------------------

Stubbing Out the COMMON BUILD-TASK TARGETS

At this point, my expectation is that the local_all target will actually get most of its work done by calling other targets, so I'm going to sub out those targets first, include them in the chain for local_all, run local_all to see what happens, tweak things if/as needed, and keep iterating over that process until all of those targets are showing up when local_all is executed, and displaying what I need to see.
The first step, then, is generating targets that at least display that they've been called. That doesn't require anything fancy, just the collection of targets and a comment in each of them to the effect of what target they are:

#############################
# COMMON BUILD-TASK TARGETS #
#############################

add_snapshot_files:
    # Calling add_snapshot_files

base_all:
    # Calling base_all

base_app:
    # Calling base_app

base_site:
    # Calling base_site

build_directory:
    # Calling build_directory

clean_build_directory:
    # Calling clean_build_directory

clean_executables:
    # Calling clean_executables

copy_to_current_builds:
    # Calling copy_to_current_builds

create_documentation:
    # Calling create_documentation

create_install_scripts:
    # Calling create_install_scripts

create_snapshot_zip:
    # Calling create_snapshot_zip

create_tarball:
    # Calling create_tarball

current_builds:
    # Calling current_builds

fix_environment:
    # Calling fix_environment

test:
    # Calling test

With that in place, calling all of the relevant targets for each of the two main targets (snapshot and local_all) is also pretty straightforward, if a bit long:

#############################
# MAIN BUILD TARGETS        #
#############################

snapshot: ZIP_FILE_NAME=$(PROJECT_NAME).zip
snapshot: BUILD_OUTPUT=$(CURRENT_BUILDS)/$(ZIP_FILE_NAME)
snapshot: BUILD_PATHS=$(BUILD_PATHS_SNAP)
snapshot: BUILD_ZIP=$(BUILD_DIRECTORY)/$(ZIP_FILE_NAME)
snapshot: current_builds build_directory base_all add_snapshot_files clean_build_directory create_snapshot_zip
    # Build complete for SNAPSHOT of $(PROJECT_NAME) project:
    # + Source ... $(PROJECT_DIRECTORY)/[$(BUILD_PATHS)]
    # + Output ... $(BUILD_OUTPUT)

local_all: ENV_PREFIX=LOCAL
local_all: BUILD_PATHS=$(BUILD_PATHS_ALL)
local_all: BUILD_OUTPUT=$(CURRENT_BUILDS)/$(ENV_PREFIX).$(PROJECT_NAME)*
local_all: current_builds build_directory base_all clean_build_directory clean_executables fix_environment create_tarball create_install_scripts copy_to_current_builds
    # TODO: Implement recursive build for projects that 
    #       this project uses: cd $(PROJECT_ROOT)/[project];$(MAKE) local_all
    # TODO: Determine if there is anything else that needs
    #       to be done for a $(ENV_PREFIX) build here.
    # Build complete for $(ENV_PREFIX) version of $(PROJECT_NAME) project:
    # + Source ... $(PROJECT_DIRECTORY)/[$(BUILD_PATHS)]
    # + Output ... $(BUILD_OUTPUT)

Calling make snapshot and make local_all then return, respectively:

# Calling current_builds
# Calling build_directory
# Calling base_all
# Calling add_snapshot_files
# Calling clean_build_directory
# Calling create_snapshot_zip
# Build complete for SNAPSHOT of idic project:
# + Source ... ~/IDreamInCode/idic/[etc test_idic usr var]
# + Output ... ~/IDIC_Builds/idic.zip

# Calling current_builds
# Calling build_directory
# Calling base_all
# Calling clean_build_directory
# Calling clean_executables
# Calling fix_environment
# Calling create_tarball
# Calling create_install_scripts
# Calling copy_to_current_builds
# TODO: Implement recursive build for projects that 
#       this project uses: cd ~/IDreamInCode/[project];make local_all
# TODO: Determine if there is anything else that needs
#       to be done for a LOCAL build here.
# Build complete for LOCAL version of idic project:
# + Source ... ~/IDreamInCode/idic/[etc usr var]
# + Output ... ~/IDIC_Builds/LOCAL.idic*

which at least demonstrate that the required sub-targets are firing as expected.

Putting some Meat in the Sub-Targets

So now it's time to actually implement the sub-targets. Since I'm not currently concerned about the items in local_all, I won't address any of the sub-targets here that are only part of that in any real detail — I'll need to finish those out at some point, but until I've figured out what an installation-script needs to do and look like, no environment-oriented build is really relevant, not even the local one. That leaves, in order of execution in the Makefile for a make snapshot build that I'll discuss in this post:

current_builds;
build_directory;
base_all;
add_snapshot_files;
clean_build_directory;
create_snapshot_zip; and
any changes or additions to the main snapshot target that arise as a result of any of the earlier targets in execution order.
target_name

To be clear: I'm planning on implementing the other targets, if only so I have a (hopefully) complete local-build-ready Makefile, but I'm not going to show or discuss them in any great detail. However far I do get with them, they'll be in the template Makefile, and that'll be downloadable at the end of this post.

The `current_builds` Target

Ultimately, the current_builds target is really only responsible for creating the final destination-directory that all of the top-level build-processes will drop their final output in. It should hopefully be no great surprise, then, that it's not a terribly complicated target. It does need the CURRENT_BUILDS variable set, but that happens at the top of the Makefile, so that's already accounted for. The complete target is:

current_builds:
    # Creating local current-builds directory
    #   at $(CURRENT_BUILDS)
    @mkdir -p $(CURRENT_BUILDS)

One item that's worth mentioning here because it'll show up elsewhere later: One of the things that I find I don't like about make is that all of the commands that get executed in a target (like the

mkdir 
-p $(CURRENT_BUILDS)

here) get echoed out to the console during a make run by default. I'm guessing that in a more normal build-process, say compiling and assembling a C or C++ program, there may be some advantage to seeing the commands that were run as well as any output from them. I don't expect that most of what I'm going to be doing in any of these targets really needs that degree of visibility, though, so I almost always hide the commands being executed (and their output) by default. That's what the @ in the @mkdir -p $(CURRENT_BUILDS) line does. I usually figure that if I need (or want) output, it's easy enough to generate it where or if I determine I do want it.

The `build_directory` Target

build_directory is just as simple a target as current_builds, because it does much the same thing, just with a different variable (BUILD_DIRECTORY, also defined in the head of the Makefile) controlling the path of the build-directory to be created:

build_directory:
    # Creating build-directory
    #   at $(BUILD_DIRECTORY)
    @mkdir -p $(BUILD_DIRECTORY)

The `base_all` Target

This is where things start to get interesting, I think...
One of the main tasks that this build-process has to do is to assemble the project's code into a safe location for further processing. In order to do that, it has to know several things about the project, all of which are based around the project-structure:

Where the project itself lives (the PROJECT_DIRECTORY variable);
Where the safe code-copy is to be made (the PROJECT_COPY variable); and
What directories in the project need to be copied (the BUILD_PATHS variable)

PROJECT_DIRECTORY and PROJECT_COPY are set in the head of the Makefile, but BUILD_PATHS is created by the various top-level targets:

#############################
# MAIN BUILD TARGETS        #
#############################
# ...
snapshot: BUILD_PATHS=$(BUILD_PATHS_SNAP)

# ...

local_all: BUILD_PATHS=$(BUILD_PATHS_ALL)

# ...

They, in turn, refer to other variables (BUILD_PATHS_SNAP and BUILD_PATHS_ALL) that are also defined in the Makefile's head.
base_all also marks the first time that I'm writing a build-target that uses programs available in the external shell. I ran into some oddities while I was working this target out, probably because of either something new in my version of make (unlikely), or some misunderstanding on my part about what a target has access to in a Makefile. Discussion of that will make more sense after I show the target, though, so here it is:

base_all:
    # Creating project-copy directory: 
    #   at $(PROJECT_COPY)
    @mkdir -p $(PROJECT_COPY)
    # Copying $(BUILD_PATHS) 
    #      to $(PROJECT_COPY):
    @for PATH in $(BUILD_PATHS); do \
        /bin/cp -r $(PROJECT_DIRECTORY)/$$PATH \
            $(PROJECT_COPY)/$$PATH; \
        echo "# + $(PROJECT_NAME)/$$PATH copied to $(PROJECT_COPY)/$$PATH"; \
    done

Like the previous two targets, base_all creates a directory — this one being the place to copy all of the project-code, defined by the PROJECT_COPY variable, which is defined in the head of the Makefile. That's nothing too new, since it's been done before.
The next step is where things got a bit odd. The target then uses a shell-loop to iterate over the BUILD_PATHS (also defined in the head), and makes a copy of the directory, with all of its children, in the PROJECT_COPY directory, before displaying messaging indicating that the copy succeeded. The odd thing, I thought, was that the cp shell-utility couldn't be found within the target while it was running. That is, with

base_all:
    # ...
    
    @for PATH in $(BUILD_PATHS); do \
        cp -r $(PROJECT_DIRECTORY)/$$PATH \
            $(PROJECT_COPY)/$$PATH; \
        echo "# + $(PROJECT_NAME)/$$PATH copied to $(PROJECT_COPY)/$$PATH"; \
    done

in the target-code, execution died in the middle of that target, reporting

/bin/sh: 2: cp: not found

I have no idea why. I've used similar command-structures (though usually calling rsync instead of cp) before and never run into this issue. Nor, frankly, does there appear to be any mention of this sort of bugaboo anywhere that I could find with a Google search. To be fair, I may not have hit on the right search-terms in my efforts, but even so — no mention of this at all leads me to think that there's something odd going on.
Still, fortunately, I could access the cp command by providing the full shell-path to it (found with which cp in a terminal). I'd like to figure out why a normal cp call is barfing, especially since the echo and mkdir commands weren't having issues, but the workaround that's in place now will do for me, even if it does make some assumptions about the system that builds are being generated on.
With this target operational, the build-output is starting to look close to complete:

# Creating local current-builds directory at ~/IDIC_Builds
# Creating build-directory at /tmp/idic-build
# Creating project-copy directory: 
#   at /tmp/idic-build/idic
# Copying etc test_idic usr var 
#      to /tmp/idic-build/idic:
# + idic/etc copied to /tmp/idic-build/idic/etc
# + idic/test_idic copied to /tmp/idic-build/idic/test_idic
# + idic/usr copied to /tmp/idic-build/idic/usr
# + idic/var copied to /tmp/idic-build/idic/var
# Adding snapshot files and directories to /tmp/idic-build/idic

and the build-copy directory is starting to show the source-files:

So far, so good...

The `add_snapshot_files` Target

As things stand right now, only one file of three (or more) that I expect to be part of any given project actually exist in my current project-structure template: the runTests.py file that will be called by the test target. Since the purpose of a snapshot build is to copy the entire active codebase for dissemination here on the blog, it needs to include all the files that might be considered formal members of the project, which will, eventually, include the Makefile that I'm working on now, any installation-scripts for the project, and maybe a text-file report of the output of the last unit-test run. Not all of these files will exist at any given time, though at least two can be specified right now:

#############################
# PROJECT SETTINGS          #
#############################

# ..

# The list of all additional files that need to be added 
# to a snapshot build's file-set
SNAPSHOT_PATHS=runTests.py $(PROJECT_NAME)-test-results.txt
# TODO: Once the other files that belong to all projects 
#       have been defined, add them to the list here, 
#       including:
#       - Makefile
#       - install-scripts
#       - Others?

Having the SNAPSHOT_PATHS defined allows the add_snapshot_files target to be fully defined:

add_snapshot_files:
    # Adding snapshot files and directories to $(PROJECT_COPY)
    @for PATH in $(SNAPSHOT_PATHS); do \
        if [ -f "$(PROJECT_DIRECTORY)/$$PATH" ]; then \
            /bin/cp -r $(PROJECT_DIRECTORY)/$$PATH $(PROJECT_COPY); \
            echo "# + $(PROJECT_NAME)/$$PATH copied to $(PROJECT_COPY)"; \
        else \
            echo "# + $(PROJECT_NAME)/$$PATH skipped"; \
        fi; \
    done

The output of the add_snapshot_files target specifically lists each file in the SNAPSHOT_PATHS list, so that it can report on the finsl disposition of each one individually:

# Adding snapshot files and directories to /tmp/idic-build/idic
# + idic/runTests.py copied to /tmp/idic-build/idic
# + idic/idic-test-results.txt skipped

The `clean_build_directory` Target

Deploying or distributing compiled .pyc files (and presumably .pyo files as well) can be troublesome. If the original .py file that a .pyc file was compiled from hasn't been modified since the last time the .pyc was compiled, it may well contain references to paths on the original system that it was compiled on. While I've never seen that cause any actual errors, I have seen cases where an unrelated error in the code raises an exception, and the traceback that accompanies it displays those original-system paths. That may not be a big deal. It wasn't anything more than an annoyance to me the few times I encountered it, but then I knew the codebase that the error resided in pretty intimately, and had no great difficulty translating from the live error module-paths to the stored paths that originated on my local development machine. Someone less familiar with a codebase than I was might well get more than a little frustrated trying to track down the source of a bug when they could be spending the time fixing it. Since I'm writing Python code, and there's no need for me to distribute .pyc or .pyo files, there's no point in keeping them in the file-set for any build at present. That is what the clean_build_directory target is all about:

clean_build_directory:
    # Removing files that shouldn't be part of a build
    #  - Removing all files matching: $(REMOVE_FILE_TYPES)
    @for FILE_SPEC in $(REMOVE_FILE_TYPES); do \
        for FILE_NAME in `find $(PROJECT_COPY) -name "$$FILE_SPEC"`; do \
            rm $$FILE_NAME; \
            echo "#    + Removed $$FILE_NAME"; \
        done; \
    done
    # - Removing individual files: $(REMOVE_FILES)
    @for FILE_SPEC in $(REMOVE_FILES); do \
        for FILE_NAME in `find $(PROJECT_COPY) -name "$$FILE_SPEC"`; do \
            if [ -f "$$FILE_NAME" ]; then \
                rm $$FILE_NAME; \
                echo "#    + Removed $$FILE_NAME"; \
            fi; \
        done; \
    done

Structurally, at least, clean_build_directory isn't much different from add_snapshot_files. The same sort of iteration over a list of files is present (twice, admittedly). The main differences are in how those files are identified, and what happens to the individual files during the iteration. Instead of copying a file into the project-copy directory from the original project-directory, it's removing files from the project-copy. Identification of the files to be removed is through the system's find utility, allowing patterns to be specified in REMOVE_FILE_TYPES and specific files in the REMOVE_FILES list.

The `create_snapshot_zip` Target

At this point, all that's remaining to do is to generate the final snapshot ZIP file, and maybe do a little clean-up (removing the build-directory, since there's nothing left to do with it):

#create_snapshot_zip:
    # Creating snapshot ZIP file:
    @cd $(BUILD_DIRECTORY);zip -qr $(BUILD_OUTPUT) $(PROJECT_NAME)
    # Removing the build-directory
    @rm -fR $(BUILD_DIRECTORY)

Final Check of the `snapshot` Target

Since the create_snapshot_zip target creates the final ZIP-file in it's final location, the BUILD_ZIP value that was originally in the snapshot target was no longer needed. It was originally going to be used to generate an intermediary ZIP-file in the build-directory, but the intermediate step wasn't necessary, so I've removed it:

snapshot: ZIP_FILE_NAME=$(PROJECT_NAME).zip
snapshot: BUILD_OUTPUT=$(CURRENT_BUILDS)/$(ZIP_FILE_NAME)
snapshot: BUILD_PATHS=$(BUILD_PATHS_SNAP)
snapshot: current_builds build_directory base_all add_snapshot_files clean_build_directory create_snapshot_zip
    # Build complete for SNAPSHOT of $(PROJECT_NAME) project:
    # + Source ... $(PROJECT_DIRECTORY)/[$(BUILD_PATHS)]
    # + Output ... $(BUILD_OUTPUT)

In the interests of showing where things have landed with being able to make snapshots from a build-process, I'm saving a copy of the current Makefile template, before I start tackling any of the local_* build-targets, that can be downloaded and examined as you see fit. I've also got a Makefile in the idic project now, one that's slightly further along than what I've shown in this post. It's also available for download below, as an item in the snapshot that it built.
I don't think I'll pursue the local_* builds much further than I have for now — until I get to a point where I need to be able to generate a local installation of some sort, I'd have no way to meaningfully test the accuracy of the process. With where I am in my efforts on the idic framework so far, that's likely to be a while, a point that I'll think over and discuss in my next post.

Makefile template

7.9kB

idic.zip

43.4kB

I Dream In Code

Tuesday, April 11, 2017

A Repeatable Build Process [2]

Common Header Variables

Stubbing Out the COMMON BUILD-TASK TARGETS

Putting some Meat in the Sub-Targets

The `current_builds` Target

The `build_directory` Target

The `base_all` Target

The `add_snapshot_files` Target

The `clean_build_directory` Target

The `create_snapshot_zip` Target

Final Check of the `snapshot` Target

No comments:

Post a Comment

Tuesday, April 11, 2017

A Repeatable Build Process [2]

Common Header Variables

Stubbing Out the COMMON BUILD-TASK TARGETS

Putting some Meat in the Sub-Targets

The current_builds Target

The build_directory Target

The base_all Target

The add_snapshot_files Target

The clean_build_directory Target

The create_snapshot_zip Target

Final Check of the snapshot Target

No comments:

Post a Comment

The `current_builds` Target

The `build_directory` Target

The `base_all` Target

The `add_snapshot_files` Target

The `clean_build_directory` Target

The `create_snapshot_zip` Target

Final Check of the `snapshot` Target