Sometimes less is more

Explode uses markup in the comments in a source file to burst the file into separate units. These units are usually compiled to form a dynamic library, test the function of the library itself, or to build special features of the would-be library for integration with a specific application via compile-time tuning. This application is also integrated with mkcmd.

To understand this document

This document assumes you are quite familiar with the standard UNIX™ shell, C or some other compiled language, and have an understanding of the UNIX filter model (pipes, input/output redirection, and some specific filters.

It also assumes that you can read make recipe files.

What is explode?

The manual page describes explode as "unpacking components from aggregated source archives". That means that explode expects to an input file that is the source code for a number of related functions (an aggregate file), which it is going to decompose into individual files -- usually 1 for each function represented.

Explode has command line options to limit this unpack operation to a subset of the possible functions (or files), or provide a table of contents, or to produce the whole aggregate file as-is. These options support are largely used in make recipes or mk (see the HTML document) markup to produce source code "just in time" to compile it. All the generated source files are removed after a successful installation so that rebuilds will pick up any updated source files.

This model differs from the common "library" abstraction in 3 ways:

The use of compile-time tunes to optimize integration
Most explode aggregates expect some macro or call-back support from the incorporating application. Usually a typedef or a set of data specific functions. This allows the compiler to optimize and check types at compile-time. (Using (void *) and (int (*)(void *)) types almost completely defeats an check the compiler could make, so don't do that.
An agile assumption of a moving target
Explode is often used when the aggregate file is changing rapidly. It is easy to build a dependency relationship on the common aggregate to the unpacked functions such that make keeps the local application up-to-date with the changing module. This allows very rapid test cycles, where a common library file might impact applications that were not really ready to integrate.
The expectation of code inspection (and introspection)
Most of the documentation for the aggregated functions is contained in the source file itself. This assures that the documents are up-to-date with the functions.

Conversion to a library file

If it usually easy to convert a stable aggregate file into a library file (static or dynamic) if there is a good set of default types and tunes for the facility represented. We simply unpack every function, build it with the default configuration, then create an ar archive of the resulting object files. Some aggregates even default to the correct tunes for this operation.

That doesn't work for some facilities. So where it doesn't we just continue to use explode.

Examples

For example aggregate files, just look in the explode library itself, use explode's -V option to display the default library path:
$ explode -V
explode: $Id: explode.m,v 6....
explode: default repository /usr/local/lib/explode:/usr/share/lib/explode:~/lib/explode

See the list in the default explode library (usually /usr/local/lib/explode). Here is a link to the master source to that directory. Note that the installation of some services includes the installation of the aggregate file that contains an interface to the service, so not all of the files installed are in the master source directory.

Example usage in a make recipe

Pulling several modules from aggregate source files from a make recipe file looks like this:
# fetch things we need from explode
dicer.h setenv.h fdrights.h avl.h:
	explode -s $@		# whole files

dicercall.c:			# specific parts
	MACHINE_H="#include \"machine.h\"" explode -u dicer,mixer -o - dicer.c >$@

ptbcall.m:
	MACHINE_H="#include \"machine.h\"" explode -u quit -o - ptbc.m >$@

setenv.c:
	MACHINE_H="/* nada */" explode -o - setenv.c >$@

avlcall.c:
	MACHINE_H="/* nada */" explode -u ins,init,fix,scan -o - avl.c >$@

fdcall.c:
	MACHINE_H="#include \"machine.h\"" explode -u recv,wrapper,send -o - fdrights.c >$@
Note that all the modules use a common machine.h (local to the build directory) which includes any compile-time tunes that are required. You'll have to read each module (HTML document for each module) to build the required machine.h.

The shell variable MACHINE_H allows different tunes for some modules, if there are common cpp macros that must have conflicting values for multiple modules. This would usually be a bug in the modules, as they should have unique name-spaces for their macros. Note that I explicitly set MACHINE_H to a comment to prevent a default value, which may be provided by each module. Your local site policy for common modules may be more (less) explicit.

Using a target formula

In the make recipe it is a little more pretty to form the file with a expr expression to create any missing parts:
ptbccmd.m ptbcclient.m:
	explode -u `expr $@ : 'ptbc\(.*\)\.m$$'` ptbc.m

This may be done on the master side, or the platform side. If it is coded on the platform side we need to avoid m4 quotes. One way would be to changequote to some alternate quote for every rule, but that looks terrible:

'changequote([,])dnl
[ptbccmd.m ptbcclient.m:
	explode -u `expr $@ : 'ptbc\(.*\)\.m$$'` ptbc.m
]changequote(`,')dnl
`
And we would have to changequote for each explode module we build against. That would repeat this spell many times:

The best way I've found is to bind 2 make macros to the m4 quote characters (near the top of the file):

 # your introduction comments
'changequote([,])dnl
[O=`
Q='
]changequote(`,')
`...
I use ${O} for m4 open-quote and ${Q} or ${C} for the close-quote. These helper macros make it easy to embed shell quotes in any make recipe you write, not just the explode extraction recipe.

Then the recipe looks like:

ptbccmd.m ptbcclient.m:
	explode -u ${O}expr $@ : ${Q}ptbc\(.*\)\.m$$${Q}${O} ptbc.m

Since the files are ephemeral and quick (easy) to recreate, I just force the clean target to remove them. That way we don't need a dependency on the path to the source file. On a platform host we may use explmux(7) to fetch the up-to-date copy from a local site policy repository.

Remote build support

For light-weight operation we can consult a local site policy repository to fetch the modules from a tcpmux service, usually called explmux.

Adding a -R specification redirects explode to fetch the module from the specified remote host.

$ explode -s -R repo.npcguild.org avl.c

It is also possible to specify a different service name. This is usually used for test build or for the very picky:

$ explode -s -R repo.npcguild.org:newbie avl.c

So in the make recipe example above we can install a macro hook to allow remote builds without a change to the recipe file:

EXPLMUX=
ptbccmd.m ptbcclient.m:
	explode ${EXPLMUX} -u `expr $@ : 'ptbc\(.*\)\.m$$'` ptbc.m
Then on the command line we can force an assignment to EXPLMUX which specifies "-Rrepo.npcguild.org" to force the explode modules source to be remote. This could also be done by setting the environment variable EXPLODE, but some build processes may not pass that through. (Under msrc use -y to yolk EXPLMUX's value into the make command-line.)

The only reason not to deploy the explode library to all your build hosts is that you can't manage it. If that's is the case, then please look into msrcmux to pull the up-to-date sources and install them.

Summary

Explode is the keeper of our data abstractions and client/server protocols. It allows us to reuse these parts with compile-time tunes and optimizations. Over 40 platform recipes in my master source repository use explode to fetch common source files.

Also see the network service explmux, which allows anonymous network clients to fetch explode modules as needed.



$Id: explode.html,v 1.13 2014/04/13 18:38:32 ksb Exp $ by .