Xclatebuffers output from multiple parallel tasks (usually started by
xapply) such that task output is kept together; because mixing output from parallel tasks makes it really hard to trace failures.
cat, and are very familiar with the shell itself, that is
In addition, you need a working understanding of the common I/O
file descriptors, and how they are manipulated by the shell
And you should be able to quote parameter lists using double-quotes or
single-quotes -- and know when to use each.
xclatewrapper collates multiple output streams such that the output from each task is complete before any other's output begins. This does not mean that the program's execution is sequenced -- just the output. The tasks may run in parallel or even in reverse order, but the output is never mixed together.
Intermingled results from multiple parallel processes create confusion. For example, the error output from "cat foo" and "make bar" mixed-up might look like:
make: cat: don't know how to make foo: bar. Stop No such file or directory
After working on large scale parallel processing for a time, I
noticed that the main restriction I had left was the serialization
of output. I had to write the output from all the processes to individual
cat those to assemble a report
I could understand. While all that was going on, I couldn't even examine
the output from the finished tasks.
I decided to code a solution that would show me the output from the finished tasks as soon as they terminated. The best way I knew how to do it was with a wrapper.
The through-put provided to 1 instance is not bound in any way to the
actions of another (given sufficient resources). For example, peer wrappers
to not share the same
listen(2) queue, or the
even the same socket options or accept filters.
This reduces 1 bottle-neck in the
client/server model, and removes some of the need for "well known ports".
To speed the implementation more, we use local-domain (unix-domain) sockets.
This removes the overhead of TCP/IP for communications, making inter-process
communications exactly as fast as a pipe.
There is a lot more information about wrappers in
wrapw documentation, which I've not
xclateis no exception. Select the master instance with the
-moption; the client mode is the default case.
The usage for a master instance of
That shows that the command "
xclate -m" is a
legal invocation of the program, and it is. When you run that command,
a new shell is created which has access to the new master process's
resources. Here is an example showing the shell history number
resets to 1 for the new shell, the process tree, and the socket
that that instance opened (which I found with the version output
wrkstn /home/ksb 85 xclate -m wrkstn /home/ksb 1 ptree $$ 117 /usr/sbin/sshd 8758 sshd: ksb@ttyp2 8761 -ksh 99919 xclate -m 99920 ksh 99984 ptree 99920 wrkstn /home/ksb 2 xclate -V xclate: $Id: xclate.m,v 2.75 2008/09/11 17:55:36 ksb... xclate: environment prefix "xcl" xclate: environment tags: "link", "list", "d", "1" xclate: protocol version 0.7 xclate: safe directory template: xclXXXXXX xclate: 1 /tmp/xcloDQ2vJ/1 [target] wrkstn /home/ksb 3 ls -las /tmp/xcloDQ2vJ/1 0 srwxr-xr-x 1 ksb wheel 0 Sep 20 15:20 /tmp/xcloDQ2vJ/1= wrkstn /home/ksb 4 exit wrkstn /home/sac1/ksb 86 ...
While an interactive shell is a possible
it is probably not the best choice.
It a lot easier to use in a script, and an interactive shell doesn't
usually require the services on an
Usually, the structure built for an
instance is a script or a tool that knows about
But we will come back to the shell example to show some of the features off later. For the time being, we need to know how to start a client program. The options for a client instance look like:
There are 3 important points in this option specification:
xidname and the
clientprocess. It is all too common the forget the
xid; I do it all the time.
xapplywhich knows how to create both the master instance and the client instances, see the xapply HTML document.
-Yimplies a major change in how
xclateprocesses the client request.
It would be a lot of typing, but one could collate output with
xclate from a command prompt.
It is much more likely to be part of a shell script, or even
designed into a large-scale parallel application.
That's why it is
an integral part of
hxmd, see the
hxmd HTML document.
How would we present the output to the reader? Which task should output first (first started or first finished)? How shall we label the output "parts" so they will be recognizable sections for the reader. We might also like to handle nested groupings (as chapters in a book, with sections in chapters, or even paragraphs in those sections).
Output which is not properly directed to a collated sections should be diverted to the end, or trapped someplace. And it would be pretty neat if we could keep a running tally of the sections still open, or finished.
Xclate provides those features, and a few more.
xclateit stands to reason that I'd use it. Oh, and I do!
xapply, which drives many parallel tasks, and might want the output from each task to be continuous. Another example would be a "make world" on a FreeBSD host. If we could put a "|xclate $component" on the end of each major component, we could make the output of the build a lot easier to read, while making it much more parallel.
xclate must be installed before
xapply is built.
On a related note,
uses options to
xclate directly, so it won't run without
The shell fragment below starts a collated stream over an
xapply with 3 tasks in parallel.
Each task stalls a bit between the output of a sequence of lines,
for a total of about 6 seconds (1+2+1+2+0). We ask for
9 iterations (first ... last).
That makes the whole
xapply command take
about 18 seconds (9*6 / 3),
but the output is not intermixed.
$ time xclate -m xapply -P3 'for i in 1 2 1 2 0 ; \ do date +"%1 %%c"; sleep $i; done |xclate %1' \ first t1 t2 t3 t4 t5 t6 t7 last ...output of the date commands... 19.09s real 0.25s user 0.46s system
The extra second overhead is the time for
fork all the processes and build all the pipes,
and also a small constant term due to code that compensates for
a race condition in a
A shorthand for this is:
which almost completely hides the xclate usage from the script writer. That code is what makes
$ time xapply -m -P3 'for i in 1 2 1 2 0 ; \ do date +"%1 %%c"; sleep $i; done' \ first t1 t2 t3 t4 t5 t6 t7 last ... more date output ... 19.08s real 0.21s user 0.48s system
xapplya wrapper: it wraps itself automatically in an
That shorthand does limit the command-line options which can be
passed directly to
For example, to pass the '-T "start %x"'
option to the xclate-filter, we must leverage an environment variable,
like this (see the
xclate manual page
$ XCLATE_1='-T "start %x"' ; export XCLATE_1 $ time xapply ...
xapplyand always passes the
-mswitch down, it also uses
xclate. The collated stream under
hxmdis really useful, as
hxmdoutput without collation is extremely jumbled.
This is such a benefit that there is no command-line option to turn off the use of xclate. Now think about how much I like command-line options and wonder "What's gotten into him?"
msrc remote build tool uses
hxmd to build the wrapper machine it needs, so
exactly the same way as
hxmd. Here is the
usage line they both need:
-Ppasses the stream on
stdin(a list of
statuspairs) to the enclosing
xclate's notification stream. If the enclosing diversion doesn't have
-rset then the
statusis removed. If
-Qwas presented it requests a diversion shutdown.
Or, when 1 or more
words are presented, it sends
the first word as a command (query) to the enclosing diversion. It then
waits for access rights to a writable file descriptor. After it gets
one from the diversion master, it pushes the rest of the
words, each terminated with a
to the channel received. The
-Q flag sends a
shutdown command to the target process (as the 4 character C-string
"-Q \000") before the name of the requested diversion.
Then in both cases
xapply's process-loop is short-circuited by a
USR1signal, it opens an
xclatestream to send synthetic exit notifications to the enclosing
xclatefor the remaining tasks.
mmsrcfrom the master recipe file
-w) plunders the master
makerecipe for macro values, it creates a synthetic notification stream for each macro name. Then uses
xclatein the recipe file to divert the macro's values to the appropriate stream.
ptbw) is used to limit the number of parallel tasks inside a collated stream, but not really used to produce collated output. But it has a subtle link to
xclate, none the less.
ptbw instance nests its UNIX domain communications
socket inside the tightest enclosing
This is an attempt to reduce the load on the filesystem be reducing
the number of meta updates for the creation and deletion of temporary
directories. Also see the ptbw HTML document.
distribwere wrapped in an
xapply -m, which is where
Hxmd builds the full
stack with detailed knowledge and intent. It does a better job of constructing
the stack than any shell programmed could by creating pipes that the
shell cannot (easily) express. Well,
with a co-process.
Xclatedoes some complex file descriptor and access rights passing behind the master source stack. Most of the time you will be unaware of these actions.
Xclate is not very useful all by itself from
the shell command-line, but it may be very useful in scripts that
produce parallel output or need access to a common resource.
hxmd, see it with
-dX(set the trace flag), or under
$Id: xclate.html,v 2.31 2014/04/13 19:18:03 ksb Exp $ by ksb.