XCLATE(1)                                                            XCLATE(1)



NAME
       xclate - collate output from parallel tasks

SYNOPSIS
       xclate  -m  [-AdnQrsv]  [-depth]  [-e  var=value]  [-H  hr]  [-i input]
       [-L cap] [-N notify] [-O output] [-T title] [-u unix] [-W widow] [util-
       ity]
       xclate  [-nQsvw]  [-IEDY]  [-depth]  [-e  var=value]  [-H  hr] [-L cap]
       [-T title] [-u unix] [xid] [client]
       xclate -h
       xclate -V
       xclate -P [-nqQv] [-depth] [-u unix] [words]

DESCRIPTION
       The UNIX pipe model for input/output redirection is simple and  elegant
       for  most  single-thread  tasks,  and scales well for some multi-thread
       applications.  New issues arise  when  the  speed  of  computation  far
       exceeds the speed of an output device.  Processes block quickly, output
       from one process gets mixed into the output from another,  and  overall
       through-put is limited to the slowest tasks using the slowest device.

       Xclate  is an attempt to recover from these issues.  Processes may out-
       put freely to a buffered stream of at least cap bytes before they might
       block  (on  output).   The output from each tasks is flushed completely
       before a new task is allowed to output.  While overall  through-put  is
       greater,  as  tasks  are cleared in the order they finish (more likely)
       than in the order they started (thus a slow task may run with  deferred
       output while several generations of quicker tasks complete).

       To accomplish this xclate has three modes.

       Manager mode
              Under  -m  it  forks  a utility with a special environment, then
              arbitrates use of the original stdout channel for any descendant
              processes  created by utility which request access.  The default
              utility in this mode is a shell, either  the  one  specified  by
              $SHELL,  or  /bin/sh.   A semi-permanent instance may be created
              when the utility is specified as : (a single colon), see -Q  and
              -u.

       Filter mode -- merge output
              Without  -m  or  any client the program reads from stdin at most
              cap bytes then waits for exclusive access to the manager's orig-
              inal  stdout.  It flushes all the buffered input, then pumps the
              remainder of its stdin to the acquired stdout.

       Command mode -- filter and merge output
              Given a client, and without -m, the program buffers stdin, as it
              would  without  a  command,  waiting for exclusive access to the
              manager's stdout.  When it gains access to the exclusive  output
              it starts the given client with the buffered stream as stdin and
              the manager's stdout, then pumps the remainder of its  stdin  to
              the new process.  When any options in the set -I, -E, -D are set
              this behavior is modified quit a bit.

       In either of the last two modes the positional  parameter  xid  is  the
       identification for this stream, usually xapply(1l)'s %1 or %u escape or
       by default xclate's process id.  Other useful values are the name of  a
       host,  directory,  or device which is the topic of the output.  Specify
       -T and -H to include a title and/or horizontal break around named  sec-
       tions.

       The  -P option allows xapply to send synthetic acknowledgments for com-
       mands that were suppressed by the receipt of a USR1  signal.   In  that
       case  words  are taken as a list of uid,status pairs to be delivered to
       the enclosing xclate's notification stream.  The  specification  of  no
       words  causes  xclate  to  read  an  exit code list from stdin.  The -r
       option provided to the active diversion is honored; that is to say  the
       exit code is removed when -r was not specified.

       This option is also used by msrc(8l), and mmsrc(8l) (only under -w), to
       send macro information from a plundered make recipe back to itself  (by
       the specification of words which have no leading digit).

OPTIONS
       If  the program is called xclate then no options are forced.  The envi-
       ronment variable $XCLATE is read for command line options,  before  the
       explicit command line.  That variable may also be changed by xclate for
       descendent processes, see ENVIRONMENT below.

       -depth
              Specify a collation sequence beyond the tightest  enclosing.   A
              depth of 0 is the default.  A depth of 1 accesses the next outer
              instance of xclate -m.

       -A
              Add the name of the controlling socket as the first line of  the
              notification  stream.  This option is used by sshw(1l) to locate
              the service, as it is used in global mode to manage  the  remote
              process redirection.

       -d
              Do   not   link  the  new  manager's  diversion  socket  in  the
              $xclate_link environment chain.  Rather the manager socket  name
              is  recorded in $xclate_d, while $xclate_link is left unchanged.
              When the manager socket is specified (via -u unix) the  environ-
              ment  variable may be totally ignored, as you already know where
              the socket is.

       -e var=value
              Set the environment variable var to the given  value  after  the
              path to the client (or utility) is found.  When no value is pro-
              vided, the implied value is the empty string.

       -e !var
              As above, but remove var from the utility process's environment.
              The character tilde (~) is also accepted.  It is not an error to
              remove a nonexistent variable name..

       -h
              Print only a brief help message.

       -i input
              The stdin given to the xclate process is replaced by this stream
              for  the  utility process.  The original stdin is available only
              via the -I option to client xclate instances.  The input  speci-
              fication  may  be  any of "|command", "<file", "<>file", "<&fd",
              "<&-", "file", "socket" or "-" as a shorthand for  "</dev/null".

       -H hr
              Add  a  horizontal  rule to end each output stream.  If a client
              instance does not specify a hr  the  master's  is  assumed.   An
              explicit  specification  of  the empty string may over-ride this
              default.  See -T for a list of escapes expanded.

       -L cap
              Put a cap on memory based output buffer.  This limits the amount
              of  stdin xclate collects before it blocks waiting for exclusive
              access to the manager's stdout.  The default  is  64k,  and  the
              value  may  be  postfixed  with a scale factor, derived from the
              mkcmd(1l) type "bytes".  The master's specification  of  cap  is
              assumed  if  none  is presented on the command-line.  Use -L '?'
              for a list of scale factors.

       -m
              Manage output for descendant processes.  This starts a new  man-
              aged  output stream (the stdout of the xclate process itself, or
              the channel created by -O).  In the environment of  the  utility
              created  some variables are installed to allow subsequent xclate
              processes to contact the manager process to gain access  to  the
              shared output stream (see below).

       -N notify
              Send  a  null  terminated string (the xid) to this file, process
              or file descriptor after each  section  is  complete.   This  is
              largely  for  hxmd(8l)'s  use.   If  specified as dash ("-") the
              default stream is /dev/tty, in which case the null  is  replaced
              with  a  newline.   The  null  is needed when the output is read
              through a pipe, and the xid might contain a newline.   The  same
              redirections available under -W are also available.

       -n
              Do  not execute commands, trace only.  This doesn't really work,
              but is required by the pipefit code.  Don't try it.

       -O output
              Specify a diversion for the collected stream, rather  than  std-
              out.   This assures that no output ends up in the stream without
              an attending xclate process.  The same diversions are allowed as
              for  -W  (below),  with the exception that a dash ("-") means to
              connect to an enclosing instance of xclate, this groups all  our
              output  inside  the  specified  diversion.   In this case -depth
              specifies the depth of the desired diversion.

       -Q
              Any manager instance started with the utility of ":"  will  con-
              tinue  to process until at least one client run with this option
              finishes.  This allows a diversion to be started by an unrelated
              process tree to complete after some client completes.  The shut-
              down is graceful and allows presently connected clients to  fin-
              ish,  as  well as some clients that may connect later.  Ptbw has
              exactly the same notion.

       -r
              The notify stream will include the exit status from each process
              before  the  xid,  separated with a comma.  This may not work in
              filter mode, as we can't see the exit code of the process in all
              cases (it works for most shells).

       -s
              Totally  silence  subtasks with no output.  This changes the way
              xclate executes some requests to place itself on the stdout side
              of  the  client  process,  but  adds substantial speed when most
              tasks produce no output.  Even the  title  (-T)  and  horizontal
              rule  (-H)  are  removed for tasks with zero length output.  See
              also xapply, which passes this option on to us.  Providing  this
              option  in  master mode sets the option for the whole diversion,
              as promised in the last release.  All hxmd's  send  xapply  this
              option to speed processing.

       -T title
              A title for each stream, some limited escapes are replaced:
                   %x   xid
                   %s   sequence number (like xapply's %u, in output order)
                   %{ENV}    the value of $ENV
                   %%   a literal percent (%)
              When the title should not have a trailing newline, make the last
              character a percent (%).

       -u unix
              Specify a forced unix domain end-point.  This  allows  unrelated
              instances  of  xclate  (on  the  host) to share a managed output
              stream.  To make this work you might have to change the mode  on
              the  manager directory.  This option is required (in general) to
              make the persistent mode (under -m with utility specified  as  a
              single :) useful.  See below.

       -v
              Be  verbose.   Show  some  of  the commands xclate is running in
              pseudo-shell.

       -V
              Show only the standerd wrapper version  banner.   This  includes
              the  API  version of the xclate protocol, which nobody whould be
              usings; but sshw won't work across  hosts  that  have  different
              versions.

       -w
              The  stdout  from the client is directed to the widow output for
              the specified depth.  This is most useful  with  depth  set  and
              multiple diversions in scope.

       -W widow
              Redirect  widowed  output  to  this file, process, or fd.  Under
              this option output which is not sent through a descendant xclate
              filter  process is gathered into a "widow" section at the end of
              the output.  The widow parameter may be  a  process  (as  "|com-
              mand"),  a  file  (as  ">file", or ">>file", or "file"), a local
              (UNIX) domain socket (as  "socket"),  any  already  opened  file
              descriptor (as ">&fd"), or a dash (as "-") for stderr.

COMMAND MODE OPTIONS
       These  options  will  let you blow your foot completely off, placing it
       into orbit.  They are not used by most automation, but allow  for  some
       very clever shell scripts (see sapply):

       -I
              The  client  is  started  with  stdin  connected to the master's
              stdin.  No buffering of stdin can be provided.

       -E
              The client is started with  stderr  connected  to  the  master's
              stderr.

       -D
              The  client  is launched from the same current working directory
              as the master's.  Really, even if that directory is hidden under
              a mount point.  See fchdir(2).

       -Y
              Change  the  controlling tty to the new stdin, stdout, or stderr
              (in that order).  See tty(4).

       These are the inspiration for the sshw wrapper.  I'll even  claim  that
       xclate is the first true wrapper, and inspired all the others.

ENVIRONMENT
       Since  xclate  is  intended to be recursive, a provision is made in the
       processing of environment  variables  for  passing  options  to  nested
       xclate processes.

       $xcl_link
              The  environment  variable  $xcl_link  is  set  to the number of
              nested xclate processes  presently  running.   It  may  also  be
              forced to the letter "d" to use the isolated diversion.

       $xcl_1, $xcl_2, ...
              Each  manager  instance of xclate sets a variable (named for the
              value of $xcl_link established when it started) to the  path  to
              the unix domain socket used to chat with that instance.

       $xcl_d
              The  path  to  the unix domain socket for the tightest enclosing
              instance of xclate -m with -d in effect.  This is sometimes ref-
              ered to a an isolated diversion.

       $XCLATE and $XCLATE_1, $XCLATE_2, ...
              After  reading  command  line options from variable $XCLATE, any
              master (-m) process removes it from the environment (before exe-
              cuting  utility  in  the  child  process).  It then installs the
              variable $XCLATE_nesting in its place, if it is present  in  the
              environment.   This allows header and horizontal rule options to
              be set before they are needed, for  example  before  any  xapply
              commands are started.

              $XCLATE     read by the first manager instance
              $XCLATE_1   read by the children of the first manager
              $XCLATE_1   read by the second manager, if not reset.
              $XCLATE_2   read by the children of the second manager
              ...         and so on...

EXAMPLES
       xclate -vV
              Show version information and any details in the present environ-
              ment.

       xclate -L \? -V
              Display the table of suffixes supported by -L, and some  version
              information.

       xclate -vm  tasks
              Assuming  that  the file tasks is a shell script that had a list
              of (long running) background jobs in it, for example:
                        xclate -T%x passwd sort /etc/passwd &
                        xclate -T%x  group sort /etc/group &
                        ...
                        wait
              The output of this command would be the output of  the  jobs  in
              some order, with each output section contiguous.

       xapply -m -P5 ...
              Wrap  a  controlled environment around an xapply process.  There
              is some real hoodoo going on here, and you don't  want  to  mess
              with it until you can use xclate very well.

       xclate -m xapply  -P5 '...  |xclate %u' ...
              This puts the pixie dust on the xapply by hand, and is in effect
              exactly what xapply does under -m.  This is akin to making home-
              made  distilled  spirits (as is one is apt to get blown-up, poi-
              soned, or shot by the law).

       xclate -m xapply -m -P6 '...' ...
              Force a new xclate  manager  into  the  process  tree  for  this
              instance  of  xapply.  Nested instances of xapply share a common
              manager by default, by explicitly starting a new managed  stream
              we  can  group  all  the  output  of this xapply together in the
              common output.

       eval chmod 0750 \${xcl_$xcl_link%/\*}
              A ksh(1) spell to make the present manager socket visible to our
              group.

       eval chmod 0750 \'dirname \${xcl_$xcl_link}\'
              The same spell under Borne shell, or csh.

       xclate -- -- ls
              The  spell  to skip the xid, one double-dash to end the options,
              one to skip the optional xid.  It is better form to put a  mean-
              ingful xid on each stream, as the default of question-mark ("?")
              is silly.

       xclate -mr -N /tmp/log.0 program ; tr '\000' '\n' /tmp/log.0
              Run a collated program, then replay the exit codes and  xid  for
              each task.

       xclate -mu /var/run/cheater : >>/var/log/cheater &
              Start  a  persistent  diversion  on the socket /var/run/cheater.
              Clients that can connect to that socket can write messages  into
              /var/log/cheater, until someone issues a client with -Q set.

       xclate -m -i "/etc/motd"  tr '[A-Z]' '[a-z]'
              A force xclate to open /etc/motd as stdin to a tr(1).

       xclate -mi "-" <$DATA xapply ...
              All  child  processes  of  the  xapply may try to acquire a file
              descriptor on the $DATA file  with  "xclate  -I".   Note  xapply
              doesn't  implement this since this option would serialize all of
              the child processes.

       xclate -mi "$SOCK_TMP" nc log.example.com 4321
              Use xclate to connect to the local domain socket $SOCK_TMP,  use
              that as stdin to a nc(1), aka netcat, to log.example.com on port
              4321.

       xclate -mi "$SOCK_TMP" sh -c "exec nc log.example.com 4321 1>&0"
              Same as the above, with a duplex connection to force the replies
              back  to the UNIX-domain socket.  Use nc -l -U $SOCK_TMP to cre-
              ate a local end-point (in a window).  In another start a network
              listener  with  nc  -l -p 4321.  In a third window run the above
              command on the same host as the local end-point, please  replace
              "log.example.com"  with  the  name  of the host running the lis-
              tener.  You should be bidirectionally connected between the  two
              windows.   An  interrupt (^C) in the "nc -l -U" window or in the
              xclate window breaks the connection.

BUGS
       This program is very confusing  to  novice  users.   The  complex  file
       descriptor  manipulations  lead  to  cries  of pain and massive denial.
       While xclate can be used over the top of  a  shell,  it  is  considered
       "poor form" to leave one of those just lying around.

       The  "command  mode" is more useful (most of the time) with input redi-
       rected from /dev/null.  This is parallel to the bug  where  ssh(1)  (or
       rsh(1)) reads input it never needs.  There is not a command line option
       to do that, because I think "</dev/null" is clear, and sometimes  you'd
       like to input text from your terminal.

       The  options  -O,  -W, and -N do not truncate any file they are told to
       open, unless you prefix it with a greater-than symbol (>),  which  must
       be quoted from the shell.

       The  name is a play on "xapply", "escalate" and "collate", and that's a
       bug all by itself.

       Some programmers are confused when xclate's %s is not in sync with xap-
       ply's %u, this results from 2 root causes: races between subtasks in an
       xapply with a large parallel factor, and races between  peer  instances
       of  xclate managed tasks unrelated to those started by xapply.  The use
       of %s, in general, is just to let some applications  re-sort  the  col-
       lated output as a post-processing filter.

AUTHOR
       Kevin S Braunsdorf
       NPCGuild.org
       msrc at ksb.npcguild.org

SEE ALSO
       sh(1),  csh(1),  cat(1),  xapply(1l),  hxmd(8l),  ptbw(1l),  wrapw(1l),
       dicer(5l), nc(1), environ(7), mkcmd(1l), make(1)



                                     LOCAL                           XCLATE(1)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | COMMAND MODE OPTIONS | ENVIRONMENT | EXAMPLES | BUGS | AUTHOR | SEE ALSO