MSRC(8)                                                                MSRC(8)

       msrc - master source remote command

       msrc  [-lwz]  [-preload]  [-Pjobs]  [-d  flags]  [-m  prereq[:postreq]]
       [-f makefile] [-u login] [-y yoke] [hxmd-options] [utility]
       msrc -h
       msrc -V

       msrc [-Algnsz] [-preload] [-B macros] [-C  configs]  [-D  name[=value]]
       [-d  flags]  [-E  compares]  [-f  makefile]  [-G  guard]  [-I  dirname]
       [-j m4prep] [-k key] [-m  prereq[:postreq]]  [-o  attributes]  [-Pjobs]
       [-R  req]  [-U  name]  [-u  login]  [-X  ex-configs] [-y yoke] [-Y top]
       [-Z zero-config] [utility]

       Think of this as a network-aware addition to make.  The goal is to  run
       a  recipe  on  every host in a (potentially long) list of hosts.  To do
       that we are going to synchronize some files from the current  directory
       to  a  remote  host, then run the utility on that remote host, from the
       updated directory.

       This is really a front-end for hxmd which makes it less  cumbersome  to
       call  rdist(1),  make(1)  and ssh(1) to update many target hosts.  This
       program completes the refactoring and replacement of distrib(8l) as the
       basis  of  "ksb's master source" software distribution and installation

       To send the appropriate files to the target host  msrc  classifies  all
       the files in the current directory into a few classes:

       Those that are sent as-is (SEND)
              Each file is updated as an exact replica on every target host.

       Those that are ignored (IGNORE)
              None of these are updated on the target host.

       Those that are macro processed for each target (MAP)
              Each file is run through m4(1) with attribute macros defined for
              each target host.  The resulting stream is sent  to  the  target
              host  for additional processing.  If the node listed is a direc-
              tory hxmd uses the file Cache.m4 as  a  marked-up  recipe  file,
              which it processes though m4, then uses to update the directory.
              The output from that command is sent to the  target  host.   See

       Those that contain options to hxmd (HXINCLUDE)
              These are used to tune parameters to the underlying hxmd machin-

       To do this msrc consults macro values from the makefile specified.   It
       appends  a  generated target to a copy of that recipe file.  The gener-
       ated target name is the basename of the program prefixed with 2  under-
       scores  ("__"),  so  the  default  one would be "__msrc".  That double-
       underscore target outputs the value of 7 make macros: INTO, MODE, SEND,
       MAP,  SUBDIR, IGNORE, and HXINCLUDE.  When some of these are left empty
       msrc tries to deduce the proper disposition  of  each  filename.   This
       chicanery  allows the author of the recipe file to use any other target
       for the work at hand, and put explicit dependencies in the recipe  file
       for the generated target.

       Usually any explicit dependencies create payload files just before they
       are transmitted to the target hosts.

       Msrc constructs two files from the extracted  make  macro  definitions,
       the  utility  provided, and the filenames in the current working direc-
       tory.  All of this information is used to create an  hxmd(8l)  instance
       to  select  the target hosts, update each, then execute utility on each

       The first file constructed is an rdist distfile, with  m4  mark-up,  to
       drive the data synchronization phase of the update.  The second file is
       a shell script provided as the control parameter  to  hxmd.   It  calls
       rdist -f on the m4 processed distfile to update the remote target, then
       ssh to obtain a shell on the remote target, source  any  entry  defini-
       tions  file,  and run the utility.  In each of these there are substan-
       tial m4 hooks to refine command-line parameters to effect  each  facil-

       Any  make  macros explicitly specified in the makefile are preferred to
       generated values constructed from the list  of  files  in  the  current
       directory.   Any  macro  given  a  non-empty value in the makefile will
       never be adulterated by msrc, unless it contains the meta  token  "+++"
       (three  plus  signs).   All  macros left empty (or undefined) assume as
       many of the remaining files as possible, based on the rules below.

       The SEND, MAP, and SUBDIR macros may contain one of  two  meta  tokens:
       +++  specifically  requests  that  unassigned files be appended to that
       macro, or . to specifically requests  that  no  (additional)  files  be
       assigned  to  that  macro.   (Typically  MAP=.  would  avoid  any macro
       replacement on every local file ending in ".host".)

       The default utility, is given in the -V output, by default it  is  make
       with  no  additional  parameters.  That strongly implies that we send a
       make(1) recipe file called either "makefile" or "Makefile" to the  tar-
       get hosts.

       Note  that  (while not listed) any option which might be passed to hxmd
       is allowed in msrc's command-line.  We actually call hxmd with our pro-
       gram name forced as argv[0].  Since hxmd accepts a default zero config-
       uration file via the basename in argv[0] we are changing  that  default
       value.   We don't do this when we see an explicit -Z passed on our com-

              The hxmd instance created to process the request  generates  all
              the temporary files in advance of the process launch.  The inte-
              ger sets the number of tasks that are  prepared  for  launch  in
              addition to number of parallel tasks (under -P below).

              Passed  to  xapply via hxmd to present ptbw tokens on the end of
              the parameter list.  This requires some  detailed  knowledge  of
              the  command-line  given  to xapply, and it not for the faint of
              heart.  To be truly useful an instance of ptbw must  already  be
              in-scope, or set via -t.

       -d P
              If  any  of flags contain the upper case letter 'P' (P for posi-
              tion) msrc traces the modifications made to the  original  make-
              file, as well as the disposition of each file in the macro lists
              on stderr.

       -d S
              If any debug option to m4 (passed  through  hxmd)  contains  the
              upper  case letter 'S' (S for send scripts) msrc outputs the two
              files constructed for hxmd on stderr (as shell  here-documents).

       -d X
              The  "trace  execution"  switch to hxmd also traces the command-
              line given to hxmd, with the argv masking shown.

       -f makefile
              Specify the make(1) recipe file msrc studies to extract any make
              macro  definitions.  We search for the files and
              before make's defaults of makefile and  Makefile.   This  allows
              the  local  control  logic to be separated from the remote logic
              (or not).

       -F literal
              This option is forced by msrc when calling hxmd so it may not be
              presented on the command-line.

              Print a brief help message.

              The  environment  built  resides on the local filesystem, rather
              than the remote host's, and the utility is run from that  tempo-
              rary directory.  This allows the use of some other file synchro-
              nization utility (rather than rdist) to update the remote  host.
              See "LOCAL MODE" below.

       -m prereq[:postreq]
              Specify  the  double-underbar  target  used to extract any macro
              values.  This is intended for scripts built on top of  msrc,  so
              use the default "__msrc" for simple tasks.

              The  optional  postreq  is  updated with make after the complete
              build with the macro MSRC_EXITCODE defined as the  integer  exit
              code  from  the  hxmd  update process.  In that case the cleanup
              process overlays the original msrc.  A trailing colon implies  a
              postreq of "__clean".

              Specify  the number of tasks to run in parallel, default 6.  See
              BUGS below.

       -u login
              If login is not the empty string it specifies  a  default  value
              for the ENTRY_LOGIN m4 macro attribute.

              Show version information, and some tuning information.

              This  flag forces the new xclate plunder logic, even if the name
              of the program is "mmsrc".  Since the program  is  installed  as
              "msrc" the option normally has no effect.

       -y yoke
              Each  yoke  specification  presents  a  command-line word to the
              invocation of make used to  plunder  the  makefile,  these  tie-
              points  allow one make recipe file to syzygy several subdirecto-
              ries.  For example to change the INTO macro in each  directory's
              control  logic.   This  option  is  most  often set in a PRE_CMD
              recursively call to msrc for each nested product.

              This option removes all the  files  from  the  macro  HXINCLUDE.
              This  saves  editing  effort when debugging or reusing code, but
              not messing with any $HXMD_PASS environment already set.

       -Z config
              Unless this option is presented the argument  vector  built  for
              hxmd  indicates  the name of the program is the same as the name
              msrc was called.  The has two effects: first error messages look
              like they come from "msrc" rather than "hxmd", second this tells
              hxmd to look for a default zero configuration file with our pro-
              gram  name  as  the  base.   Thus any local configuration may be
              selected by default (and be distinct from hxmd's defaults).

       This is a list of the make(1) macros expected, and  how  each  is  con-
       structed or derived.

              This is the name of our remote source cache on each target host.
              A default value is  available  only  when  the  current  working
              directory contains the string "/msrc/".  In that case the string
              '/msrc/' in that string is  replaced  with  '/src/'  to  form  a
              default  value.   Otherwise  the  makefile  must  set  a  single
              explicit value.  The explicit value dot (.)  forces  the  target
              directory to be the remote login's home directory.  This creates
              unexpected peril, as it is very easy to delete all the files  in
              that directory.

              This  allows  the makefile to request local mode.  Otherwise the
              invoker must specify the command-line option -l, which they  are
              apt  to  forget.   There  explicit  values are allowed: "local",
              "remote" and "auto".  Local adds -l to the command-line, if  not
              specified.   Remote forbids the use of -l, as well as the use of
              mmsrc(8l).  Auto allows either, and is the  default  value  when
              none is provided.

       All  the  macros  below  should  consist  of filenames from the current
       directory, or possibly  regular  expressions  (for  IGNORE)  which  are
       teated  as  literals  in  this phase.  Use an absolute or relative path
       only with care.

       Next msrc assembles a list of all the  local  filenames  which  do  not
       already  appear  in any of the macros below (as given by the makefile).
       The code in msrc deduces the disposition of each filename first  elimi-
       nates  any  that  already  appear in a defined macro, then includes any
       remaining filenames in exactly one of these  macros,  which  must  have
       started off undefined (or empty):

              The  is constructed from the list of directory nodes in the cur-
              rent directory that are not any  of  "RCS",  "SCCS",  "CVS",  or
              "SVN".   As  always this macro is left as-is when it is provided
              in the makefile.  These directories are added to  the  generated
              distfile, see below.

       For  any  non-directory  node  we select the first macro not set in the
       makefile which might accept each unmatched file.   Note  that  a  macro
       accepts all unmatched files, accepting one doesn't change the fact that
       the makefile didn't provide a value.

              Any file or directory ending in ".host" is added to  MAP  macro,
              unless  that  macro  was  given in the makefile (without the +++
              markup).  The ".host" suffix is removed on the remote host,  and
              the  resulting name is added to IGNORE.  The name ".host" itself
              is left untouched to avoid an empty filename on the remote host.
              Any  file mapped by this macro is run though hxmd's m4 expansion
              for the files parameter, the  results  thus  generated  are  the
              contents of the remote version.

              Any  filename  ending  in  ".hxmd"  is added to HXINCLUDE macro,
              unless that macro was given in the makefile.   This  is  a  list
              files containing options to hxmd.  The special name dot (".") is
              ignored so that you can force the macro to a  fixed  value  that
              does not tamper with the environment.  A single file a with only
              a double-dash ("--") in it forces the variable to be set to  the
              empty string (which hxmd ignores).  Lines in each file beginning
              with an octothorp ("#") are ignored, and newlines are  converted
              to spaces.

              Any other name is added to the SEND macro, unless that macro was
              given in the makefile.

              Any name not yet consumed is added to IGNORE, unless that  macro
              was given in the makefile.

       The source paths provisioned in SEND and MAP provide data files for the
       processed files in the platform directory, but the names of  the  files
       in  the  platform  directory  may be spelled differently in the target.
       There are two different ways to compress the source path.

       Any source path which contains an explicit mention of the dot  link  in
       its  path  (as "/./") has the leading elements up to that point removed
       in the destination  path.   The  path  "/usr/local/lib/other/./chase.c"
       when   sent   to   "/tmp/lark"   results   in   the   destination  path
       "/tmp/lark/chase.c".  If this rule applies to a path the next rule can-
       not be applied.  This is copied from rsync's markup rules.

       Any  source  path  that starts with a reference to the parent directory
       (as "../") has that element removed in the destination path.  This rule
       is  repeated  until all leading dot-dot elements are removed.  To avoid
       this rule prefix the path with "././" and  the  previous  rule  applied
       first.   This  is based on the operational aspect of msrc which says we
       are updating a target directory, not  any  sibling  directories.   This
       allows  many  subdirectories  to  share  a  common file from the parent

       The distfile construction builds 3 labeled  file  distribution  stanzas
       one  for  "myself",  one  for "dirs" and one for "files".  See EXAMPLES
       below for a spell to look at the constructed distfile.  In  outline  it
       consists  of  a stanza to replicate the current directory, the directo-
       ries in SUBDIR (without descending into them) then replicate the  files
       from  from SEND excepting the patterns from IGNORE.  Then for each file
       in MAP a separate stanza is constructed to copy the m4  processed  file
       to the new name on the remote host.

       The  script construction builds a long m4 file with many ifdef hooks in
       it (see M4 HOOKS below) to help fine-tune  the  actions.   The  utility
       specified on the msrc command-line is quoted to preserve the meaning as
       it is passed through both m4 and the ssh shell command evaluation.

       Under -l we build a shell command to allocate  a  temporary  directory,
       fill  it  with the processed files from the local master, change direc-
       tory to that temporary work area, run utility,  then  remove  the  work
       area.  This allows processing for nodes that are unwilling or unable to
       accept rdist or ssh connections.  This also allows meta-processes  that
       should  run in a context that represents each target host, but need not
       be resident on that host.

       Tactics to think about at this point:

       ?   use rsync to update a host from the cached sources

       ?   stash an archive (tar(1) comes to mind) of the processed files away
           for the client to fetch later

       ?   run  a report function for the host, pulling data back to the local

       ?   create a nested instance of msrc to process the  source  again  for
           clients related to the target host

       The  ephemeral  work  areas are selected from a pool we construct under
       $TMPDIR, via a ptbw we install in the process tree (using -d as not  to
       violate  any  use  of that wrapper the client might need).  The same m4
       hooks are presented to the local process,  which  is  always  run  with
       stdin redirected from /dev/null.

       The  ptbw  instance  is  omitted by msrc, as an optimization, under -P1
       since only one directory may be active at a time.

       The catenated  text  of  the  files  in  HXINCLUDE  sets  the  variable
       $HXMD_PASS  in  the environment of the new hxmd instance.  This is used
       to pass options down to the m4 file processing, or  configuration  file
       parts of hxmd while preserving any use of $HXMD (under -l).

       For compatibility with mmsrc, please only use the set of options common
       to both mmsrc and hxmd, with the addition of -z to  prevent  $HXMD_PASS
       from   lingering  in  the  environment.   Options  to  enclosed  xclate
       instances, like -T, won't work when using mmsrc to boot-strap.

       The command-line passed to hxmd process always includes a specific  -P,
       any  -d  flags,  an  explicit  -F0, the per-host update script, the MAP
       files and the provision script.  The  process  may  be  wrapped  in  an
       instance  of  ptbw  -d or not, but in either case the detached instance
       environment variable is preserved (viz. $ptbw_d).

       Since hxmd makes extensive use of m4(1) we provide many m4 macro  defi-
       nitions  as  hooks  that  your  config (given under -C, -X or -Z) might
       specify to extend or change the behavior of the processing.  These  may
       also  be  specified  a  -D  parameters  to  m4  on the command-line, or
       included in a m4prep file under -j.

              The name of the secure shell to use for remote target,  defaults
              to "ssh".  Note that most version of rdist want an absolute path
              to the remote shell program: always set this in your  configura-
              tion file.

              A  shell  word that represents the command rdist should use as a
              path to the remote shell program.  Be aware  the  rdist  doesn't
              like  spaces in this value.  It is presented to SDIST below with
              a "-P" prefix.  Defaults to "SSH" or  some  other  local  value.
              (This is only for distrib compatibility.)

              The path to rdist (or a work-alike) on the local host, otherwise
              "rdist" (This is distrib compatible.)

              The path to rdistd(8) on the  remote  host  presented  to  SDIST
              below  prefixed with "-p", otherwise not used.  (This is distrib

              The combination of RDIST_PATH, RSH_COMMAND, RDISTD_PATH and  the
              command-line option to introduce the constructed distfile.

              A  filename  to source (via the . command) to set shell variable
              definitions in the remote shell, like $PATH or $TZ.  The default
              value should be set by the zero configuration file as local site
              policy allows.  The  default  is  none,  but  might  be  set  to
              "/usr/local/lib/distrib/local.defs"   if   you  previously  used
              "smsrc" or dmz(8l), which are example scripts I use.

              Our remote login name.  When defined msrc adds ENTRY_LOGIN@ as a
              prefix on the HOST macro hxmd defines for us.

              Placed  after the loader line, only when defined, this hook pro-
              vides a way to introduce m4 include markup at  the  top  of  the
              file.   The  macro  parameter ($1) is either the word "local" or
              the word "remote".

              Placed before we update the target directory, only when defined.
              This  provides  a hook to debug the shell code.  An example com-
              mand line usage might look like -DINIT_CMD='set -x'.

              Placed just before the invocation of utility and after the  con-
              struction  of  the  working  directory,  only when defined.  For
              example this command might recurse into some  subdirectories  to
              prepare  them  for the update of the current directory.  If this
              invokes a shell exit, then we never run utility, which might  be
              exactly what you wanted.

              Placed  after  the  utility  to  run  some cleanup action on the
              localhost, when defined.   Almost  never  defined  in  practice.
              This  might  be  used to force hxmd to see a specific exit code,
              for retry logic.

              This is defined on the hxmd command-line as  either  "local"  or
              "remote".   It  would  be poor form to provide any other defini-

       In the context of the three "_CMD" macros above 5 shell parameters  are
       available to transmit our details to recursive calls to msrc:

       ${1} -- where
              Under -l this is the local cache directory, otherwise ".".

       ${2} -- which
              The make recipe file used to process this directory.

       ${3} -- how
              The  mode  ("local"  or  "remote")  specified,  "auto" is always
              replaced with the selected mode.

       ${4} -- who
              The credentials suggested to access the target host,  either  as
              login@host  or  host.   In  local mode the string "localhost" is
              suggested, which doesn't really imply that a local ssh login  is

       ${5} -- why
              The value of INTO, either from the makefile, a yoke, or from the
              default rule.  This is the directory we need to  update  eventu-
              ally (under -l for example).

       ${6}... -- ptbw tokens under -R
              The value of the tokens recovered by xapply under -R.

       Note  that the spelling of these parameters is important as m4 consumes
       similar names as parameters to its macros.  Viz. when they are  spelled
       without the curly braces they are usually replaced by m4 with the empty

       Under local mode (-l) these same parameters are available to the  util-
       ity itself.  In remote mode they are presently not available.

       Rather  than using the three "_CMD" macros one may use m4 divert markup
       to include preparatory code above, or  actions  in  the  remote  update
       script.   Diversion  0  usually  includes  shell  function definitions.
       Diversion 2 inserts code  after  the  positional  parameters  are  set.
       Diversion  4  inserts  code  before INIT_CMD.  Diversion 6 inserts code
       before the PRE_CMD.  Diversion 8 inserts code before the POST_CMD.

       The common convention for the above is to use the INCLUDE_CMD macro  to
       include a file with diverted shell commands.  Anther way to use this is
       to include a macro definition of INCLUDE_CMD in a file under -j,  which
       should  be  ignored  by any other markup.  For additional clues see the
       output from -d S.

       Get rid of the distracting noise from rdist with a shell redirection in
       RDIST_PATH:  unlike  the  -p and -P options to rdist the mark-up we use
       allows the redirection.  Just use a configuration line like:
              RDIST_PATH='rdist -q >/dev/null'
       to squelch the normal rdist output, but not the errors.  The real  clue
       here  is that a shell redirection can appear within the parameter list,
       which some folks never remember.

       Since hxmd is documented to run all the m4 processing in sequence  (one
       slot at a time, left to right) it is safe to depend on this behavior in
       msrc.  But this is only true  if  you  define  MAP  yourself,  if  msrc
       defines  MAP  it  may  select  a different order for the files (left to
       right) than you'd expect.

       In the remote case the distfile is always placed after any  MAP  files,
       and  the script is always before them.  Under -l the code to select the
       ephemeral directory is always first, the  code  to  update  that  local
       directory is always last.

       msrc -V
              Display the standard version information and the default control

       msrc -C/dev/null -d S true 2>&1 |less
              Look at the constructed distfile, and  control  script  for  the
              current directory (but don't update any hosts).

       msrc -C/dev/null -l -d S true 2>&1 |less
              Look  at  the  constructed  control  for  a local update of this

       msrc -C/dev/null -d P true 2>&1 |less
              Look at the constructed double-underscore makefile  target,  and
              make  macros for the current directory, as extracted or derived.

       msrc -B HASSRC -C -- make all
              Update all hosts from  with  the  macro  HASSRC  defined.
              This is very much like what distrib did under -S.

       cd final && msrc -ly INTO=/tmp/space/final -B ... make source
              Suggest  a value of INTO for the directory final that places the
              platform source under our /tmp/space directory.  The  source  is
              positioned  in  an  ephemeral platform directory, then the "make
              source" recipe must copy it into "/tmp/space/final".  For a more
              direct route use mmsrc as:

              cd final && mmsrc -y INTO=/tmp/space/final -B ... :

              which builds the platform source directly at INTO.

       msrc -d SPX -d fl ...
              Since  hxmd only passes the last -d down to m4 the second option
              clears the unknown flags (which  prevents  m4  from  carping  on
              Solaris hosts).

       # $Install(*): msrc %s -DENTRY_LOGIN=bob op make %M
              Apply mk(1l) to update each host as "bob" (the builder) then use
              op(1l) to escalate privileges to install the results.   This  is
              very  similar  to  the common control line in each distrib-based
              master source Makefile.

       msrc -C
              This is the simplest example I could  build:  given  a  list  of
              hosts in the file
                   SSH="/usr/bin/ssh" # local absolute path to ssh(1)
              and a recipe in Makefile:
                        date; pwd; uname -n
              we  output  the  date,  current  working directory, and hostname
              (note the double-dollars so make passes a dollar to the  shell).

       msrc -l -C
              The  same  as above, but us a local ephemeral working directory,
              rather than the defined INTO.

       msrc -zP10 -C -C -B2 ./regression \*.data
              Use hxmd to intersect and,  run  the  regression
              program  on each host against *.data.  Take those hosts 10 wide.
              Ignore the HXINCLUDE in the local make recipe.

       msrc -zP10 -C -m :purge -B2 ./regression \*.data
              Almost the same as above, but trigger a purge(8l) target in  the
              control recipe file after the show is over.

       msrc -lR2 -ttokens -Gsulaco -DPRE_CMD='echo ${6},${7}' ...
              Output  the  pair  of  ptbw tokens bound to this update from the
              tokens file, before we run any update for "sulaco".  Usually the
              echo(1)  is  replaced  by  a script which does some task-related

       The use of the make target "__msrc" while plundering  the  makefile  is
       presumptuous.  This is also a feature, as the makefile may force depen-
       dencies on that target to create a  powerful  "preparation  work"  fea-

       Because  the  shell  won't  duplicate a file descriptor greater than 9,
       msrc depends on having at least 1 file-descriptor in the range 3  to  9
       available for plundering the makefile.

       We  do  not guarantee that we call make exactly once.  Depending on the
       availability of low numbered file descriptors  in  the  msrc  processes
       environment  we  may  execute multiple instances of make, and therefore
       any dependent checks.

       It is also a fine-line feature that renaming (linking to) msrc  is  the
       only way to change the default zero-configuration file.

       Maybe  the  positional  parameters  should be available to remote mode.
       This would require much more quoting in the remote execution phase.  It
       also  breaks  due to ssh's command trapping, which double evaluates the
       quotes in the SSH_ORIGINAL_COMMAND string.  If you  want  them  on  the
       remote side, then send them yourself.

       We  should really get the default for -P from ptbw.  That would make it
       all so well connected.  But you can do it with -P given the output of:
              ptbw -V | sed -n -e 's/.*tokens: //p'

       Before hxmd is installed (and xapply, xclate, et cetera) it is not pos-
       sible  to  install  this  program with itself: you'll have to use mmsrc
       (mini-msrc) to get the ball moving.   Use  the  msrc_base  package,  or
       there is a README and a web page to help you in the source directory.

       KS Braunsdorf, NonPlayer Character Guild
       msrc swirl ksb notspamDot

       sh(1),  m4(1), rdist(1), make(1), rsync(1), rdist(1), ssh(1), hxmd(8l),
       xapply(1l), xclate(1l), ptbw(1l), mmsrc(8l), efmd(8l), mk(1l)

                                     LOCAL                             MSRC(8)