-
Notifications
You must be signed in to change notification settings - Fork 16
Examples
mfisk edited this page Sep 13, 2010
·
19 revisions
The classic map-reduce example is to compute word frequency in a text corpus:
fm map -i "/etext/*" "sed -f words.sed | fm split -n 9 |> sort | uniq -c"
Let’s walk through this example step-by-step: First, the FileMap command is “fm”. You must always specify a sub-command and in this case it is “map”, which is used to specify and execute a computation. The “-i” option specifies a wildcard expression that specifies the input files to process. These files need to have been stored in advance into the replicated, virtual filesystem provided by FileMap. To do so, use commands like this:
fm mkdir /etext
fm store /tmp/*.txt /etext/
If you want to follow along, you can download a few e-texts from [http://www.gutenberg.org/ Project Gutenberg ]:
cd /tmp
for f in shplk10 sread10 spoem10 mysky10 leave10 gryms10; do wget http://www.gutenberg.org/dirs/etext05/$f.txt; done
The remainder of the command is an execution pipeline that is applied to each input file. While the Unix pipe (|) symbol is used, each element of the pipeline is actually passed an input filename (the original or intermediate results) and has its stdout and stderr saved in files.
The first pipeline element uses sed to extract a list of words, one per line, from a text document. The words.sed script is as follows:
s/[\"\(\'\-\/]*\([a-zA-Z]*\) */\1\
/g
The “fm split” command partitions an input file into a specified number of files (9 in this case). In this case, a hash is computed on each input line to determine which output partition the line belongs to. Importantly, the same input word always gets put in the same partition. Commands that know how to partition data use the FMOUTPUT environment variable to obtain the output filename base. To that base, they add an extension number for the partition (in this case .1 through .9).
The |> operator is a reduce operation. First, all of the input files prior to this point in the pipeline are redistributed among the nodes. All files with the same suffix will be stored to the same node(s). The command on the right-hand side of the |> operator is responsible for reducing all of the input files for a given partition. This means that it will be executed with a list of input files constituting a partition. Each partition is processed separately (in parallel). In the case of the Unix sort command, it produces a single sorted output from multiple inputs.
Finally, the output of each sorted partition is passed to a “uniq -c” instance, which is the Unix command to tally up the number of identical consecutive lines (words in this case). The “fm map” command implicitly concatenates the outputs of the last stage of the pipeline and outputs them to stdout. In this case, this prints our final word frequency list.
Let’s do some directory listings to look at the input and intermediate files. The format is very similar to traditional POSIX ls, but with a few differences. First, the second column, which would be a hard-link count normally, shows which nodes contained the file or directory (* means all nodes). If more than one node contains a file of the same size, permissions, and ownership, then the file’s modification time is not shown (since it is unlikely to be the same for all nodes). Also, directories are always considered to have a 0 size.
$ fm ls /etext/* /etext/*/*/* /reduce/* -rw-r--r-- 2 mfisk wheel 82910 May 18 2003 /etext/gryms10.txt -rwxr-xr-x 2 mfisk wheel 88862 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0 drwxr-xr-x * mfisk wheel 0 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d -rwxr-xr-x 1,2 mfisk wheel 0 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- * mfisk wheel 14360 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- * mfisk wheel 7794 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 8710 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 9457 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 8994 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 0,2 mfisk wheel 11063 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 0,2 mfisk wheel 12434 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 8735 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 0,2 mfisk wheel 7315 - /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/gryms10.txt.d/sed=20=2Df=20words=2Esed.stderr -rw-r--r-- 1 mfisk wheel 761240 Jun 13 2006 /etext/leave10.txt -rwxr-xr-x 1 mfisk wheel 808239 Jan 04 21:39 /etext/leave10.txt.d/sed=20=2Df=20words=2Esed.0 drwxr-xr-x 1 mfisk wheel 0 Jan 04 21:39 /etext/leave10.txt.d/sed=20=2Df=20words=2Esed.0.d -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:39 /etext/leave10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.new -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:39 /etext/leave10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.stderr -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:39 /etext/leave10.txt.d/sed=20=2Df=20words=2Esed.stderr -rw-r--r-- 2 mfisk wheel 123953 Jun 28 2003 /etext/mysky10.txt -rwxr-xr-x 2 mfisk wheel 131993 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0 drwxr-xr-x * mfisk wheel 0 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d -rwxr-xr-x 1,2 mfisk wheel 0 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- * mfisk wheel 20952 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- * mfisk wheel 10277 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 12864 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 12786 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 15778 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 0,2 mfisk wheel 14487 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 0,2 mfisk wheel 20991 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 11759 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 0,2 mfisk wheel 12099 - /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/mysky10.txt.d/sed=20=2Df=20words=2Esed.stderr -rw-r--r-- 2 mfisk wheel 544127 Jul 25 2003 /etext/shplk10.txt -rwxr-xr-x 2 mfisk wheel 574994 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0 drwxr-xr-x * mfisk wheel 0 - /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d -rwxr-xr-x 1,2 mfisk wheel 0 - /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 87239 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1 -rw-r--r-- 0 mfisk wheel 20480 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 902 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 44811 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2 -rw-r--r-- 0,1 mfisk wheel 12288 - /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 55636 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3 -rw-r--r-- 1 mfisk wheel 28672 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 106 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 59864 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4 -rw-r--r-- 1 mfisk wheel 40960 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 74151 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5 -rw-r--r-- 1 mfisk wheel 65536 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 59487 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6 -rw-r--r-- 0 mfisk wheel 57344 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 0,2 mfisk wheel 84918 - /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1,2 mfisk wheel 57945 - /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 106 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 0,2 mfisk wheel 50943 - /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/shplk10.txt.d/sed=20=2Df=20words=2Esed.stderr -rw-r--r-- 1 mfisk wheel 147020 Jul 02 2003 /etext/spoem10.txt -rwxr-xr-x 1 mfisk wheel 158183 Jan 04 21:39 /etext/spoem10.txt.d/sed=20=2Df=20words=2Esed.0 drwxr-xr-x 1 mfisk wheel 0 Jan 04 21:39 /etext/spoem10.txt.d/sed=20=2Df=20words=2Esed.0.d -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:39 /etext/spoem10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.new -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:39 /etext/spoem10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.stderr -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:39 /etext/spoem10.txt.d/sed=20=2Df=20words=2Esed.stderr -rw-r--r-- 2 mfisk wheel 685019 Sep 03 2003 /etext/sread10.txt -rwxr-xr-x 2 mfisk wheel 725916 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0 drwxr-xr-x * mfisk wheel 0 - /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d -rwxr-xr-x 1,2 mfisk wheel 0 - /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 106 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.0.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 1 mfisk wheel 57344 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1 -rw-r--r-- 2 mfisk wheel 132505 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1 -rw-r--r-- 0 mfisk wheel 53248 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 106 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.1.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 55629 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2 -rw-r--r-- 0,1 mfisk wheel 20480 - /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.2.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 63022 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3 -rw-r--r-- 1 mfisk wheel 32768 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.3.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 72464 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4 -rw-r--r-- 1 mfisk wheel 45056 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 212 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.4.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 84937 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5 -rw-r--r-- 1 mfisk wheel 53248 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 106 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.5.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 74413 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6 -rw-r--r-- 0 mfisk wheel 53248 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.6.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 113323 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7 -rw-r--r-- 0 mfisk wheel 94208 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.7.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 66929 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8 -rw-r--r-- 1 mfisk wheel 57344 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 106 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.8.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rw-r--r-- 2 mfisk wheel 62694 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9 -rw-r--r-- 0 mfisk wheel 53248 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9.d/=25=28FM=29=20_local=20restore=20=2Ds.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.9.d/=25=28FM=29=20_local=20restore=20=2Ds.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.0.d/fm=20split=20=2Dn=209.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:39 /etext/sread10.txt.d/sed=20=2Df=20words=2Esed.stderr -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.0.0 -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.0.0.d/uniq=20=2Dc.0 -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.0.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.0.stderr -rwxr-xr-x 2 mfisk wheel 255056 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.1.0 -rwxr-xr-x 2 mfisk wheel 28640 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.1.0.d/uniq=20=2Dc.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.1.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.1.stderr -rwxr-xr-x 2 mfisk wheel 118511 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.2.0 -rwxr-xr-x 2 mfisk wheel 27978 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.2.0.d/uniq=20=2Dc.0 -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.2.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 2 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.2.stderr -rwxr-xr-x 1 mfisk wheel 83015 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.3.0 -rwxr-xr-x 1 mfisk wheel 22150 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.3.0.d/uniq=20=2Dc.0 -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.3.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.3.stderr -rwxr-xr-x 1 mfisk wheel 108261 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.4.0 -rwxr-xr-x 1 mfisk wheel 23047 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.4.0.d/uniq=20=2Dc.0 -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.4.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.4.stderr -rwxr-xr-x 1 mfisk wheel 143558 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.5.0 -rwxr-xr-x 1 mfisk wheel 23700 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.5.0.d/uniq=20=2Dc.0 -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.5.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.5.stderr -rwxr-xr-x 0 mfisk wheel 136142 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.6.0 -rwxr-xr-x 0 mfisk wheel 24772 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.6.0.d/uniq=20=2Dc.0 -rwxr-xr-x 0 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.6.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 0 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.6.stderr -rwxr-xr-x 0 mfisk wheel 212552 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.7.0 -rwxr-xr-x 0 mfisk wheel 25900 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.7.0.d/uniq=20=2Dc.0 -rwxr-xr-x 0 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.7.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 0 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.7.stderr -rwxr-xr-x 1 mfisk wheel 135783 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.8.0 -rwxr-xr-x 1 mfisk wheel 27814 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.8.0.d/uniq=20=2Dc.0 -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.8.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 1 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.8.stderr -rwxr-xr-x 0 mfisk wheel 123606 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.9.0 -rwxr-xr-x 0 mfisk wheel 27030 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.9.0.d/uniq=20=2Dc.0 -rwxr-xr-x 0 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.9.0.d/uniq=20=2Dc.stderr -rwxr-xr-x 0 mfisk wheel 0 Jan 04 21:40 /reduce/gwoyaVtC5uUIp0UtYhzRMXGEtHI.9.stderr
As you see, the output of a map operation is put in a file whose name is the map operation and whose location is a subdirectory of the input file (subdirectories have the same name as the input file, but with a .d added to the end). For example, the input file /etext/gryms10.txt was used as an input to the "sed -f words.sed operation, so the stdout of that operation would be “/etext/gryms10.txt/sed -f words.sed.0”. To simplify filenames, the [http://en.wikipedia.org/wiki/Quoted_printable Quoted-Printable] encoding is used for all but simple alphanumeric characters in the filename. Thus, the actual file name is sed=20=2Df=20words=2Esed.0. In addition, you will find a file with the .stderr extension that contains the stderr output from the process. At the completion of a map operation, any stderr files for any stage of the entire pipeline are displayed.
Since the output of the sed stage was used as input to the “fm split” stage, a “sed=20=2Df=20words=2Esed.0.d” directory is created. Any command can create multiple output files to support additional downstream parallelism. “fm split” is one of these “partitioning” commands. So in addition to the .0 file for stdout, “fm split” creates a number of output files that have the same base file name, but with additional numbered extensions (.1 through .9 in this case). (All commands are run with the FMOUTPUT environment variable set to the base filename they should use for additional outout).
The reduce operator, “|>” triggers an implicit “fm _local restore” stage that redistributes the data so that all files with the same extension reside on the same node for the reduce operation. The reduce operation itself is placed in a new directory since it is derived from multiple inputs. Each stage of a pipeline is called a job and the “fm jobs” command will list any currently-running jobs. The reduce output is placed in a subdirectory in /reduce which is named by the job-id. (Job IDs are digests of the internal job description files which can be found in /jobs). Reduce operations are run with a list of files as their arguments. In our case, one instance of sort will get all of the .0 inputs, another instance will get all of the .1 inputs, and so on.
The output of the reduce operation is input to the final pipeline stages using this the same process as the initial map operations. When the final stage of the pipeline completes, all of the final output files will be concatenated together and delivered on stdout.