-
Notifications
You must be signed in to change notification settings - Fork 16
Examples
The classic map-reduce example is to compute word frequency in a text corpus:
fm map -i "/etext/*" "egrep -o '[A-Za-z]+' | fm partition -Z -f 1 -n 9 <> sort | uniq -c"
Let’s walk through this example step-by-step: First, the FileMap command is “fm”. You must always specify a sub-command and in this case it is “map”, which is used to specify and execute a computation. The “-i” option specifies a wildcard expression that specifies the input files to process. These files need to have been stored in advance into the replicated, virtual filesystem provided by FileMap. To do so, use commands like this:
fm mkdir /etext
fm store /tmp/*.txt /etext/
If you want to follow along, you can download a few e-texts from Project Gutenberg
(Mac users can use curl -O instead of wget);
cd /tmp
for f in shplk10 sread10 spoem10 mysky10 leave10 gryms10; do wget http://gutenberg.readingroo.ms/etext05/$f.txt; done
The remainder of the command is an execution pipeline that is applied to each input file.
The first pipeline element uses egrep to extract a list of words, one per line, from a text document.
The “partition” command partitions an input file into a specified number of files (9 in this case). In this case, a hash is computed on each input line to determine which output partition the line belongs to. Importantly, the same input word always gets put in the same partition.
The <> operators do a shuffle and reduce. First, all of the input files prior to this point in the pipeline are redistributed among the nodes. All files with the same suffix will be stored to the same node(s). The command on the right-hand side of the > operator is responsible for reducing all of the input files for a given partition. This means that it will be executed with a list of input files constituting a partition. Each partition is processed separately (in parallel). In the case of the Unix sort command, it produces a single sorted output from multiple inputs.
Finally, the output of each sorted partition is piped to a “uniq -c” instance, which is the Unix command to tally up the number of identical consecutive lines (words in this case). The “fm map” command implicitly concatenates the outputs of the last stage of the pipeline and outputs them to stdout. In this case, this prints our final word frequency list.
Let’s do some directory listings to look at the input and intermediate files. The format is very similar to traditional POSIX ls, but with a few differences. First, the second column, which would be a hard-link count normally, shows which nodes contained the file or directory (* means all nodes). If more than one node contains a file of the same size, permissions, and ownership, then the file’s modification time is not shown (since it is unlikely to be the same for all nodes). Also, directories are always considered to have a 0 size.
$ ./fm ls -l /etext/* /etext/* /etext/*/* /etext/*/*/* /etext/*/*/*/* /etext/*/*/*/*/* /reduce/*/* /reduce/*/*/* reduce/*/*/*/* reduce/*/*/*/*/* -rw-r--r-- 1 mfisk wheel 8.3e4 May 26 20:27 /etext/gryms10.txt drwxr-xr-x 3 mfisk wheel 0 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 drwxrwxr-x 2 mfisk wheel 0 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 -rw-rw-r-- 1 mfisk wheel 4.8e2 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/.status -rw-r--r-- 1 mfisk wheel 1.1e4 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 1 mfisk wheel 1.1e4 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 1 mfisk wheel 8.2e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/2 -rw-rw-r-- 2 mfisk wheel 8.9e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/3 -rw-r--r-- 1 mfisk wheel 6.2e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-rw-r-- 1 mfisk wheel 6.2e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-r--r-- 1 mfisk wheel 8.4e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-rw-r-- 1 mfisk wheel 8.4e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-r--r-- 1 mfisk wheel 9.9e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-rw-r-- 1 mfisk wheel 9.9e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-r--r-- 1 mfisk wheel 6.7e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 1 mfisk wheel 6.7e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 1 mfisk wheel 8.9e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/8 -rw-r--r-- 1 mfisk wheel 8.1e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-rw-r-- 1 mfisk wheel 8.1e3 May 26 20:50 /etext/gryms10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-r--r-- 1 mfisk wheel 3.1e2 May 26 20:27 /etext/leave10.txt drwxr-xr-x 3 mfisk wheel 0 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 drwxrwxr-x 2 mfisk wheel 0 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 -rw-rw-r-- 1 mfisk wheel 4.7e2 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/.status -rw-r--r-- 1 mfisk wheel 13 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 1 mfisk wheel 13 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 1 mfisk wheel 23 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/2 -rw-rw-r-- 2 mfisk wheel 7 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/3 -rw-r--r-- 1 mfisk wheel 32 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-rw-r-- 1 mfisk wheel 32 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-r--r-- 1 mfisk wheel 33 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-rw-r-- 1 mfisk wheel 33 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-r--r-- 1 mfisk wheel 25 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-rw-r-- 1 mfisk wheel 25 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-r--r-- 1 mfisk wheel 28 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 1 mfisk wheel 28 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 1 mfisk wheel 59 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/8 -rw-r--r-- 1 mfisk wheel 22 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-rw-r-- 1 mfisk wheel 22 May 26 20:50 /etext/leave10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-r--r-- 1 mfisk wheel 1.2e5 May 26 20:27 /etext/mysky10.txt drwxr-xr-x 2 mfisk wheel 0 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 drwxrwxr-x 3 mfisk wheel 0 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 -rw-rw-r-- 1 mfisk wheel 4.8e2 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/.status -rw-rw-r-- 1 mfisk wheel 1.6e4 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 2 mfisk wheel 1.3e4 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/2 -rw-rw-r-- 2 mfisk wheel 1.2e4 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/3 -rw-r--r-- 1 mfisk wheel 9.9e3 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-rw-r-- 1 mfisk wheel 9.9e3 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-rw-r-- 1 mfisk wheel 1.4e4 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-rw-r-- 1 mfisk wheel 1.7e4 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-r--r-- 1 mfisk wheel 9.1e3 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 1 mfisk wheel 9.1e3 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 2 mfisk wheel 1.1e4 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/8 -rw-r--r-- 1 mfisk wheel 1.2e4 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-rw-r-- 1 mfisk wheel 1.2e4 May 26 20:50 /etext/mysky10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-r--r-- 1 mfisk wheel 5.4e5 May 26 20:27 /etext/shplk10.txt drwxr-xr-x 3 mfisk wheel 0 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 drwxrwxr-x 2 mfisk wheel 0 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 -rw-rw-r-- 1 mfisk wheel 4.8e2 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/.status -rw-r--r-- 1 mfisk wheel 7.4e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 1 mfisk wheel 7.4e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 1 mfisk wheel 5.5e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/2 -rw-rw-r-- 2 mfisk wheel 5.7e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/3 -rw-r--r-- 1 mfisk wheel 4.4e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-rw-r-- 1 mfisk wheel 4.4e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-r--r-- 1 mfisk wheel 6.1e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-rw-r-- 1 mfisk wheel 6.1e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-r--r-- 1 mfisk wheel 7.9e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-rw-r-- 1 mfisk wheel 7.9e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-r--r-- 1 mfisk wheel 5e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 1 mfisk wheel 5e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 1 mfisk wheel 5.1e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/8 -rw-r--r-- 1 mfisk wheel 4.6e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-rw-r-- 1 mfisk wheel 4.6e4 May 26 20:50 /etext/shplk10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-r--r-- 1 mfisk wheel 1.5e5 May 26 20:27 /etext/spoem10.txt drwxr-xr-x 3 mfisk wheel 0 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 drwxrwxr-x 3 mfisk wheel 0 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 -rw-rw-r-- 1 mfisk wheel 4.8e2 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/.status -rw-r--r-- 1 mfisk wheel 2.2e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 1 mfisk wheel 2.2e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 2 mfisk wheel 1.3e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/2 -rw-rw-r-- 2 mfisk wheel 1.4e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/3 -rw-r--r-- 1 mfisk wheel 1.2e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-rw-r-- 1 mfisk wheel 1.2e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-r--r-- 1 mfisk wheel 1.5e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-rw-r-- 1 mfisk wheel 1.5e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-r--r-- 1 mfisk wheel 1.9e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-rw-r-- 1 mfisk wheel 1.9e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-r--r-- 1 mfisk wheel 1.3e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 1 mfisk wheel 1.3e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 2 mfisk wheel 1.4e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/8 -rw-r--r-- 1 mfisk wheel 1.2e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-rw-r-- 1 mfisk wheel 1.2e4 May 26 20:50 /etext/spoem10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-r--r-- 1 mfisk wheel 6.9e5 May 26 20:27 /etext/sread10.txt drwxrwxr-x 3 mfisk wheel 0 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 drwxr-xr-x 2 mfisk wheel 0 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9 -rw-rw-r-- 1 mfisk wheel 4.8e2 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/.status -rw-rw-r-- 1 mfisk wheel 8.7e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-r--r-- 1 mfisk wheel 8.7e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/1 -rw-rw-r-- 2 mfisk wheel 6.2e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/2 -rw-rw-r-- 2 mfisk wheel 6.5e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/3 -rw-rw-r-- 1 mfisk wheel 5.2e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/4 -rw-rw-r-- 1 mfisk wheel 6.3e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-r--r-- 1 mfisk wheel 6.3e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/5 -rw-rw-r-- 1 mfisk wheel 8.7e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-r--r-- 1 mfisk wheel 8.7e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/6 -rw-rw-r-- 1 mfisk wheel 5.4e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-r--r-- 1 mfisk wheel 5.4e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/7 -rw-rw-r-- 2 mfisk wheel 6.2e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/8 -rw-rw-r-- 1 mfisk wheel 5.6e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-r--r-- 1 mfisk wheel 5.6e4 May 26 20:50 /etext/sread10.txt.d/egrep+-o+=5BA-Za-z=5D=2B+=7C+fm+partition+-Z+-f+1+-n+9/9 -rw-rw-r-- 1 mfisk wheel 3.8e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/1.d/.status -rwxrwxr-x 1 mfisk wheel 2.9e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/1.d/stdout -rw-rw-r-- 1 mfisk wheel 3.6e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/2.d/.status -rwxrwxr-x 1 mfisk wheel 2.8e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/2.d/stdout -rw-rw-r-- 1 mfisk wheel 3.8e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/3.d/.status -rwxrwxr-x 1 mfisk wheel 2.9e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/3.d/stdout -rw-rw-r-- 1 mfisk wheel 3.8e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/4.d/.status -rwxrwxr-x 1 mfisk wheel 2.9e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/4.d/stdout -rw-rw-r-- 1 mfisk wheel 3.8e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/5.d/.status -rwxrwxr-x 1 mfisk wheel 2.9e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/5.d/stdout -rw-rw-r-- 1 mfisk wheel 3.8e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/6.d/.status -rwxrwxr-x 1 mfisk wheel 3e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/6.d/stdout -rw-rw-r-- 1 mfisk wheel 3.6e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/7.d/.status -rwxrwxr-x 1 mfisk wheel 2.8e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/7.d/stdout -rw-rw-r-- 1 mfisk wheel 3.8e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/8.d/.status -rwxrwxr-x 1 mfisk wheel 2.9e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/8.d/stdout -rw-rw-r-- 1 mfisk wheel 3.8e2 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/9.d/.status -rwxrwxr-x 1 mfisk wheel 2.9e4 May 26 20:51 /reduce/A9eqAhCbC2F3Cyv1oBX+LAF_qOU/9.d/stdout 3.3 MB
As you see, the output of a map operation is put in a file whose name is the map operation and whose location is a subdirectory of the input file (subdirectories have the same name as the input file, but with a .d added to the end). For example, the input file /etext/gryms10.txt was used as an input, the output will be placed in a new subdirectory of “gryms10.txt.d”. Since the command run was “egrep -o ‘[A-Za-z]+’ | fm partition -Z -f 1 -n 9”, a subdirectory is created from that command string, but in a Quoted-Printable encoding is used for all but simple alphanumeric characters in the filename. Thus, the actual path is /etext/gryms10.txt.d/egrep+-o+=5CS=2B+=7C+fm+partition+-Zn+9/.
Any command can create multiple output files to support additional downstream parallelism. “fm partition” is one of these “partitioning” commands and creates a number of output files numbered 1 through 9. (All map commands are run with their current working directory set to the output directory).
In addition, you may find files with the names stdout or .stderr if the process wrote to stdout or stderr. Finally, a .status file is created at completion containing statistics about the execution of the process.
The shuffle operator, “<” (visualize the symbol as a fan-out) redistributes the data so that all files from the previous stage of the pipeline with the same basename reside on the same node for the reduce operation. The reduce operation itself is placed in a new directory since it is derived from multiple inputs. Each stage of a pipeline is called a job and the “fm jobs” command will list any currently-running jobs. The reduce output is placed in a subdirectory in /reduce which is named by the job-id. (Job IDs are digests of the internal job description files which can be found in /jobs). Reduce operations are run with a list of files as their arguments. In our case, one instance of sort will get all of the 0 inputs, another instance will get all of the 1 inputs, and so on.
The output of the reduce operation is input to the final pipeline stages using this the same process as the initial map operations. When the final stage of the pipeline completes, all of the final output files will be concatenated together and delivered on stdout.