datastat.1

.TH datastat 1
.SH NAME
datastat \- simple data statistics on numeric column files
.SH SYNOPSIS
.B datastat
[options] [filename]
.SH OPTIONS
  \-h|\-\-help ....... This help message
  \-\-no\-avg ........ Suppress average in output
  \-\-dev ........... Show standard deviation in output
  \-\-1qt ........... Show first quartile
  \-\-2qt|--med ..... Show 2nd quartile (median)
  \-\-3qt ........... Show 3rd quartile
  \-\-min ........... Show minimum in output
  \-\-max ........... Show maximum in output
  \-\-sum ........... Show sum in output
  \-\-cnt ........... Show count of values in output
.SH DESCRIPTION
.B datastat
is an open-source command-line tool that allows one to
compute simple statistics over columns of numeric data in text files
by aggregating rows on the basis of the values of specified key
columns.
.PP
Following the common UNIX tradition, if no filename is given, then
input is read from the standard input.  Output is provided on the
standard output.
.PP
The simplest usage of the tool is to compute the average of all
columns in a file with many rows. For example:

.B datastat myfile.dat

will produce on the standard output a single row, containing as many
columns as in the input file, and with each value being the average of
all the values in the corresponding column. If one wanted also the
standard deviation, then

.B datastat --dev myfile.dat

would provide an output file where, for each input file column, there
are two output values, one with the average and the following one with
the standard deviation of all the values in that column.

A more complex usage is when you need to aggregate data in the input
file based on some key columns. For example, the input file contains 3
columns with the first two columns being configuration options for some
experiment, and the third column being the actual output of the
experimentation. The file may have many such rows, with repeated
entries per configuration. The user would like to compute the average
values aggregated depending on the first configuration parameter:

.B datastat -k 1-2 myfile.dat

This will produce an output with multiple rows, one for each value
pairs within the first two columns of the input file, and for each row
one can find the average of each configuration on the third column.

Other statistics that can be easily computed over all the values
within a column, or all the values within each key value set, include
the standard deviation, the minimum, the maximum and the elements
count.

datastat has been purposedly kept at a minimum of functionality. Its
power resides in pipe-ing it with other common UNIX tools for
column-based numeric table processing, including grep, sed, cut,
paste, awk and sort. Ultimately, it may be extremely useful to use
datastat in combination with said tools in a GNUplot plotting script.
.SH EXAMPLES
.PP
.B cat myfile.dat | datastat -k 3-5
.PP
Use columns 3, 4 and 5 as key, and average columns 1, 2 and any other column
of the input file from the 6th onward.
.PP
.B cat myfile.dat | datastat -k 1,3,5
Use columns 1, 3 and 5 as key, and average columns 2, 4 and any other from 6 onward.
.PP
.B cat myfile.dat | datastat -k 1,3,5 --dev
.PP
In the output file: columns 1, 4 and 7 show, respectively, the contents of
the key columns 1, 3 and 5 of the input file. Output columns 2 and 3 show
the average and standard deviation of input column 2; output columns 5 and 6
show the average and standard deviation of input column 4; any further
input column from the 6th onward produces a further column pairs in the
output file, from column 8th onward, with the corresponding average and
standard deviation values.
.PP
Imagine the file myfile.dat contains the values
.PP
.nf
  Europe Italy 10
  Europe Germany 20
  Europe Italy 20
  Europe Germany 40
  US Texas 120
  US Florida 140
.fi
.PP
Now, to know the average among all countries, one can use:
.PP
.B cat myfile.dat | cut -d ' ' -f 3 | datastat
  58.3333
.PP
The use of
.B cut
is needed to suppress non-numeric columns, otherwise
.B datastat
would try to compute averages on them as well.
.PP
However, the interesting use of datastat is to know the averages within
Europe and US:
.PP
.B cat myfile.dat | cut -d ' ' -f 1,3 | datastat -k 1
.nf
  Europe 22.5
  US 130
.fi
.PP
and the averages by individual country:
.PP
.B datastat -k 1,2 myfile.dat
.nf
  Europe Germany 30
  Europe Italy 15
  US Florida 140
  US Texas 120
.fi
.SH AUTHOR
Written by Tommaso Cucinotta (firstname.lastname@gmail.com)
.SH SEE ALSO
cat(1), grep(1), sed(1), cut(1), paste(1), awk(1), sort(1), join(1)