Text processing tools (terminal)

Without input file

$ yes

Repeats a line (by default, y) infinitely.

• yes LINE | head -n 10 repeats LINE 10 times.

With input file(s)

$ cat

Concatenate the input files together in sequence.

• -s: suppress/squeeze multiple consecutive blank lines
• -n: number all lines (cf. nl)
• -b: number non-blank lines

$ zcat

Like cat, but for gzipped files.

$ tac

Like cat, in reverse: lines are printed in reverse order. (GNU but not BSD.)

• To print the last line of (contiguous) groups sharing all but the first 2 fields in common: tac FILE | uniq -f 2 | tac

$ wc

Counts lines/words/characters in the specified file(s), individually and in total. By default, displays lines, then words, then characters.

• -l: count lines
• -w: count words
• -c: count characters

Typically a single input stream

Encoding

$ file

Determines the encoding of a text file, or indicates that the argument is a directory or pipe.

$ iconv

Converts the encoding of a text file.

• iconv -f ISO-8859-1 -t UTF-8 FILE converts from ISO-8859-1 to UTF-8

Filtering/extracting by position

$ cut

Extracts fields from a file, based on delimiters or character positions.

• cut -f1 FILE retrieves the first (tab-separated) column from the file
• cut -d' ' -f1,3 FILE retrieves the first and third space-separated tokens from each line
• cut -d'
' -f20-30 FILE (with a line break) supposedly retrieves the 20th-30th lines of the file, though this doesn’t seem to work in OS X. Equivalently: head -n 30 | tail -n 20
• -s to omit lines without any delimiter

$ head, tail

Extracts a certain amount of text from the beginning or end of a file.

If multiple files are matched by the argument(s), a header indicating the filename will be displayed.

• -n N: number of lines to retrieve (default: 10)
• -c N: number of characters (bytes) to retrieve
• tail -n +N, tail -c +N: N indicates an offset relative to the beginning of the file; the rest of the file after that offset will be extracted
• head -n -N, head -c -N: offset relative to the end of the file (GNU but not BSD implementation)
• head -n 100 FILE | tail -n 1 retrieves the 100th line of the file
• tail -f FILE monitors the end of the file, writing to stdout as the file is appended to

Filtering/extracting by content

$ uniq

Filters out duplicate lines of input.

• -c: prefix each line with a count
• -i: case-insensitive
• -f N: ignore the first N (whitespace-separated) fields of each line
• -s N: ignore the first N characters of each line
• -w N: ignore all but the first N characters of each line
• Note: If some parts of the line are ignored, the kept and discarded lines may differ. The first line with a given “key” will be the one that is kept.
• other options for filtering repeated or non-repeated lines

$ grep + friends

Searches text by regular expression.

• -i: case-insensitive
• -o: only show the matched part of the line (if multiple matches on an input line, these will be on separate output lines)
• -w: match only whole words
• -l: list only files in which matches were found
• -r: recursive
• -n: include matching lines and line numbers
• -v (--invert-match): filter out matches
• -c (--count): give counts of the matches within each file instead of the matches themselves
• -E or egrep: extended regex syntax: unescaped +, (, and ) serve as operators
• -F or fgrep: literal string matching (no regexes)
• -H: suppress filename when displaying matches
• zgrep searches zip files
• bzgrep searches bz files

Augmenting

$ nl

Adds line numbers to a file. (Cf. cat -n.) Options control formatting and counting of the line numbers, including:

• -v STARTNUM: initial counter value (default: 1)
• -i INCREMENT (default: 1)
• -w WIDTH: number of characters to be occupied by line numbers (default: 6)
• -s SEP: separator to follow every line number (default: tab)
• -b t: number non-empty lines (default); -b a: number all lines; -b pREGEX: number lines matching the regular expression pattern REGEX

Reordering

sort

Sorts the input lines.

• PUNCTUATION/SPECIAL CHARS MAY BE IGNORED depending on the value of the LC_COLLATE environment variable
• -f: ignores case
• -n: “string numerical value”
• -g: general numeric sort
• -i: ignores nonprinting characters
• -r: reverse
• -u: unique
• -k: sort key (field offsets)
• -t: field delimiter

• To sort by the first column, keeping only the last record for each: tac FILE | sort -k1,1 -u

$ shuf

Randomly permutes the input lines. (GNU but not BSD)

rev

Reverses each line of the file.

Rearranging (changing spacing)

$ expand, unexpand

Convert tabs to spaces and vice versa.

$ fold

Wrap input text so no line is more than a specified number of characters wide.

• -w WIDTH: wrap lines in a file to be no more than WIDTH characters long (default: 80)
• -s: breaks lines at word spaces

column

Formats text into columns (by default, based on whitespace delimiters).

Dividing

$ split

split FILE OUTPREFIX breaks up the input into smaller files by size.

Output files are named by the specified prefix and some number of lowercase alphabetic characters (configurable with -a; defaults to 2, i.e. aa, ab, etc.). In some implementations -d can be provided to request decimal rather than alphabetic suffixes. One of the following may be provided to determine the splitting behavior:

• -l NUMLINES: number of lines in each output file (default: 1000)
• -b BYTESIZE: size of each output file; BYTESIZE can even be in kilobytes (10k) or megabytes (10m)

$ csplit

Breaks up the input into smaller files by content.

Main arguments are the file, followed by one or more patterns indicating split points. Each pattern may be a line number, a regexp (optionally with a line offset), or a number of lines followed by {REPEATS} to indicate REPEATS blocks of the specified number of lines. The line matching the pattern begins a new output file. Output files are numbered with decimal digits.

• -f OUTPREFIX (default: xx)
• -n NUMDIGITS (default: 2)

Replacing

$ tr

Translates characters, e.g. lowercasing text in a file or replacing newlines with spaces.

$ sed

Text substitution by regular expression matching.

• sed 's/K\.? ?V?\.? ?/K/g' FILE replaces all matches of the pattern in the file
• sed '/^$/d' FILE filters out blank lines of the file
• Depending on the implementation, sed may or may not not support backslash-denoted characters and character classes such as \t, \s, and [[:space:]]. (\t and \s are supported in GNU but not BSD implementations.) Tabs and newlines can be entered as literals.
• Unless -E is specified, the plus operator and parenthesized subexpressions MUST HAVE BACKSLASH ESCAPES when using sed/grep: e.g. \(.*\).\+

Two input streams

Call them FILE1 and FILE2, respectively.

$ diff

Compare two files line-by-line. (Cf. diff3 for three files.) Can also compare directories.

• -i: ignore case
• -w: ignore whitespace
• -B: ignore blank lines
• -I REGEX: ignore differences among lines that all match the regular expression pattern REGEX
• -x REGEX: exclude files that match the pattern
• -r: recursively compare subdirectories
• -s: report identical files (file comparison only)
• -u: unified diff display format
• -y: side-by-side display (cf. sdiff)
• --suppress-common-lines

Other options are useful when comparing code, e.g. -p, -F, -D, and -E.

sdiff

Compare files side-by-side, as with diff -y. Many of the diff options are available; -s is short for --suppress-common-lines.

$ comm

Displays lines common or unique to two SORTED files, organized into columns according to their commonality.

• -1 to suppress the first column (lines only in FILE1)
• -2 to suppress the second column (lines only in FILE2)
• -3 to suppress the third column (lines common to both files)
• -i for case-insensitive comparison
• if the files are sorted, join -t'
' FILE1 FILE2 is much faster than comm -1 -2 FILE1 FILE2

$ join

Merges the lines of two sorted text files based on the presence of a common field.

• -t CHAR to indicate a delimiter (by default, it is any sequence of whitespace). For instance, join -t'
' FILE1 FILE2 lists all common lines, assuming the two files are sorted.

Typically two or more input streams

$ paste

Combines/merges lines of multiple files.

• paste FILE1 FILE2 … joins corresponding lines with tabs (vertically)
• paste -s FILE1 FILE2 … joins horizontally, with corresponding lines of the input files displayed as columns
• paste -d'\n' FILE1 FILE2 … interleaves lines of the files

Three input streams

diff3

Like diff, but for three files.

Sniffing network traffic

$ tcpdump

$ tcpdump -ni en1

Ex:
     sudo tcpdump -ni en1 'dst 100.10.10.5 and tcp and port http'

$ ngrep

$ ngrep -q -W byline port 8090

Ex:
    proxy container
    ngrep -q -W byline port 8090

ngrep

$ httpry

$ httpry -v

Ex:
    httpry -i eth3 -n 3

httpry httpry

$ tcpflow

$ tcpflow 

Ex:

   tcpflow -p -c -i eth0 port 80 | grep -oE '(GET|POST|HEAD) .* HTTP/1.[01]|Host: .*'

tcpflow