pino-clf    

Transform Pino HTTP logs into Common Log Format.

This CLI tool and module is a transform transport for the pino which outputs Common Log Format.

Supports

• express-pino-logger
• restify-pino-logger
• koa-pino-logger
• pino-http
• hapi-pino

Usage

$ npm install -g pino-clf

$ pino-clf -h

  
    pino-clf [-d] [-k] [-a] type

    type                common | combined (default)

    -d | --dest |       stderr | stdout (default) or Number or Array of Numbers
    --destination       Specify file descriptor(s) to send log(s) to [access, referral, agent]
    
    -a | --ancillary    stderr | stdout or Number. Specify JSON logs fd
    
    -k | --keep         true | false (default) Retain transformed logs in ancillary output

Common Log Format

The Common Log Format is a frequently used log format, that has a very mature and well established ecosystem built around it.

It takes two forms: "Combined" and "Common".

When the type is "Common" this can break down into (up to) three separate log streams:

• Access log (also known as Common log)
• Referall log
• Agent log

Access log example:

127.0.0.1 - Aladdin [21/Jul/2016:17:34:52 -0060] "GET /api/activity/component HTTP/1.1" 200 -

Referral log example:

[21/Jul/2016:17:34:52 -0060] "http://localhost:20000/"

Agent log example:

[21/Jul/2016:17:34:52 -0060] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"

The "Combined" format combines all three formats into one, for instance:

127.0.0.1 - Aladdin [21/Jul/2016:17:34:52 -0060] "GET /api/activity/component HTTP/1.1" 200 - "http://localhost:20000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"

Example

Spin up a server that uses a pino http logger (see the Supports section), pipe it to pino-clf and desribe the format in tokenized form

$ node server | pino-clf
127.0.0.1 - Aladdin [21/Jul/2016:17:34:52 -0060] "GET /api/activity/component HTTP/1.1" 200 - "http://localhost:20000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"

The default output is Combined. To use three log output use the -d flag and specify up to three file descriptors (first is access log, second referral log, third agent log):

node server | pino-clf -d [3,4,5] 3>> ./access.log 4>> ./referral.log 5>> ./agent.log

The above will append the Access, Agent, and Referral logs to each corresponding log file.

We can skip the middle log (referall) by using an file desriptor of 0 or null as the middle element:

node server | pino-clf -d [3,0,4] 3>> ./access.log 4>> ./agent.log

Destination (-d)

By default, logs are output to STDOUT, however we can set the -d (alias, --dest, --destination), flag to a a stderr, or a number (1 for stdout, 2 for stderr, 3 or more for custom file descriptor):

$ node server | pino-clf -d stderr

The above is equivalent to:

$ node server | pino-clf -d 2 combined

We can also direct formatted log output to custom file descriptors, but we must use bash redirection (in some form) from that file descriptor, otherwise the process will most likely immediately crash (this is to do with how unix works).

$ node server | pino-clf -d 8 8> ./combined.log

Ancillary Output (-a)

By default, any logs which aren't an HTTP log (meaning, they don't have req and res properties and the msg isn't "request complete") are filtered out.

However, we can specify an ancillary (secondary) output for other log messages, using the -a (alias --ancillary) flag.

The following will write reformatted HTTP logs to STDOUT and original JSON logs which are not HTTP logs to STDERR.

$ node server | pino-clf -a 2 -d 1

Keep Original HTTP JSON Logs (-k)

The -a (--ancillary) flag can be coupled with the -k (--keep) flag so that raw HTTP JSON logs are also piped to the ancillary output stream, along with any filtered output.

The following will pipe all formatted logs to the 4 file descriptor which is redirected to a file, while all original JSON logs (instead of non-HTTP logs) are written to STDOUT.

$ node server | pino-clf -k -a 1 -d 4 4> ./combined.logs

Programmatic API

clf(type?, destination?, ancillary?)

Returns a stream that we write Pino JSON logs to.

type (String | Object)

• combined (default)
• common

If object form type is {type, keep}

• keep (Boolean, false) - preserve original HTTP Logs, as well as other JSON logs when writing to ancillary stream

destination (Stream | Array)

Stream or streams to write output to.

If the type is common destination can be an array of up to three streams, to enable the three log output format.

• destination0 - access log
• destination1 - referral log
• destination2 - agent log

ancillary (Stream)

Stream to output original JSON logs to.

LICENSE

MIT

Acknowledgements

• Inspired by the morgan logger
• Sponsored by nearForm