From 5f88f1d8d2fd7639237ae7b038655d6fa2ff3b62 Mon Sep 17 00:00:00 2001 From: Ichinose Shogo Date: Tue, 18 Jun 2019 17:23:24 +0900 Subject: [PATCH] add description about ltsv and logfmt parsers ltsv and logfmt parsers are already supported, but there are no document. - Add new parser: ltsv https://github.com/fluent/fluent-bit/pull/646 - Add new parser: logfmt https://github.com/fluent/fluent-bit/pull/871 I added the document about them. --- SUMMARY.md | 2 ++ parser/README.md | 2 +- parser/logfmt.md | 25 +++++++++++++++++++++++++ parser/ltsv.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 73 insertions(+), 1 deletion(-) create mode 100644 parser/logfmt.md create mode 100644 parser/ltsv.md diff --git a/SUMMARY.md b/SUMMARY.md index 70023c582..a4df846fc 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -67,6 +67,8 @@ * [Parsers](parser/README.md) * [JSON Parser](parser/json.md) * [Regular Expression Parser](parser/regular_expression.md) + * [LTSV Parser](parser/ltsv.md) + * [Logfmt Parser](parser/logfmt.md) * [Decoders](parser/decoder.md) * [Filter Plugins](filter/README.md) * [Grep](filter/grep.md) diff --git a/parser/README.md b/parser/README.md index bd78d10a5..98cf4d204 100644 --- a/parser/README.md +++ b/parser/README.md @@ -26,7 +26,7 @@ Multiple parsers can be defined and each section have it own properties. The fol | Key | Description | | :--- | :--- | | Name | Set an unique name for the parser in question. | -| Format | Specify the format of the parser, the available options here are: [json](json.md) or [regex](regular_expression.md). | +| Format | Specify the format of the parser, the available options here are: [json](json.md), [regex](regular_expression.md), [ltsv](ltsv.md) or [logfmt][logfmt.md]. | | Regex | If format is _regex_, this option _must_ be set specifying the Ruby Regular Expression that will be used to parse and compose the structured message. | | Time\_Key | If the log entry provides a field with a timestamp, this option specify the name of that field. | | Time\_Format | Specify the format of the time field so it can be recognized and analyzed properly. Fluent-bit uses `strptime(3)` to parse time so you can ferer to [strptime documentation](https://linux.die.net/man/3/strptime) for available modifiers. | diff --git a/parser/logfmt.md b/parser/logfmt.md new file mode 100644 index 000000000..13b23524b --- /dev/null +++ b/parser/logfmt.md @@ -0,0 +1,25 @@ +# Logfmt Parser + +The **logfmt** parser allows to parse the logfmt format described in https://brandur.org/logfmt . +A more formal description is in https://godoc.org/github.com/kr/logfmt . + +Here is an example configuration: + +```python +[PARSER] + Name logfmt + Format logfmt +``` + +The following log entry is a valid content for the parser defined above: + +```text +key1=val1 key2=val2 +``` + +After processing, it internal representation will be: + +```text +[1540936693, {"key1"=>"val1", + "key2"=>"val2"}] +``` diff --git a/parser/ltsv.md b/parser/ltsv.md new file mode 100644 index 000000000..9af29341f --- /dev/null +++ b/parser/ltsv.md @@ -0,0 +1,45 @@ +# LTSV Parser + +The **ltsv** parser allows to parse [LTSV](http://ltsv.org/) formatted texts. + +Labeled Tab-separated Values (LTSV format is a variant of Tab-separated Values (TSV). Each record in a LTSV file is represented as a single line. Each field is separated by TAB and has a label and a value. The label and the value have been separated by ':'. + +Here is an example how to use this format in the apache access log. + +Config this in httpd.conf: + +```text +LogFormat "host:%h\tident:%l\tuser:%u\ttime:%t\treq:%r\tstatus:%>s\tsize:%b\treferer:%{Referer}i\tua:%{User-Agent}i" combined_ltsv +CustomLog "logs/access_log" combined_ltsv +``` + +The parser.conf: + +```python +[PARSER] + Name access_log_ltsv + Format ltsv + Time_Key time + Time_Format [%d/%b/%Y:%H:%M:%S %z] + Types status:integer size:integer +``` + +The following log entry is a valid content for the parser defined above: + +```text +host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET / HTTP/1.1 status:200 size:16218 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0 +host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/plugins/bootstrap/css/bootstrap.min.css HTTP/1.1 status:200 size:121200 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0 +host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/css/headers/header-v6.css HTTP/1.1 status:200 size:37706 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0 +host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/css/style.css HTTP/1.1 status:200 size:1279 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0 +``` + +After processing, it internal representation will be: + +```text +[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET / HTTP/1.1", "status"=>200, "size"=>16218, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}] +[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/plugins/bootstrap/css/bootstrap.min.css HTTP/1.1", "status"=>200, "size"=>121200, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}] +[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/css/headers/header-v6.css HTTP/1.1", "status"=>200, "size"=>37706, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}] +[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/css/style.css HTTP/1.1", "status"=>200, "size"=>1279, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}] +``` + +The time has been converted to Unix timestamp \(UTC\).