Skip to content

Commit

Permalink
add description about ltsv and logfmt parsers
Browse files Browse the repository at this point in the history
ltsv and logfmt parsers are already supported, but there are no document.

- Add new parser: ltsv fluent/fluent-bit#646
- Add new parser: logfmt fluent/fluent-bit#871

I added the document about them.
  • Loading branch information
shogo82148 committed Jun 18, 2019
1 parent 607f800 commit 5f88f1d
Show file tree
Hide file tree
Showing 4 changed files with 73 additions and 1 deletion.
2 changes: 2 additions & 0 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@
* [Parsers](parser/README.md)
* [JSON Parser](parser/json.md)
* [Regular Expression Parser](parser/regular_expression.md)
* [LTSV Parser](parser/ltsv.md)
* [Logfmt Parser](parser/logfmt.md)
* [Decoders](parser/decoder.md)
* [Filter Plugins](filter/README.md)
* [Grep](filter/grep.md)
Expand Down
2 changes: 1 addition & 1 deletion parser/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Multiple parsers can be defined and each section have it own properties. The fol
| Key | Description |
| :--- | :--- |
| Name | Set an unique name for the parser in question. |
| Format | Specify the format of the parser, the available options here are: [json](json.md) or [regex](regular_expression.md). |
| Format | Specify the format of the parser, the available options here are: [json](json.md), [regex](regular_expression.md), [ltsv](ltsv.md) or [logfmt][logfmt.md]. |
| Regex | If format is _regex_, this option _must_ be set specifying the Ruby Regular Expression that will be used to parse and compose the structured message. |
| Time\_Key | If the log entry provides a field with a timestamp, this option specify the name of that field. |
| Time\_Format | Specify the format of the time field so it can be recognized and analyzed properly. Fluent-bit uses `strptime(3)` to parse time so you can ferer to [strptime documentation](https://linux.die.net/man/3/strptime) for available modifiers. |
Expand Down
25 changes: 25 additions & 0 deletions parser/logfmt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Logfmt Parser

The **logfmt** parser allows to parse the logfmt format described in https://brandur.org/logfmt .
A more formal description is in https://godoc.org/github.com/kr/logfmt .

Here is an example configuration:

```python
[PARSER]
Name logfmt
Format logfmt
```

The following log entry is a valid content for the parser defined above:

```text
key1=val1 key2=val2
```

After processing, it internal representation will be:

```text
[1540936693, {"key1"=>"val1",
"key2"=>"val2"}]
```
45 changes: 45 additions & 0 deletions parser/ltsv.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# LTSV Parser

The **ltsv** parser allows to parse [LTSV](http://ltsv.org/) formatted texts.

Labeled Tab-separated Values (LTSV format is a variant of Tab-separated Values (TSV). Each record in a LTSV file is represented as a single line. Each field is separated by TAB and has a label and a value. The label and the value have been separated by ':'.

Here is an example how to use this format in the apache access log.

Config this in httpd.conf:

```text
LogFormat "host:%h\tident:%l\tuser:%u\ttime:%t\treq:%r\tstatus:%>s\tsize:%b\treferer:%{Referer}i\tua:%{User-Agent}i" combined_ltsv
CustomLog "logs/access_log" combined_ltsv
```

The parser.conf:

```python
[PARSER]
Name access_log_ltsv
Format ltsv
Time_Key time
Time_Format [%d/%b/%Y:%H:%M:%S %z]
Types status:integer size:integer
```

The following log entry is a valid content for the parser defined above:

```text
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET / HTTP/1.1 status:200 size:16218 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/plugins/bootstrap/css/bootstrap.min.css HTTP/1.1 status:200 size:121200 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/css/headers/header-v6.css HTTP/1.1 status:200 size:37706 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
host:127.0.0.1 ident:- user:- time:[10/Jul/2018:13:27:05 +0200] req:GET /assets/css/style.css HTTP/1.1 status:200 size:1279 referer:http://127.0.0.1/ ua:Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
```

After processing, it internal representation will be:

```text
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET / HTTP/1.1", "status"=>200, "size"=>16218, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/plugins/bootstrap/css/bootstrap.min.css HTTP/1.1", "status"=>200, "size"=>121200, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/css/headers/header-v6.css HTTP/1.1", "status"=>200, "size"=>37706, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
[1531222025.000000000, {"host"=>"127.0.0.1", "ident"=>"-", "user"=>"-", "req"=>"GET /assets/css/style.css HTTP/1.1", "status"=>200, "size"=>1279, "referer"=>"http://127.0.0.1/", "ua"=>"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"}]
```

The time has been converted to Unix timestamp \(UTC\).

0 comments on commit 5f88f1d

Please sign in to comment.