parser: support 'types' feature

[nokute78]

to support #308

I added 'types' feature of parser like fluentd.
note: http://docs.fluentd.org/v0.12/articles/parser_regexp#types

We can define data types at parsers.conf.
The syntax is

types <field_name_1>:<type_name_1> <field_name_2>:<type_name_2> ...

Now these type_names are supported.

• integer
• float
• bool

Parsers.conf example

[PARSER]
    Name dummy_test
    Format regex
    Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>[^ ]+)$
    Types INT:integer FLOAT:float BOOL:bool

Configuration File

Try to parse "100 12.1 true strings"

[SERVICE]
    Parsers_File /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/parsers.conf

[INPUT]
    Name dummy
    dummy {"test":"100 12.1 true strings"}


[FILTER]
    Name parser
    Match *
    Key_Name test
    Parser dummy_test

[OUTPUT]
    Name stdout
    Match *

Output example

$ bin/fluent-bit -c dummy2.conf 
Fluent-Bit v0.12.0
Copyright (C) Treasure Data

[2017/06/29 22:48:59] [ info] [engine] started
[0] dummy.0: [1498744140.000618457, {"INT"=>100, "FLOAT"=>12.100000, "BOOL"=>true, "STRING"=>"strings"}]
[1] dummy.0: [1498744141.000545821, {"INT"=>100, "FLOAT"=>12.100000, "BOOL"=>true, "STRING"=>"strings"}]
[2] dummy.0: [1498744142.001582373, {"INT"=>100, "FLOAT"=>12.100000, "BOOL"=>true, "STRING"=>"strings"}]
[3] dummy.0: [1498744143.002502398, {"INT"=>100, "FLOAT"=>12.100000, "BOOL"=>true, "STRING"=>"strings"}]

If no "types" option, the output is like this. All field are string.

$ bin/fluent-bit -c dummy2.conf 
Fluent-Bit v0.12.0
Copyright (C) Treasure Data

[2017/06/29 22:49:50] [ info] [engine] started
[0] dummy.0: [1498744191.001413632, {"INT"=>"100", "FLOAT"=>"12.1", "BOOL"=>"true", "STRING"=>"strings"}]
[1] dummy.0: [1498744192.000833594, {"INT"=>"100", "FLOAT"=>"12.1", "BOOL"=>"true", "STRING"=>"strings"}]
[2] dummy.0: [1498744193.000141314, {"INT"=>"100", "FLOAT"=>"12.1", "BOOL"=>"true", "STRING"=>"strings"}]
[3] dummy.0: [1498744194.000462812, {"INT"=>"100", "FLOAT"=>"12.1", "BOOL"=>"true", "STRING"=>"strings"}]

[edsiper]

thanks!

[isavcic]

Where in the world is the documentation for this???

[nokute78]

@isavcic Would you see old docs ?
https://docs.fluentbit.io/manual/v/1.2/parser#configuration-parameters
It seems to be gone recent documentation update...

key	description
Types	Specify the data type of parsed field. The syntax is types <field_name_1>:<type_name_1> <field_name_2>:<type_name_2> .... The supported types are string(default), integer, bool, float, hex.