in_gelf: New input plugin for ingest GELF messages

[manuelluis]

New input plugin for ingest GELF messages from TCP and UDP.
In UDP support for gzip and zlib compressed messages, and chunked messages.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• [N/A] Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[manuelluis]

==955281==
==955281== HEAP SUMMARY:
==955281==     in use at exit: 16 bytes in 1 blocks
==955281==   total heap usage: 1,043,862 allocs, 1,043,861 frees, 10,939,968,072 bytes allocated
==955281==
==955281== 16 bytes in 1 blocks are still reachable in loss record 1 of 1
==955281==    at 0x484086F: malloc (vg_replace_malloc.c:380)
==955281==    by 0x4ED918E: strdup (strdup.c:42)
==955281==    by 0x518B1AD: ??? (in /usr/lib64/libselinux.so.1)
==955281==    by 0x518B252: ??? (in /usr/lib64/libselinux.so.1)
==955281==    by 0x401060D: call_init (dl-init.c:74)
==955281==    by 0x401060D: call_init (dl-init.c:30)
==955281==    by 0x40106FB: _dl_init (dl-init.c:121)
==955281==    by 0x40010C9: ??? (rtld.c:658)
==955281==    by 0x4: ???
==955281==    by 0x1FFF000326: ???
==955281==    by 0x1FFF000333: ???
==955281==    by 0x1FFF000336: ???
==955281==    by 0x1FFF000338: ???
==955281==
==955281== LEAK SUMMARY:
==955281==    definitely lost: 0 bytes in 0 blocks
==955281==    indirectly lost: 0 bytes in 0 blocks
==955281==      possibly lost: 0 bytes in 0 blocks
==955281==    still reachable: 16 bytes in 1 blocks
==955281==         suppressed: 0 bytes in 0 blocks
==955281==
==955281== For lists of detected and suppressed errors, rerun with: -s
==955281== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[manuelluis]

Configuration options

Option	Default	Description
Mode	udp	Defines transport protocol mode: tcp or udp.
Listen	0.0.0.0	Specify the network ip to bind for listen incomming connections.
Port	12201	Port to listen for incoming connections.
Strict_Parser	true	If is true only allow GELF messages that strict follow the protocol.
Buffer_Chunk_Size	65536	The buffer to store the incoming GELF messages. In UDP mode is the receive buffer, in TCP mode is the initial buffer size.
Buffer_Max_Size	65536	Specify the maximum buffer size to receive a TCP GELF message. If not set, the default size will be the value of Buffer_Chunk_Size.

UDP example

[INPUT]
    Name          gelf
    Mode          udp
    Listen        0.0.0.0
    Port          12201
    Strict_Parser true

[OUTPUT]
    Name   stdout
    Match  *

$ echo -n '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' | nc -w1 -u 127.0.0.1 12201`

$ ./fluent-bit -c in_gelf_udp.conf
Fluent Bit v1.9.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/10/03 23:20:17] [ info] [engine] started (pid=955677)
[2021/10/03 23:20:17] [ info] [storage] version=1.1.3, initializing...
[2021/10/03 23:20:17] [ info] [storage] in-memory
[2021/10/03 23:20:17] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/03 23:20:17] [ info] [cmetrics] version=0.2.1
[2021/10/03 23:20:17] [ info] [in_gelf] UDP buffer size set to 65536 bytes
[2021/10/03 23:20:17] [ info] [in_gelf] UDP server binding 0.0.0.0:12201
[2021/10/03 23:20:17] [ info] [sp] stream processor started
[0] gelf.0: [1633296025.426598821, {"host"=>"example.org", "short_message"=>"A short message", "level"=>5, "some_info"=>"foo"}]

TCP example

[INPUT]
    Name             gelf
    Mode             tcp
    Listen           0.0.0.0
    Port             12201
    Strict_Parser    true
    buffer_max_size  1048576

[OUTPUT]
    Name   stdout
    Match  *

$ echo -n -e '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }'"\0" | nc -w1 127.0.0.1 12201

$ ./fluent-bit -c in_gelf_tcp.conf
Fluent Bit v1.9.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/10/03 23:22:49] [ info] [engine] started (pid=955750)
[2021/10/03 23:22:49] [ info] [storage] version=1.1.3, initializing...
[2021/10/03 23:22:49] [ info] [storage] in-memory
[2021/10/03 23:22:49] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/03 23:22:49] [ info] [cmetrics] version=0.2.1
[2021/10/03 23:22:49] [ info] [in_gelf] TCP server binding 0.0.0.0:12201
[2021/10/03 23:22:49] [ info] [sp] stream processor started
[0] gelf.0: [1633296183.019387697, {"host"=>"example.org", "short_message"=>"A short message", "level"=>5, "some_info"=>"foo"}]

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[patrick-stephens]

Apologies @manuelluis but looks like the CI doesn't like your branch name so I'll resolve #4999 to sort it. Unfortunately probably means we have to wait for that to land before we can run the integration tests.

[azagarelz]

Hey @patrick-stephens, does the CI still fail because of the "/" in the branch name?
Would be awesome to have this input in fluentbit

[patrick-stephens]

It shouldn't so if you rebase let's see

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[phibos]

Would be great to have this feature. Looks like the only thing missing is a rebase against master and check if it build successfully. @manuelluis do you have the time to make a rebase or can we help with it?

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[stevesowerbutts]

Hey @manuelluis @azagarelz, this feature would be really helpful. Is there any chance someone would have time to have a look at this?

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[patrick-stephens]

Hey @manuelluis @azagarelz, this feature would be really helpful. Is there any chance someone would have time to have a look at this?

Note that anyone can pick this PR up plus we would need an active maintainer/code owner to ensure bug fixes and other changes in the future before merging.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.