out_logdna: refactored LogDNA URI formation to support configurable endpoints #8051

mirko-lazarevic · 2023-10-17T09:00:36Z

Previously, the LogDNA URI was hardcoded to use the /logs/ingest endpoint. This change introduces flexibility by allowing users to specify their own endpoint through the logdna_endpoint configuration. This can be particularly useful for users with different LogDNA setups or those using Super Tenancy.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#1236

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

…ndpoints Previously, the LogDNA URI was hardcoded to use the '/logs/ingest' endpoint. This change introduces flexibility by allowing users to specify their own endpoint through the 'logdna_endpoint' configuration. This can be particularly useful for users with different LogDNA setups or those using Super Tenancy. Signed-off-by: Mirko Lazarevic <[email protected]>

mirko-lazarevic · 2023-10-17T09:15:11Z

Testing

Example configuration file for the change

[SERVICE]
    Flush           5
    Daemon          Off
    Log_Level       debug
    Parsers_File    parsers.conf

[INPUT]
    Name    dummy
    Dummy   {"message":"This is test log statement"}

[OUTPUT]
    name            logdna
    match           *
    api_key        <your_api_key>
    logdna_host     logs.logdna.com
    logdna_port     443
    logdna_endpoint /logs/ingest
    tags            fluentbit

Debug log output from testing the change

# ./bin/fluent-bit -c ../conf/out_logdna.conf
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/10/17 09:10:26] [ info] Configuration:
[2023/10/17 09:10:26] [ info]  flush time     | 5.000000 seconds
[2023/10/17 09:10:26] [ info]  grace          | 5 seconds
[2023/10/17 09:10:26] [ info]  daemon         | 0
[2023/10/17 09:10:26] [ info] ___________
[2023/10/17 09:10:26] [ info]  inputs:
[2023/10/17 09:10:26] [ info]      dummy
[2023/10/17 09:10:26] [ info] ___________
[2023/10/17 09:10:26] [ info]  filters:
[2023/10/17 09:10:26] [ info] ___________
[2023/10/17 09:10:26] [ info]  outputs:
[2023/10/17 09:10:26] [ info]      logdna.0
[2023/10/17 09:10:26] [ info] ___________
[2023/10/17 09:10:26] [ info]  collectors:
[2023/10/17 09:10:26] [ info] [fluent bit] version=2.2.0, commit=39e52bf041, pid=21381
[2023/10/17 09:10:26] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/10/17 09:10:26] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/10/17 09:10:26] [ info] [cmetrics] version=0.6.3
[2023/10/17 09:10:26] [ info] [ctraces ] version=0.3.1
[2023/10/17 09:10:26] [ info] [input:dummy:dummy.0] initializing
[2023/10/17 09:10:26] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/10/17 09:10:26] [debug] [dummy:dummy.0] created event channels: read=21 write=22
[2023/10/17 09:10:26] [debug] [logdna:logdna.0] created event channels: read=23 write=24
[2023/10/17 09:10:26] [ info] [output:logdna:logdna.0] configured, hostname=(null)
[2023/10/17 09:10:26] [ info] [sp] stream processor started
[2023/10/17 09:10:27] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:28] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:29] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:30] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:31] [debug] [task] created task=0x7f383c0d12c0 id=0 OK
[2023/10/17 09:10:31] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:31] [debug] [upstream] KA connection #31 to logs.logdna.com:443 is connected
[2023/10/17 09:10:31] [debug] [http_client] not using http_proxy for header
[2023/10/17 09:10:32] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"0c6bc3cc-0909-4377-89a8-e9471fd89101:5887:ld82","status":"ok"}
[2023/10/17 09:10:32] [debug] [upstream] KA connection #31 to logs.logdna.com:443 is now available
[2023/10/17 09:10:32] [debug] [out flush] cb_destroy coro_id=0
[2023/10/17 09:10:32] [debug] [task] destroy task=0x7f383c0d12c0 (task_id=0)
[2023/10/17 09:10:32] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:33] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:34] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:35] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:36] [debug] [task] created task=0x7f383c0d1620 id=0 OK
[2023/10/17 09:10:36] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:36] [debug] [upstream] KA connection #31 to logs.logdna.com:443 has been assigned (recycled)
[2023/10/17 09:10:36] [debug] [http_client] not using http_proxy for header
[2023/10/17 09:10:36] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"d641c240-c09f-46f7-b9a1-9eb657928df2:21022:ld82","status":"ok"}
[2023/10/17 09:10:36] [debug] [upstream] KA connection #31 to logs.logdna.com:443 is now available
[2023/10/17 09:10:36] [debug] [out flush] cb_destroy coro_id=1
[2023/10/17 09:10:36] [debug] [task] destroy task=0x7f383c0d1620 (task_id=0)
[2023/10/17 09:10:37] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:38] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:39] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:40] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:41] [debug] [task] created task=0x7f383c0d1510 id=0 OK
[2023/10/17 09:10:41] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
[2023/10/17 09:10:41] [debug] [upstream] KA connection #31 to logs.logdna.com:443 has been assigned (recycled)
[2023/10/17 09:10:41] [debug] [http_client] not using http_proxy for header
[2023/10/17 09:10:41] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"6055713e-56f8-424b-8b5e-02a333b5ba73:52345:ld82","status":"ok"}
[2023/10/17 09:10:41] [debug] [upstream] KA connection #31 to logs.logdna.com:443 is now available
[2023/10/17 09:10:41] [debug] [out flush] cb_destroy coro_id=2
[2023/10/17 09:10:41] [debug] [task] destroy task=0x7f383c0d1510 (task_id=0)
[2023/10/17 09:10:42] [debug] [input chunk] update output instances with new chunk size diff=265, records=1, input=dummy.0
^C[2023/10/17 09:10:42] [engine] caught signal (SIGINT)
[2023/10/17 09:10:42] [debug] [task] created task=0x7f383c0d1710 id=0 OK
[2023/10/17 09:10:42] [ warn] [engine] service will shutdown in max 5 seconds
[2023/10/17 09:10:42] [ info] [input] pausing dummy.0
[2023/10/17 09:10:42] [debug] [upstream] KA connection #31 to logs.logdna.com:443 has been assigned (recycled)
[2023/10/17 09:10:42] [debug] [http_client] not using http_proxy for header
[2023/10/17 09:10:42] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"39080b63-e383-465f-9f0d-c6ecb60b56e5:12819:ld82","status":"ok"}
[2023/10/17 09:10:42] [debug] [upstream] KA connection #31 to logs.logdna.com:443 is now available
[2023/10/17 09:10:42] [debug] [out flush] cb_destroy coro_id=3
[2023/10/17 09:10:42] [debug] [task] destroy task=0x7f383c0d1710 (task_id=0)
[2023/10/17 09:10:43] [ info] [engine] service has stopped (0 pending tasks)
[2023/10/17 09:10:43] [ info] [input] pausing dummy.0

Valgrind output

# valgrind ./bin/fluent-bit -c ../conf/out_logdna.conf
==21376== Memcheck, a memory error detector
==21376== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==21376== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==21376== Command: ./bin/fluent-bit -c ../conf/out_logdna.conf
==21376==
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/10/17 08:57:29] [ info] [fluent bit] version=2.2.0, commit=39e52bf041, pid=21376
[2023/10/17 08:57:29] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/10/17 08:57:29] [ info] [cmetrics] version=0.6.3
[2023/10/17 08:57:29] [ info] [ctraces ] version=0.3.1
[2023/10/17 08:57:29] [ info] [input:dummy:dummy.0] initializing
[2023/10/17 08:57:29] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/10/17 08:57:29] [ info] [output:logdna:logdna.0] configured, hostname=(null)
[2023/10/17 08:57:29] [ info] [sp] stream processor started




[2023/10/17 08:57:36] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"3340f2f7-decf-42a0-9716-e0b87f339267:37724:ld82","status":"ok"}
[2023/10/17 08:57:39] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"cc2d3f1b-2400-4b0a-9690-66f1f23019a0:28387:ld82","status":"ok"}
[2023/10/17 08:57:44] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"cf020579-f952-4454-a139-372fca5a0883:23594:ld82","status":"ok"}
[2023/10/17 08:57:49] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"e2a38645-b04a-42aa-99ca-c3f392594f75:26025:ld82","status":"ok"}
[2023/10/17 08:57:54] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"d0bee337-6d03-4361-8746-15975e02ce74:32680:ld82","status":"ok"}
[2023/10/17 08:57:59] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"f05cf839-fb31-4fd9-aaa0-df79e03a739b:41117:ld82","status":"ok"}
[2023/10/17 08:58:04] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"cf5c456f-8ff8-4aa7-a594-3cd22b4f29da:38296:ld82","status":"ok"}
^C[2023/10/17 08:58:09] [engine] caught signal (SIGINT)
[2023/10/17 08:58:09] [ warn] [engine] service will shutdown in max 5 seconds
[2023/10/17 08:58:09] [ info] [input] pausing dummy.0
[2023/10/17 08:58:09] [ info] [task] dummy/dummy.0 has 1 pending task(s):
[2023/10/17 08:58:09] [ info] [task]   task_id=0 still running on route(s): logdna/logdna.0
[2023/10/17 08:58:09] [ info] [input] pausing dummy.0
[2023/10/17 08:58:09] [ info] [output:logdna:logdna.0] logs.logdna.com:443, HTTP status=200
{"batchID":"6e6a1364-fc96-44b2-af8d-4dc686ba5348:30685:ld82","status":"ok"}
[2023/10/17 08:58:10] [ info] [engine] service has stopped (0 pending tasks)
[2023/10/17 08:58:10] [ info] [input] pausing dummy.0
==21376==
==21376== HEAP SUMMARY:
==21376==     in use at exit: 0 bytes in 0 blocks
==21376==   total heap usage: 20,270 allocs, 20,270 frees, 6,736,563 bytes allocated
==21376==
==21376== All heap blocks were freed -- no leaks are possible
==21376==
==21376== For lists of detected and suppressed errors, rerun with: -s
==21376== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

ScarletTanager · 2023-10-17T12:00:11Z

This is particularly important for us, as we need the endpoint flexibility.

mirko-lazarevic · 2023-10-25T06:51:13Z

I noticed that not all checks passed for the pull request. Is there anything I need to address or adjust on my end? Please let me know if I can assist in moving this forward.

mirko-lazarevic · 2023-11-06T11:30:16Z

A friendly heads-up to code owners @edsiper @leonardo-albertovich @fujimotos and @koleini to check out the pull request when they can. Many thanks.

agup006 · 2023-11-16T17:08:24Z

@cosmo0920 would you be able to check this again? Specifically in the unit tests

cosmo0920 · 2023-11-17T08:45:32Z

The failed unit tests are core_chunk_trace and flb-it-log:

2 - flb-rt-core_chunk_trace (Subprocess aborted)
105 - flb-it-log (Failed)

ref: https://github.com/fluent/fluent-bit/actions/runs/6544903788/job/18323010415?pr=8051#step:4:4018
ref: https://github.com/fluent/fluent-bit/actions/runs/6544903788/job/18323010281?pr=8051#step:4:3810

They are not related to this PR and they are flaky tests...

ScarletTanager · 2023-11-18T15:40:29Z

@cosmo0920 Thanks for looking - but how can we move this forward when unrelated flaky tests fail?

ScarletTanager · 2023-11-18T15:42:22Z

@agup006 According to GH, we need a review from one of @edsiper @fujimotos @koleini @leonardo-albertovich .

mirko-lazarevic · 2024-01-09T12:51:21Z

🏓

mirko-lazarevic · 2024-02-05T12:22:16Z

🏓

mirko-lazarevic · 2024-04-19T13:14:38Z

🏓

tchrono

LGTM

mirko-lazarevic · 2024-07-02T14:47:11Z

🏓

agup006 · 2024-07-02T14:48:24Z

@edsiper could we get this for 3.1.1?

mirko-lazarevic · 2024-07-23T20:20:24Z

🏓

…ndpoints (fluent#8051) Previously, the LogDNA URI was hardcoded to use the '/logs/ingest' endpoint. This change introduces flexibility by allowing users to specify their own endpoint through the 'logdna_endpoint' configuration. This can be particularly useful for users with different LogDNA setups or those using Super Tenancy. Signed-off-by: Mirko Lazarevic <[email protected]>

github-actions bot added the docs-required label Oct 17, 2023

mirko-lazarevic marked this pull request as ready for review October 17, 2023 09:14

mirko-lazarevic requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners October 17, 2023 09:14

mirko-lazarevic temporarily deployed to pr October 18, 2023 07:56 — with GitHub Actions Inactive

mirko-lazarevic temporarily deployed to pr October 18, 2023 08:27 — with GitHub Actions Inactive

cosmo0920 approved these changes Oct 18, 2023

View reviewed changes

tchrono approved these changes Apr 19, 2024

View reviewed changes

lecaros modified the milestones: Fluent Bit v3.0.3, Fluent Bit v3.0.4 Apr 24, 2024

lecaros modified the milestones: Fluent Bit v3.0.4, Fluent Bit v3.0.6 May 23, 2024

cosmo0920 approved these changes Jul 3, 2024

View reviewed changes

cosmo0920 modified the milestones: Fluent Bit v3.0.6, Fluent Bit v3.1.7 Aug 22, 2024

edsiper merged commit 9e5111b into fluent:master Aug 27, 2024
47 of 49 checks passed

mirko-lazarevic mentioned this pull request Aug 28, 2024

pipeline: output: logdna: add documentation for logdna_endpoint config param fluent/fluent-bit-docs#1236

Merged

mirko-lazarevic deleted the out-logdna-configurable-endpoints branch August 29, 2024 18:17

BrewTestBot mentioned this pull request Sep 2, 2024

fluent-bit 3.1.7 Homebrew/homebrew-core#183215

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_logdna: refactored LogDNA URI formation to support configurable endpoints #8051

out_logdna: refactored LogDNA URI formation to support configurable endpoints #8051

mirko-lazarevic commented Oct 17, 2023 •

edited

Loading

mirko-lazarevic commented Oct 17, 2023 •

edited

Loading

ScarletTanager commented Oct 17, 2023

mirko-lazarevic commented Oct 25, 2023

mirko-lazarevic commented Nov 6, 2023

agup006 commented Nov 16, 2023 •

edited

Loading

cosmo0920 commented Nov 17, 2023 •

edited

Loading

ScarletTanager commented Nov 18, 2023

ScarletTanager commented Nov 18, 2023

mirko-lazarevic commented Jan 9, 2024

mirko-lazarevic commented Feb 5, 2024

mirko-lazarevic commented Apr 19, 2024

tchrono left a comment

mirko-lazarevic commented Jul 2, 2024

agup006 commented Jul 2, 2024

mirko-lazarevic commented Jul 23, 2024

out_logdna: refactored LogDNA URI formation to support configurable endpoints #8051

out_logdna: refactored LogDNA URI formation to support configurable endpoints #8051

Conversation

mirko-lazarevic commented Oct 17, 2023 • edited Loading

mirko-lazarevic commented Oct 17, 2023 • edited Loading

Testing

Example configuration file for the change

Debug log output from testing the change

Valgrind output

ScarletTanager commented Oct 17, 2023

mirko-lazarevic commented Oct 25, 2023

mirko-lazarevic commented Nov 6, 2023

agup006 commented Nov 16, 2023 • edited Loading

cosmo0920 commented Nov 17, 2023 • edited Loading

ScarletTanager commented Nov 18, 2023

ScarletTanager commented Nov 18, 2023

mirko-lazarevic commented Jan 9, 2024

mirko-lazarevic commented Feb 5, 2024

mirko-lazarevic commented Apr 19, 2024

tchrono left a comment

Choose a reason for hiding this comment

mirko-lazarevic commented Jul 2, 2024

agup006 commented Jul 2, 2024

mirko-lazarevic commented Jul 23, 2024

mirko-lazarevic commented Oct 17, 2023 •

edited

Loading

mirko-lazarevic commented Oct 17, 2023 •

edited

Loading

agup006 commented Nov 16, 2023 •

edited

Loading

cosmo0920 commented Nov 17, 2023 •

edited

Loading