Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

input_chunk: Update tag value extraction for backoff logs #2806

Merged
merged 1 commit into from
Nov 27, 2020
Merged

input_chunk: Update tag value extraction for backoff logs #2806

merged 1 commit into from
Nov 27, 2020

Conversation

JeffLuoo
Copy link
Contributor

@JeffLuoo JeffLuoo commented Nov 25, 2020 •
edited
Loading

Signed-off-by: Jeff Luo [email protected]

Backoff logs are pulled by a input plugin that users are not able to config. And also the plugin doesn't have the tag value that can be used to route the input chunks to their destination. I found that there is a function flb_input_chunk_get_tag that can be used to extract the tag value of the input chunk from metadata.

#2765

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

  • Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@JeffLuoo
Copy link
Contributor Author

JeffLuoo commented Nov 25, 2020 •
edited
Loading

Config used to generate backoff logs:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    debug
    storage.path               /var/log/fluent-bit-buffers/
    storage.sync               full
    storage.checksum           off
    storage.backlog.mem_limit  10M

[INPUT]
    Name          random
    Tag           k8s_container.1.2.3
    Samples      -1
    Interval_Sec  1
    Interval_NSec 0
    storage.type  filesystem

[OUTPUT]
    Name  stackdriver
    Match k8s_container.*
    Resource    k8s_containerr
    k8s_cluster_name test_cluster_name
    k8s_cluster_location  test_cluster_location
    retry_limit    false
    storage.total_limit_size 10000

I used k8s_containerr (an extra r) to generate the backoff logs.

@JeffLuoo
Copy link
Contributor Author

Config used to test routing:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    debug
    storage.path               /var/log/fluent-bit-buffers/
    storage.sync               full
    storage.checksum           off
    storage.backlog.mem_limit  10M

[INPUT]
    Name          random
    Tag           k8s_container.1.2.3
    Samples      -1
    Interval_Sec  60
    Interval_NSec 0
    storage.type  filesystem

[OUTPUT]
    Name  stdout
    Match k8s_container.*
    retry_limit    false
    storage.total_limit_size 10000

Valgrind:

Fluent Bit v1.7.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2020/11/25 18:23:04] [ info] Configuration:
[2020/11/25 18:23:04] [ info]  flush time     | 1.000000 seconds
[2020/11/25 18:23:04] [ info]  grace          | 5 seconds
[2020/11/25 18:23:04] [ info]  daemon         | 0
[2020/11/25 18:23:04] [ info] ___________
[2020/11/25 18:23:04] [ info]  inputs:
[2020/11/25 18:23:04] [ info]      random
[2020/11/25 18:23:04] [ info] ___________
[2020/11/25 18:23:04] [ info]  filters:
[2020/11/25 18:23:04] [ info] ___________
[2020/11/25 18:23:04] [ info]  outputs:
[2020/11/25 18:23:04] [ info]      stdout.0
[2020/11/25 18:23:04] [ info] ___________
[2020/11/25 18:23:04] [ info]  collectors:
[2020/11/25 18:23:04] [ info] [engine] started (pid=19828)
[2020/11/25 18:23:04] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2020/11/25 18:23:04] [debug] [storage] [cio scan] opening path /var/log/fluent-bit-buffers/
[2020/11/25 18:23:04] [debug] [storage] [cio stream] new stream registered: random.0
[2020/11/25 18:23:04] [debug] [storage] [cio scan] opening stream random.0
[2020/11/25 18:23:04] [debug] [storage] random.0:19806-1606328574.370999908.flb mapped OK
[2020/11/25 18:23:04] [debug] [storage] random.0:19806-1606328573.157124144.flb mapped OK
[2020/11/25 18:23:04] [debug] [storage] random.0:19806-1606328572.77290258.flb mapped OK
[2020/11/25 18:23:04] [debug] [storage] [cio stream] new stream registered: random.0
[2020/11/25 18:23:04] [debug] [storage] [cio stream] new stream registered: storage_backlog.1
[2020/11/25 18:23:04] [ info] [storage] version=1.0.6, initializing...
[2020/11/25 18:23:04] [ info] [storage] root path '/var/log/fluent-bit-buffers/'
[2020/11/25 18:23:04] [ info] [storage] full synchronization mode, checksum disabled, max_chunks_up=128
[2020/11/25 18:23:04] [ info] [storage] backlog input plugin: storage_backlog.1
[2020/11/25 18:23:04] [debug] [input:random:random.0] interval_sec=60 interval_nsec=0
[2020/11/25 18:23:04] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 9.5M
[2020/11/25 18:23:04] [ info] [input:storage_backlog:storage_backlog.1] register random.0/19806-1606328572.77290258.flb
[2020/11/25 18:23:04] [ info] [input:storage_backlog:storage_backlog.1] register random.0/19806-1606328573.157124144.flb
[2020/11/25 18:23:04] [ info] [input:storage_backlog:storage_backlog.1] register random.0/19806-1606328574.370999908.flb
[2020/11/25 18:23:04] [debug] [stdout:stdout.0] created event channels: read=20 write=21
[2020/11/25 18:23:04] [debug] [router] match rule random.0:stdout.0
[2020/11/25 18:23:04] [ info] [sp] stream processor started
[2020/11/25 18:23:05] [debug] [storage] random.0:19806-1606328572.77290258.flb mapped OK
[2020/11/25 18:23:05] [ info] [input:storage_backlog:storage_backlog.1] queueing random.0:19806-1606328572.77290258.flb
[2020/11/25 18:23:05] [debug] Before loop [router] tag: k8s_container.1.2.3��, 19
[2020/11/25 18:23:05] [debug] In the loop [router] matching rule for output stdout.0:1, tag:k8s_container.1.2.3��, tag_len:19
[2020/11/25 18:23:05] [debug] [input chunk] update plugin stdout.0 fs_chunk_size by 32 bytes, the current fs_chunk_size is 32 bytes
[2020/11/25 18:23:05] [debug] [storage] random.0:19806-1606328573.157124144.flb mapped OK
[2020/11/25 18:23:05] [ info] [input:storage_backlog:storage_backlog.1] queueing random.0:19806-1606328573.157124144.flb
[2020/11/25 18:23:05] [debug] Before loop [router] tag: k8s_container.1.2.3��, 19
[2020/11/25 18:23:05] [debug] In the loop [router] matching rule for output stdout.0:1, tag:k8s_container.1.2.3��, tag_len:19
[2020/11/25 18:23:05] [debug] [input chunk] update plugin stdout.0 fs_chunk_size by 32 bytes, the current fs_chunk_size is 64 bytes
[2020/11/25 18:23:05] [debug] [storage] random.0:19806-1606328574.370999908.flb mapped OK
[2020/11/25 18:23:05] [ info] [input:storage_backlog:storage_backlog.1] queueing random.0:19806-1606328574.370999908.flb
[2020/11/25 18:23:05] [debug] Before loop [router] tag: k8s_container.1.2.3��, 19
[2020/11/25 18:23:05] [debug] In the loop [router] matching rule for output stdout.0:1, tag:k8s_container.1.2.3��, tag_len:19
[2020/11/25 18:23:05] [debug] [input chunk] update plugin stdout.0 fs_chunk_size by 32 bytes, the current fs_chunk_size is 96 bytes
[2020/11/25 18:23:06] [debug] [task] created task=0x4c0e780 id=0 OK
==19828== Warning: client switching stacks?  SP change: 0x57da978 --> 0x4c14f00
==19828==          to suppress, use: --max-stackframe=12343928 or greater
==19828== Warning: client switching stacks?  SP change: 0x4c14e78 --> 0x57da978
==19828==          to suppress, use: --max-stackframe=12344064 or greater
==19828== Warning: client switching stacks?  SP change: 0x57da978 --> 0x4c14e78
==19828==          to suppress, use: --max-stackframe=12344064 or greater
==19828==          further instances of this message will not be shown.
[0] k8s_container.1.2.3: [1606328572.072804549, {"rand_value"=>17061250026118445780}]
[0] k8s_container.1.2.3: [1606328573.156927725, {"rand_value"=>11574707884188692272}]
[0] k8s_container.1.2.3: [1606328574.370976987, {"rand_value"=>4505357670832906560}]
[2020/11/25 18:23:06] [debug] [task] created task=0x4c0e920 id=1 OK
[2020/11/25 18:23:06] [debug] [task] created task=0x4c0eac0 id=2 OK
[2020/11/25 18:23:06] [ info] [engine] flush backlog chunk '19806-1606328572.77290258.flb' succeeded: task_id=0, input=storage_backlog.1 > output=stdout.0
[2020/11/25 18:23:06] [debug] [task] destroy task=0x4c0e780 (task_id=0)
[2020/11/25 18:23:06] [ info] [engine] flush backlog chunk '19806-1606328573.157124144.flb' succeeded: task_id=0, input=storage_backlog.1 > output=stdout.0
[2020/11/25 18:23:06] [debug] [task] destroy task=0x4c0e920 (task_id=1)
[2020/11/25 18:23:06] [ info] [engine] flush backlog chunk '19806-1606328574.370999908.flb' succeeded: task_id=0, input=storage_backlog.1 > output=stdout.0
[2020/11/25 18:23:06] [debug] [task] destroy task=0x4c0eac0 (task_id=2)
^C[2020/11/25 18:23:09] [engine] caught signal (SIGINT)
[2020/11/25 18:23:09] [ info] [input] pausing storage_backlog.1
[2020/11/25 18:23:09] [ warn] [engine] service will stop in 5 seconds
[2020/11/25 18:23:13] [ info] [engine] service stopped
==19828== 
==19828== HEAP SUMMARY:
==19828==     in use at exit: 0 bytes in 0 blocks
==19828==   total heap usage: 395 allocs, 395 frees, 519,034 bytes allocated
==19828== 
==19828== All heap blocks were freed -- no leaks are possible
==19828== 
==19828== For lists of detected and suppressed errors, rerun with: -s
==19828== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

@JeffLuoo JeffLuoo mentioned this pull request Nov 25, 2020
Closed
fujimotos
fujimotos approved these changes Nov 27, 2020
View reviewed changes
src/flb_input_chunk.c Show resolved Hide resolved
@fujimotos
Copy link
Member

Aside from a minor point above, I basically think this is a good patch to merge

Note: I confirmed this patch on Windows 2019 (combined with #2810)

Before:

% fluent-bit.exe -c 2806.conf
...
[2020/11/27 03:55:40] [ info] [storage] backlog input plugin: storage_backlog.1
[2020/11/27 03:55:40] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 4.8M
[2020/11/27 03:55:40] [ info] [input:storage_backlog:storage_backlog.1] register tail.0/2904-1606447265.894561100.flb
[2020/11/27 03:55:40] [ info] [sp] stream processor started
[2020/11/27 03:55:41] [ info] [input:storage_backlog:storage_backlog.1] queueing tail.0:2904-1606447265.894561100.flb
[2020/11/27 03:55:41] [ warn] [input chunk] no matching route for backoff log chunk 2904-1606447265.894561100.flb

After:

% fluent-bit.exe -c 2806.conf
...
[2020/11/27 03:56:59] [ info] [storage] backlog input plugin: storage_backlog.1
[2020/11/27 03:56:59] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 4.8M
[2020/11/27 03:56:59] [ info] [input:storage_backlog:storage_backlog.1] register tail.0/2904-1606447265.894561100.flb
[2020/11/27 03:56:59] [ info] [sp] stream processor started
[2020/11/27 03:57:00] [ info] [input:storage_backlog:storage_backlog.1] queueing tail.0:2904-1606447265.894561100.flb
[0] win32.fs: [1606447265.894559200, {"log"=>"Thu Nov 26 11:01:51 JST 2020 Windows File Storage support test."}]
[2020/11/27 03:57:01] [ info] [engine] flush backlog chunk '2904-1606447265.894561100.flb' succeeded: task_id=0, input=storage_backlog.1 > output=stdout.0

@edsiper
Copy link
Member

edsiper commented Nov 27, 2020

thanks @fujimotos for the review.

@JeffLuoo JeffLuoo requested a review from fujimotos November 27, 2020 16:38
@JeffLuoo
Copy link
Contributor Author

@fujimotos Thanks for the review!

@edsiper edsiper merged commit 5041e9a into fluent:master Nov 27, 2020
@edsiper
Copy link
Member

edsiper commented Nov 27, 2020

thanks!

edsiper pushed a commit that referenced this pull request Dec 2, 2020
@JeffLuoo @edsiper
input_chunk: Update tag value extraction for backoff logs (#2806)
c822500 
Signed-off-by: Jeff Luo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JeffLuoo @fujimotos @edsiper