Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_forward: release buf when no connection available and time_as_integer is true(#6082) #6210

Merged
merged 1 commit into from
Oct 15, 2022

Conversation

nokute78
Copy link
Collaborator

Fixes #6082

This patch is to free memory when time_as_integer is true and no connection is available.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • [N/A] Backport to latest stable release.

Configuration

fluent-bit -i dummy -o forward -p time_as_integer=on

Debug/Valgrind output

$ valgrind --leak-check=full bin/fluent-bit -i dummy -o forward -p time_as_integer=on
==43252== Memcheck, a memory error detector
==43252== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==43252== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==43252== Command: bin/fluent-bit -i dummy -o forward -p time_as_integer=on
==43252== 
[error] setting up section 'output' plugin property 'time_as_integer'
Fluent Bit v2.0.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/10/15 07:58:02] [ info] [fluent bit] version=2.0.0, commit=095d090aeb, pid=43252
[2022/10/15 07:58:02] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2022/10/15 07:58:02] [ info] [cmetrics] version=0.5.2
[2022/10/15 07:58:02] [ info] [input:dummy:dummy.0] initializing
[2022/10/15 07:58:02] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2022/10/15 07:58:02] [ info] [output:forward:forward.0] worker #0 started
[2022/10/15 07:58:02] [ info] [output:forward:forward.0] worker #1 started
[2022/10/15 07:58:02] [ info] [sp] stream processor started
==43252== Warning: client switching stacks?  SP change: 0x6ffb8b8 --> 0x549e9b0
==43252==          to suppress, use: --max-stackframe=28692232 or greater
==43252== Warning: client switching stacks?  SP change: 0x549e908 --> 0x6ffb8b8
==43252==          to suppress, use: --max-stackframe=28692400 or greater
==43252== Warning: client switching stacks?  SP change: 0x6ffb8b8 --> 0x549e908
==43252==          to suppress, use: --max-stackframe=28692400 or greater
==43252==          further instances of this message will not be shown.
[2022/10/15 07:58:04] [error] [output:forward:forward.0] no upstream connections available
[2022/10/15 07:58:04] [ warn] [engine] failed to flush chunk '43252-1665788283.181337898.flb', retry in 11 seconds: task_id=0, input=dummy.0 > output=forward.0 (out_id=0)
[2022/10/15 07:58:05] [error] [output:forward:forward.0] no upstream connections available
[2022/10/15 07:58:05] [ warn] [engine] failed to flush chunk '43252-1665788284.197366321.flb', retry in 7 seconds: task_id=1, input=dummy.0 > output=forward.0 (out_id=0)
^C[2022/10/15 07:58:05] [engine] caught signal (SIGINT)
[2022/10/15 07:58:05] [ warn] [engine] service will shutdown in max 5 seconds
[2022/10/15 07:58:05] [error] [output:forward:forward.0] no upstream connections available
[2022/10/15 07:58:05] [ info] [input] pausing dummy.0
[2022/10/15 07:58:05] [error] [output:forward:forward.0] no upstream connections available
[2022/10/15 07:58:05] [error] [output:forward:forward.0] no upstream connections available
[2022/10/15 07:58:05] [ warn] [engine] failed to flush chunk '43252-1665788285.138903169.flb', retry in 1 seconds: task_id=2, input=dummy.0 > output=forward.0 (out_id=0)
[2022/10/15 07:58:05] [ warn] [engine] chunk '43252-1665788283.181337898.flb' cannot be retried: task_id=0, input=dummy.0 > output=forward.0
[2022/10/15 07:58:05] [ warn] [engine] chunk '43252-1665788284.197366321.flb' cannot be retried: task_id=1, input=dummy.0 > output=forward.0
[2022/10/15 07:58:06] [ info] [task] dummy/dummy.0 has 1 pending task(s):
[2022/10/15 07:58:06] [ info] [task]   task_id=2 still running on route(s): forward/forward.0 
[2022/10/15 07:58:06] [error] [output:forward:forward.0] no upstream connections available
[2022/10/15 07:58:06] [ info] [input] pausing dummy.0
[2022/10/15 07:58:06] [ warn] [engine] chunk '43252-1665788285.138903169.flb' cannot be retried: task_id=2, input=dummy.0 > output=forward.0
[2022/10/15 07:58:07] [ info] [engine] service has stopped (0 pending tasks)
[2022/10/15 07:58:07] [ info] [input] pausing dummy.0
[2022/10/15 07:58:07] [ info] [output:forward:forward.0] thread worker #0 stopping...
[2022/10/15 07:58:07] [ info] [output:forward:forward.0] thread worker #0 stopped
[2022/10/15 07:58:07] [ info] [output:forward:forward.0] thread worker #1 stopping...
[2022/10/15 07:58:07] [ info] [output:forward:forward.0] thread worker #1 stopped
==43252== 
==43252== HEAP SUMMARY:
==43252==     in use at exit: 0 bytes in 0 blocks
==43252==   total heap usage: 1,713 allocs, 1,713 frees, 1,988,031 bytes allocated
==43252== 
==43252== All heap blocks were freed -- no leaks are possible
==43252== 
==43252== For lists of detected and suppressed errors, rerun with: -s
==43252== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@nokute78 nokute78 temporarily deployed to pr October 14, 2022 22:59 Inactive
@nokute78 nokute78 temporarily deployed to pr October 14, 2022 22:59 Inactive
@nokute78 nokute78 temporarily deployed to pr October 14, 2022 23:15 Inactive
@edsiper edsiper added backport to v1.9.x Used to tag items that must be backported to such version. and removed docs-required labels Oct 15, 2022
@edsiper edsiper merged commit c5fb8bf into fluent:master Oct 15, 2022
@edsiper
Copy link
Member

edsiper commented Oct 15, 2022

cc: @lecaros

@nokute78
Copy link
Collaborator Author

I sent a patch #6212 for 1.9 branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport to v1.9.x Used to tag items that must be backported to such version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Excessive memory usage although filesystem storage is enabled
2 participants