emit log entries when more than 0 after exclusion #618
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the namespace excluded in both stdout & stderr streams, then no issue since we don’t tail the logs of the container of that namespace completely.
But, If the namespace excluded either to stdout or stderr, then we drop messages of excluded stream after receiving log lines.
This is depends on the timing but If the batch, we received from the fluent bit tail plugin contains the all log lines of excluded stream then instead of dropping them, because of the bug in ciprod06112021, agent try to ingest incorrect path which will fail with 403 (which is expected) but those log lines keep retried.
This will not issue of the number of dropped messages but if the dropped messages are very high and that will cause, the number of log lines are retrying are very high, then that will impact the memory usage of td-agent-bit (aka Fluent bit) and eventually agent will OOM if the memory usage keep grows because of the retries .
In this customer case - Azure/AKS#2457, the namespace excluded “baapi2-test” excluded for stdout but not stderr, the containers in this namespace generating very high number of stdout logs which are keep retrying incorrect path instead of dropping those log lines.