Move k8s metadata enrichment from fluentd to otel-collector #192
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change is the first step towards moving away from fluentd logs collection. Fluentd "kubernetes_metadata_filter" plugin is known as a performance bottleneck in k8s logs collection with fluentd. So moving away from it already gives performance benefits. Also it mitigates issues when fluentd ends up hammering k8s API. This change is compatible with otel gateway, if `otelCollector.enabled=true` all logs are forwarded through the gateway and logs enrichment is happening there reducing load on k8s API. Drawbacks: - `container.image.name` logs attribute cannot be provided at the moment until this issue is resolved: open-telemetry/opentelemetry-collector-contrib#5235 Additional changes: - Extra attribute added by default `k8s.pod.labels.app`. Value for this attribute is taken from pod's "app" label if it's set. The change is needed to support istio use case. We also want to enable more of these attributes going forward #189.
emaderer
reviewed
Sep 14, 2021
| @@ -531,9 +531,9 @@ image: | |||
|
|
|||
| otelcol: | |||
| # The registry and name of the opentelemetry collector image to pull | |||
| repository: quay.io/signalfx/splunk-otel-collector | |||
| repository: quay.io/signalfx/splunk-otel-collector-dev | |||
There was a problem hiding this comment.
Why are pointing to dev? wouldn't that be unstable?
There was a problem hiding this comment.
I'm not going to release the chart yet. I'll bump the version to the next stable one with the chart release. There is functionality here that depends on the latest changes in https://github.com/signalfx/splunk-otel-collector
emaderer
approved these changes
Sep 14, 2021
Co-authored-by: Ryan Fitzpatrick <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change is the first step towards moving away from fluentd logs collection.
Fluentd "kubernetes_metadata_filter" plugin is known as a performance bottleneck in k8s logs collection with fluentd. So moving away from it already gives performance benefits. Also it mitigates issues when fluentd ends up hammering k8s API.
This change is compatible with otel gateway, if
otelCollector.enabled=trueall logs are forwarded through the gateway and logs enrichment is happening there reducing load on k8s API.Drawbacks:
container.image.namelogs attribute cannot be provided at the moment until this issue is resolved: [k8sprocessor] Add an option to fetch container data open-telemetry/opentelemetry-collector-contrib#5235Additional changes:
k8s.pod.labels.app. Value for this attribute is taken from pod's "app" label if it's set. The change is needed to support istio use case. We also want to enable more of these attributes going forward Capture some pod labels by default #189.