-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to set X-Scope-OrgID dynamically with headerssetterextension to send metrics to Grafana Mimir #27901
Comments
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
The main problem is that extensions don't get access to the telemetry data, and the header extension was mostly done with tenancy information coming from the connection (usually, HTTP headers) instead of coming from the telemetry itself. That said, I believe we have an issue tracking this feature request, although there's no ETA for implementing it. I can commit to reviewing a PR if you decide to contribute with a fix for that. |
Thanks for the answer @jpkrohling ! First of all I am quite new to OT, so maybe I am missing something, I have a couple of questions:
Thanks! |
|
For us, the first point is not a solution. Maybe for other people is fine, but we do not want to deal with "static" tenants (and also we can't). Regarding the second point, I am wondering if it is correct (or elegant) that a processor uses the context in that way. It seems the context holds information about the HTTP connection, then, does it make sense to add some data there which never was part of it? (just because there is an extension which can use it later). Maybe is better, one of this options:
Ofc, the advantage of the processor is that it can be used for any kind o headers and exporters, but not sure if it would be the correct way ... |
The context isn't necessarily the network context, but the "execution context". At some point, we need to split the batches per tenant, and when doing so, it's OK to mark the context as belonging to that specific tenant. About your alternatives, I believe the first wouldn't be desirable, as we want users of other distributions to be able to send data to Mimir with the regular OTLP exporter. Having a specific exporter for Mimir doesn't prevent that, but sends the wrong signal, IMO. The second one could work as well, but it still requires "something" to split the batches into connections, as each connection going to Mimir should have its own X-Scope-OrgID. |
Ok, I see your point and I agree with you. I have been looking to all components available to see the different options and I have found these:
With those two pieces, by adding a 3rd one which takes the resource attribute used for batching and grouping, then we get what is needed, right? We would need to create a new processor So, the pipeline would look something like this (notice the new
Do you think this makes sense? and will it work? I cannot promise that I will have time for a PR right now, but maybe in the near future I can do it or somebody else can take this idea. |
I believe it would indeed work, thank you for looking into this! If you decide to work on this, please follow the following guidelines for a new component: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-new-components |
Hi @jpkrohling. I have managed to create a prototype and it is working. I will work on it a bit more (to do the same for traces and logs) and follow the guide to add a new component. Most likely I will get int contact with you about the process, but probably not this week. |
@jriguera Have you elaborated your prototype further? I have the same requirement to set the X-Scope-OrdID dynamically on metrics (and traces/logs in the future). |
Yes, I did, we have it working in production. But I had no time to compile it with the latest version (it is working with version 0.91.0) |
I can provide a working configuration if you need it. We assign the tenant value from the namespace (so that is the default) or if there is an specific annotation in the pods, it takes the value of the tenant from there. It works for metrics, traces and logs. |
Thank you very much for your quick reply. The contextprocessor's README should be fine for the moment. |
Hi @jriguera - Glad to see that you have added "cotextprocessor" , Is this now part of the latest otel version 0.95.0 too ? |
No, this is not part of the otel collector standard distribution. You have to compile it by yourself (instructions are on the repo) and get the binary. I do not think it can be part of the standard distribution, I think the functionality provided by |
Component(s)
extension/headerssetter
What happened?
Description
We are trying to use the collector as a K8S daemonset to discover pods with annotations
and send metrics to Grafana Mimir setting the tenant from the namespace or from another
K8S annotation.
We hare deploying the
open-telemetry/opentelemetry-collector
with helm as a daemonset(see configuration below)
We are deploying podinfo as workload with
the annotations:
Steps to Reproduce
Given this collector configuration:
Then deploy a a pod (podinfo) in kubernetes with the defined annotations and wait for the metrics flow.
Metrics are successfully send to the
prometheusremotewrite/mimir
endpoint (also theotlphttp/mimir
works),but the tenant header (
X-Scope-OrgID
) expected by Grafana Mimir is not set in the HTTP request.Expected Result
Set dynamically the header based on a resource attribute.
As described in the readme of Headers Setter extension, the
intended use case to enable multi-tenancy for observability backends
such as Tempo, Mimir, Loki.
See a header
X-Scope-Orgid
with the value of the K8S namespaceActual Result
Only the header
X-Scope-OrgID3
is set from a static valuetenant3
.The other headers
X-Scope-OrgID
are not dynamically set.Log ouput from mockserver (see headers section):
Collector version
Image: otel/opentelemetry-collector-contrib:0.87.0
Environment information
Environment
Deployed in K8S with the official helm chart from https://open-telemetry.github.io/opentelemetry-helm-charts
OpenTelemetry Collector configuration
Helm configuration:
Log output
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: