Loki exporter is not setting X-scoppe-OrgID(tenant id) based on log attribute (loki.tenant) #26557
Comments
sairamsadanala
added
bug
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
I was able to get it working with a local Loki. What's your remote like? Is it Grafana Cloud Logs? Here's what I did: Here's my collector config: receivers:
otlp:
protocols:
grpc:
processors:
transform:
error_mode: ignore
log_statements:
- context: log
statements:
- set(attributes["loki.tenant"], "tenant")
exporters:
logging:
verbosity: detailed
loki:
endpoint: http://localhost:3100/loki/api/v1/push
service:
pipelines:
logs:
receivers: [otlp]
processors: [transform]
exporters: [logging, loki] |
|
I am using loki endpoint running on AWS EKS. It is working for the logs that were being processed by transform processor (extracting lokt.tenant) but it is not working for Attributes processor. attributes processed by Transform processor: Body: Str({"timestamp": "2023-09-08T17:59:01.965505Z", "kubernetes": {"host": "XXXXXXX", "namespace_name": "XXXX-testing"}}) attributes processed by Attributes processor: Body: Str({"timestamp": "2023-09-08T18:05:01.580359Z", "kubernetes": {"host": "XXXXXXX"}}) |
|
error - log output LogRecord #0 |
|
This isn't about the Loki exporter then, right? Your output shows the tenant for the attributes processor to be set to "mynamespace". You seem to have a configuration problem with the attributes processor instead, but perhaps I'm still missing something? What's the expected value for the tenant when using the attributes processor? |
|
expected value for tenant is "mytenantname" which is happening as shown above (loki.tenant: Str(mytenantname)). |
|
Something didn't seem right with my previous example, and I found out that I was running Loki with no auth. I'm not sure how I got an empty result for the first check, but I think I found the source of confusion of this ticket: the value for the receivers:
otlp:
protocols:
grpc:
processors:
transform:
error_mode: ignore
log_statements:
- context: log
statements:
- set(attributes["tenant"], "my-tenant")
- set(attributes["loki.tenant"], "tenant")
- set(resource.attributes["restenant"], "my-restenant")
- set(resource.attributes["loki.tenant"], "restenant")
exporters:
logging:
verbosity: detailed
loki:
endpoint: http://localhost:3100/loki/api/v1/push
service:
pipelines:
logs:
receivers: [otlp]
processors: [transform]
exporters: [logging, loki]In this example, the tenant will be set to "my-restenant", taken from the resource attribute "restenant". In your case, either use the header setter extension to use the value mytenantname as the source of information, or store it on another attribute and reference that via loki.tenant. |
|
Agree with Juraci. That would fulfill the second Juraci's proposal: |
|
thanks @mar4uk @jpkrohling. I have removed the suggested change.. My config file I am stilling getting the same error [401 unauthorized no orgid].
Am I missing anything here? are there any conflicts using Transform and Attributes processor together to set loki.tenant hint? It is working fine with Transform processor case but not with Attributes case. Is there way we can use Transform Processor to extract value from log.file.path attribute and set loki.enant hint? |
|
o, I got it. you should use value instead of from_attribute. The See |
|
Thanks @mar4uk @jpkrohling.. My apologies for the delayed response. It solved my issue and thanks a ton again, |
Component(s)
exporter/loki
What happened?
Description
I am setting up otelcol-contrib hub to receive telemetry(logs) from different sources and exporting them to loki adding tenant(X-Scope-OrgID) dynamically. Otelcol-contrib hub receives log.file.path: Str(/var/log/pods/_XXXX-sample-XX-default-1_d30d866a-b217-4d9a-a4c3-6dd8efb4a79a/XXX-XXX-XX-XXX/0.log) attribute and need to extract value from log.file.path attribute and set loki.tenant attribute while exporting logs to loki endpoint.
Steps to Reproduce
Expected Result
Loki exporter should set orgid if loki.tenant hint exist, while exporting logs to loki endpoint.
Actual Result
("error": "Permanent error: HTTP 401 "Unauthorized": no org id", "dropped_items": XX}.
Collector version
0.82.0
Environment information
Environment
OS: AWS Kubernetes
OpenTelemetry Collector configuration
Log output
attributes processed by Transform processor: Body: Str({"timestamp": "2023-09-08T17:59:01.965505Z", "kubernetes": {"host": "XXXXXXX", "namespace_name": "XXXX-testing"}}) Attributes: -> log.file.name: Str(mylog.log) -> log.file.path: Str(/var/log/mylog.log) -> loki.tenant: Str(tenant) -> tenant: Str(XXXX-testing) *************************************************************** attributes processed by Attributes processor: Body: Str({"timestamp": "2023-09-08T18:05:01.580359Z", "kubernetes": {"host": "XXXXXXX"}}) Attributes: -> log.file.name: Str(mylog.log) -> log.file.path: Str(/var/log/pods/mynamespace_XXXX-sample-es-default-1_d30d866a-b217-4d9a-a4c3-6dd8efb4a79a/XXXXXXX/mylog.log) -> loki.tenant: Str(mynamespace)Additional context
it is using an attributes processor and a transform processor to extract Kubernetes namespace. Transform processor looks for log body and extract kubernetes namespace and set loki.tenant attribute, and the attributes processor extract namespace value from log.file.path attribute and set loki.tenant.
Otelcol-contrib able to send the logs to loki when the logs are processed by Transform processor but failing ("error": "Permanent error: HTTP 401 "Unauthorized": no org id", "dropped_items": XX} when logs are processed by Attributes processor. I believe it is because of the tenant value is missing as show above.
As per loki exporter documentation, If the loki.tenant hint attribute is present in both resource and log attributes, then the look-up for a tenant value from resource attributes takes precedence. It is working when the logs are processed by Transform but not in Attributes processor case. Am I missing anything here?
The text was updated successfully, but these errors were encountered: