Otelcol-contrib attributes extract action issue

[sairamsadanala]

Component(s)

processor/attributes

What happened?

Description

I am setting up otelcol-contrib hub to receive telemetry(logs) from different sources and exporting them to loki adding tenant(X-Scope-OrgID) dynamically. Otelcol-contrib hub receives log.file.path: Str(/var/log/pods/_XXXX-sample-XX-default-1_d30d866a-b217-4d9a-a4c3-6dd8efb4a79a/XXX-XXX-XX-XXX/0.log) attribute and need to extract value from log.file.path attribute and set loki.tenant attribute while exporting logs to loki endpoint.

Steps to Reproduce

  extensions:
    health_check:
      endpoint: 0.0.0.0:13133

  processors:
    memory_limiter: null
    attributes:
      actions:
        - key: "loki.tenant"
          action: extract
          pattern: \/var\/log\/pods\/([^_]+)_.*\/\d+\.log           
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318
          include_metadata: true
  exporters:
    logging:
      verbosity: detailed
    loki:
      endpoint: "http://aaaa-bbb.ccc.xx/loki/api/v1/push"
      tls:
          insecure: false
          insecure_skip_verify: true
  service:
    extensions:
      - health_check
    pipelines:
      logs:
        exporters:
          - logging
          - loki
        receivers:
          - otlp
        processors:
          - memory_limiter
          - attributes

Expected Result

we should see the loki.tenant

Actual Result

error: failed to build pipelines: failed to create "attributes" processor, in pipeline "logs": error creating AttrProc. Field "pattern" contains at least one unnamed matcher group at the 0-th actions

Collector version

0.82.0

Environment information

Environment

OS: AWS Kubernetes

OpenTelemetry Collector configuration

extensions:
    health_check:
      endpoint: 0.0.0.0:13133

  processors:
    memory_limiter: null
    attributes:
      actions:
        - key: loki.tenant
          action: extract
          pattern: \/var\/log\/pods\/([^_]+)_.*\/\d+\.log           
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318
          include_metadata: true
  exporters:
    logging:
      verbosity: detailed
    loki:
      endpoint: "http://aaaa-bbb.ccc.xx/loki/api/v1/push"
      tls:
          insecure: false
          insecure_skip_verify: true
  service:
    extensions:
      - health_check
    pipelines:
      logs:
        exporters:
          - logging
          - loki
        receivers:
          - otlp
        processors:
          - memory_limiter
          - attributes

Log output

Error: failed to build pipelines: failed to create "attributes" processor, in pipeline "logs": error creating AttrProc. Field "pattern" contains at least one unnamed matcher group at the 0-th actions
2023/09/X XX:XX:XXcollector server run finished with error: failed to build pipelines: failed to create "attributes" processor, in pipeline "logs": error creating AttrProc. Field "pattern" contains at least one unnamed matcher group at the 0-th actions

Additional context

No response

[github-actions]

Pinging code owners:

• processor/attributes: @boostchicken

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @sairamsadanala, to confirm my understanding, you're receiving the log.file.path attribute, and want to extract information from the value of this attribute to set the value of a new attribute named loki.tenant, is that correct?

I think the config is essentially backwards, where your key should be the existing key. It should look like this:

attributes:
      actions:
        - key: log.file.path
           action: extract

The logic here is that we want to extract a new attribute from the value of the specified key log.file.path. The regex is then used to match a part of the value of the log.file.path, and name whatever part is matched loki.tenant.

The error message you're getting is saying that you have a matcher group in your regex, but there isn't a name specified to set what's matched. Refer to this example to see how to name the matched group.

Happy to help if I missed something here or if there's any confusion.

[sairamsadanala]

Hello @crobert-1, Thanks for the response.

If I understand correctly is this what you are suggesting?
attributes:
actions:
- key: "log.file.path"
action: extract
pattern: /var/log/pods/(?P<loki.tenant>.)_./\d+.log

Value of log.file.path attribute is something like /var/log/pods/_XXXX-sample-XX-default-1_d30d866a-b217-4d9a-a4c3-6dd8efb4a79a/XXX-XXX-XX-XXX/0.log. I guess the pattern that I mentioned above is invalid.

1. Is this the correct approach?
2. Any suggestions on pattern for the sample value.

[crobert-1]

1. Yes, your config is the correct idea. One thing to note with this functionality is that I don't see a way to directly name an attribute with a . character in it, as it's interpreted as a REGEX character. What you can do is name it loki_tenant, then add more operations to the attributes processor to rename it (upsert from loki_tempo, then delete loki_tempo).
2. I'm not familiar enough with your use case to know the best regex here. Theoretically, if you don't care about anything else in the file path after the loki_tenant value, you can just match what should separate the loki_tenant value from the remaining path. It would look something like: \/var\/log\/pods\/(?P<loki_tenant>[^\/]*). This would set loki_tempo to _XXXX-sample-XX-default-1_d30d866a-b217-4d9a-a4c3-6dd8efb4a79a in your example.

[sairamsadanala]

Thanks @crobert-1 I am able to extract the value and set as loki.tenant as suggested I am running into other issues now.

my otelcol-contrib configuration-

exporters:
  logging:
    verbosity: detailed
  loki:
    endpoint: http://XXXXXXXXXXXXXXXX/loki/api/v1/push
    tls:
      insecure: false
      insecure_skip_verify: true
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
processors:
  attributes:
    actions:
    - action: extract
      key: log.file.path
      pattern: \/var\/log\/pods\/(?P<loki_temp>[0-9a-z-]*)_.*
    - action: upsert
      from_attribute: loki_temp
      key: loki.tenant
    - action: delete
      key: loki_temp
  transform:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
      - set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")
      - set(attributes["tenant"], attributes["cache"]["kubernetes"]["namespace_name"])
      - set(attributes["loki.tenant"], "tenant")
      - delete_key(attributes, "cache")
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        include_metadata: true

service:
  extensions:
  - health_check
  pipelines:
    logs:
      exporters:
      - logging
      - loki
      processors:
      - memory_limiter
      - transform
      - attributes
      receivers:
      - otlp

it is using attributes processor and transform. Transform processor looks for log body and extractkubernetes namespace and set loki.tenant, and as you are aware attributes processor extract namespace value from log.file.path and set loki.tenant.

attributes processed by Transform processor:

Body: Str({"timestamp": "2023-09-08T17:59:01.965505Z", "kubernetes": {"host": "XXXXXXX", "namespace_name": "XXXX-testing"}})
Attributes:
-> log.file.name: Str(mylog.log)
-> log.file.path: Str(/var/log/mylog.log)
-> loki.tenant: Str(tenant)
-> tenant: Str(XXXX-testing)
attributes processed by Attributes processor:

Body: Str({"timestamp": "2023-09-08T18:05:01.580359Z", "kubernetes": {"host": "XXXXXXX"}})
Attributes:
-> log.file.name: Str(mylog.log)
-> log.file.path: Str(/var/log/pods/mynamespace_XXXX-sample-es-default-1_d30d866a-b217-4d9a-a4c3-6dd8efb4a79a/XXXXXXX/mylog.log)
-> loki.tenant: Str(mynamespace)

Otelcol-contrib able to send the logs to loki when the logs are processed by Transform processor but failing ("error": "Permanent error: HTTP 401 "Unauthorized": no org id", "dropped_items": XX}. I believe it is because the tenant value is missing as show above.

As per loki exporter documentation, If the loki.tenant hint attribute is present in both resource and log attributes, then the look-up for a tenant value from resource attributes takes precedence. It is working when the logs are processed by Transform but not in Attributes processor case. Am I missing anything year?

[crobert-1]

I'm not very familiar with the loki exporter, so I'll defer to the code owners. Hopefully they can help with usage here!

[github-actions]

Pinging code owners for exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sairamsadanala]

@gramidt @gouthamve @jpkrohling @mar4uk
Can you please help me?

[sairamsadanala]

/label receiver/otlphttp help-wanted -exporter/loki

[sairamsadanala]

/label receiver/otlphttp help-wanted exporter/loki

[github-actions]

Pinging code owners for exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

I'm closing this in favor of #26557