k8sattributes processor attributes not appearing in Azure Monitor Logs #24210

zparnold · 2023-07-11T15:15:05Z

Component(s)

exporter/azuremonitor

What happened?

Description

When I use the logging exporter I can see the k8sattributes values added in the collector logs. But when I add the azuremonitor exporter, there are no k8sattributes in the log lines. I expect to see them in the customDimensions but they are nowhere.

Steps to Reproduce

Setup collector in daemonset mode, adding otlp as receiver, k8sattributes processor, and azuremonitor exporter
Generate a log line
See that the k8sattributes don't make it to azuremonitor

Expected Result

To have the K8s pod name and other attributes present in the customDimensions section

Actual Result

(There are no k8sattributes to be found anywhere)

Collector version

v0.80.0

Environment information

Environment

OpenTelemetryOperator deployed collector

OpenTelemetry Collector configuration

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: default
spec:
  mode: daemonset
  hostNetwork: true
  serviceAccount: collector
  env:
    - name: KUBE_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
      k8sattributes:
    exporters:
      azuremonitor:
        instrumentation_key: xxxxxxxxxxxxx
      logging:
        verbosity: detailed
    extensions:
        zpages:
        health_check:
    service:
      extensions: [zpages, health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [azuremonitor]
        logs:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [azuremonitor, logging]
        metrics:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [azuremonitor]

Log output

otel-collector-bwhnc otc-container 2023-07-11T14:27:32.624Z     info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 6}
otel-collector-bwhnc otc-container 2023-07-11T14:27:32.624Z     info    ResourceLog #0
otel-collector-bwhnc otc-container Resource SchemaURL: https://opentelemetry.io/schemas/1.20.0
otel-collector-bwhnc otc-container Resource attributes:
otel-collector-bwhnc otc-container      -> container.id: Str(fda1b2f2263d79ceb923e17da9db2a22b1af5bccc413edd1b99804ce085041cb)
otel-collector-bwhnc otc-container      -> host.arch: Str(amd64)
otel-collector-bwhnc otc-container      -> host.name: Str(otel-agent-test-85657f98cf-97c77)
otel-collector-bwhnc otc-container      -> os.description: Str(Linux 5.4.0-42-generic)
otel-collector-bwhnc otc-container      -> os.type: Str(linux)
otel-collector-bwhnc otc-container      -> process.command_args: Slice(["/usr/lib/jvm/java-17-openjdk-amd64/bin/java","-jar","/app.jar"])
otel-collector-bwhnc otc-container      -> process.executable.path: Str(/usr/lib/jvm/java-17-openjdk-amd64/bin/java)
otel-collector-bwhnc otc-container      -> process.pid: Int(7)
otel-collector-bwhnc otc-container      -> process.runtime.description: Str(Private Build OpenJDK 64-Bit Server VM 17.0.7+7-Ubuntu-0ubuntu120.04)
otel-collector-bwhnc otc-container      -> process.runtime.name: Str(OpenJDK Runtime Environment)
otel-collector-bwhnc otc-container      -> process.runtime.version: Str(17.0.7+7-Ubuntu-0ubuntu120.04)
otel-collector-bwhnc otc-container      -> service.name: Str(otel-agent-test)
otel-collector-bwhnc otc-container      -> telemetry.auto.version: Str(1.27.0)
otel-collector-bwhnc otc-container      -> telemetry.sdk.language: Str(java)
otel-collector-bwhnc otc-container      -> telemetry.sdk.name: Str(opentelemetry)
otel-collector-bwhnc otc-container      -> telemetry.sdk.version: Str(1.27.0)
otel-collector-bwhnc otc-container      -> k8s.pod.ip: Str(10.240.226.19)
otel-collector-bwhnc otc-container      -> k8s.pod.name: Str(otel-agent-test-85657f98cf-97c77)
otel-collector-bwhnc otc-container      -> k8s.namespace.name: Str(kube-system)
otel-collector-bwhnc otc-container      -> k8s.pod.start_time: Str(2023-07-10 19:30:45 +0000 UTC)
otel-collector-bwhnc otc-container      -> k8s.pod.uid: Str(f032e276-6839-4c0c-8047-cc0c47df10c5)
otel-collector-bwhnc otc-container      -> k8s.deployment.name: Str(otel-agent-test)
otel-collector-bwhnc otc-container      -> k8s.node.name: Str(qn8-dm-kwiap104)
otel-collector-bwhnc otc-container      -> container.image.name: Str(creguie1bmrepo001.azurecr.io/architecture/otel-test)
otel-collector-bwhnc otc-container      -> container.image.tag: Str(latest)
otel-collector-bwhnc otc-container ScopeLogs #0
otel-collector-bwhnc otc-container ScopeLogs SchemaURL:
otel-collector-bwhnc otc-container InstrumentationScope com.msci.index.otelagentdemo.RollController
otel-collector-bwhnc otc-container LogRecord #0
otel-collector-bwhnc otc-container ObservedTimestamp: 2023-07-11 14:27:32.340106624 +0000 UTC
otel-collector-bwhnc otc-container Timestamp: 2023-07-11 14:27:32.34 +0000 UTC
otel-collector-bwhnc otc-container SeverityText: INFO
otel-collector-bwhnc otc-container SeverityNumber: Info(9)
otel-collector-bwhnc otc-container Body: Str(Anonymous player is rolling the dice: 2)
otel-collector-bwhnc otc-container Trace ID: e4dc6e8a9aa18f44d11361791ce9ffa9
otel-collector-bwhnc otc-container Span ID: 20e07ebb30a4091f
otel-collector-bwhnc otc-container Flags: 1
otel-collector-bwhnc otc-container ScopeLogs #1
otel-collector-bwhnc otc-container ScopeLogs SchemaURL:
otel-collector-bwhnc otc-container InstrumentationScope org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor
otel-collector-bwhnc otc-container LogRecord #0
otel-collector-bwhnc otc-container ObservedTimestamp: 2023-07-11 14:27:32.340881268 +0000 UTC
otel-collector-bwhnc otc-container Timestamp: 2023-07-11 14:27:32.34 +0000 UTC
otel-collector-bwhnc otc-container SeverityText: DEBUG
otel-collector-bwhnc otc-container SeverityNumber: Debug(5)
otel-collector-bwhnc otc-container Body: Str(Using 'text/html', given [text/html, application/xhtml+xml, image/webp, image/apng, application/xml;q=0.9, */*;q=0.8, application/signed-exchange;v=b3;q=0.7] and supported [text/plain, */*, application/json, application/*+json])
otel-collector-bwhnc otc-container Trace ID: e4dc6e8a9aa18f44d11361791ce9ffa9
otel-collector-bwhnc otc-container Span ID: 20e07ebb30a4091f
otel-collector-bwhnc otc-container Flags: 1
otel-collector-bwhnc otc-container LogRecord #1
otel-collector-bwhnc otc-container ObservedTimestamp: 2023-07-11 14:27:32.341067229 +0000 UTC
otel-collector-bwhnc otc-container Timestamp: 2023-07-11 14:27:32.341 +0000 UTC
otel-collector-bwhnc otc-container SeverityText: DEBUG
otel-collector-bwhnc otc-container SeverityNumber: Debug(5)
otel-collector-bwhnc otc-container Body: Str(Writing ["2"])
otel-collector-bwhnc otc-container Trace ID: e4dc6e8a9aa18f44d11361791ce9ffa9
otel-collector-bwhnc otc-container Span ID: 20e07ebb30a4091f
otel-collector-bwhnc otc-container Flags: 1
otel-collector-bwhnc otc-container ScopeLogs #2
otel-collector-bwhnc otc-container ScopeLogs SchemaURL:
otel-collector-bwhnc otc-container InstrumentationScope org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping
otel-collector-bwhnc otc-container LogRecord #0
otel-collector-bwhnc otc-container ObservedTimestamp: 2023-07-11 14:27:32.339495631 +0000 UTC
otel-collector-bwhnc otc-container Timestamp: 2023-07-11 14:27:32.339 +0000 UTC
otel-collector-bwhnc otc-container SeverityText: DEBUG
otel-collector-bwhnc otc-container SeverityNumber: Debug(5)
otel-collector-bwhnc otc-container Body: Str(Mapped to com.msci.index.otelagentdemo.RollController#index(Optional))
otel-collector-bwhnc otc-container Trace ID: e4dc6e8a9aa18f44d11361791ce9ffa9
otel-collector-bwhnc otc-container Span ID: ef482aef3435b87d
otel-collector-bwhnc otc-container Flags: 1
otel-collector-bwhnc otc-container ScopeLogs #3
otel-collector-bwhnc otc-container ScopeLogs SchemaURL:
otel-collector-bwhnc otc-container InstrumentationScope org.springframework.web.servlet.DispatcherServlet
otel-collector-bwhnc otc-container LogRecord #0
otel-collector-bwhnc otc-container ObservedTimestamp: 2023-07-11 14:27:32.338889173 +0000 UTC
otel-collector-bwhnc otc-container Timestamp: 2023-07-11 14:27:32.338 +0000 UTC
otel-collector-bwhnc otc-container SeverityText: DEBUG
otel-collector-bwhnc otc-container SeverityNumber: Debug(5)
otel-collector-bwhnc otc-container Body: Str(GET "/rolldice", parameters={})
otel-collector-bwhnc otc-container Trace ID: e4dc6e8a9aa18f44d11361791ce9ffa9
otel-collector-bwhnc otc-container Span ID: ef482aef3435b87d
otel-collector-bwhnc otc-container Flags: 1
otel-collector-bwhnc otc-container LogRecord #1
otel-collector-bwhnc otc-container ObservedTimestamp: 2023-07-11 14:27:32.341944354 +0000 UTC
otel-collector-bwhnc otc-container Timestamp: 2023-07-11 14:27:32.341 +0000 UTC
otel-collector-bwhnc otc-container SeverityText: DEBUG
otel-collector-bwhnc otc-container SeverityNumber: Debug(5)
otel-collector-bwhnc otc-container Body: Str(Completed 200 OK)
otel-collector-bwhnc otc-container Trace ID: e4dc6e8a9aa18f44d11361791ce9ffa9
otel-collector-bwhnc otc-container Span ID: ef482aef3435b87d
otel-collector-bwhnc otc-container Flags: 1
otel-collector-bwhnc otc-container      {"kind": "exporter", "data_type": "logs", "name": "logging"}

Additional context

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2023-07-11T15:17:24Z

Pinging code owners:

exporter/azuremonitor: @pcwiese

See Adding Labels via Comments if you do not have permissions to add labels yourself.

zparnold · 2023-07-11T15:25:14Z

For additional context, when we add elasticsearch/log as an exporter, we do get those attributes:

pcwiese · 2023-07-12T23:09:09Z

@bastbu Can you take a look here? I know you made a change a few months ago to include resource attributes in the log exporter...

bastbu · 2023-07-13T06:54:47Z

I tried to reproduce it, but for me all resource attributes originating from the k8sattributesprocessor show up correctly in Application Insights.

From the original description I can see that the k8s.pod.ip, which comes from the k8sattributesprocessor, shows up correctly in Application Insights.

The Elastic Resource dimensions do not show the net.host.name or net.protocol.name dimensions which should be visible as well, if the Resource.* fields are alphabetically ordered. Can we confirm that the configuration for the two collectors is the same and that it's not a configuration issue?

crobert-1 · 2023-08-31T23:28:46Z

/label waiting-for-author

github-actions · 2023-11-08T03:29:56Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/azuremonitor: @pcwiese

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2024-01-07T05:19:07Z

This issue has been closed as inactive because it has been stale for 120 days with no activity.

zparnold added bug Something isn't working needs triage New item requiring triage labels Jul 11, 2023

github-actions bot added the exporter/azuremonitor label Jul 11, 2023

github-actions bot added the waiting for author label Aug 31, 2023

This was referenced Sep 3, 2023

Weekly Report: 2023-08-27 - 2023-09-03 kevinslin/opentelemetry-collector-contrib#20

Open

Weekly Report: 2023-08-27 - 2023-09-03 kevinslin/opentelemetry-collector-contrib#21

Open

github-actions bot mentioned this issue Sep 6, 2023

Weekly Report: 2023-08-30 - 2023-09-06 kevinslin/opentelemetry-collector-contrib#22

Open

atoulme removed the needs triage New item requiring triage label Sep 9, 2023

github-actions bot added the Stale label Nov 8, 2023

github-actions bot added the closed as inactive label Jan 7, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

k8sattributes processor attributes not appearing in Azure Monitor Logs #24210

k8sattributes processor attributes not appearing in Azure Monitor Logs #24210

zparnold commented Jul 11, 2023 •

edited

Loading

github-actions bot commented Jul 11, 2023

zparnold commented Jul 11, 2023

pcwiese commented Jul 12, 2023

bastbu commented Jul 13, 2023

crobert-1 commented Aug 31, 2023

github-actions bot commented Nov 8, 2023

github-actions bot commented Jan 7, 2024

k8sattributes processor attributes not appearing in Azure Monitor Logs #24210

k8sattributes processor attributes not appearing in Azure Monitor Logs #24210

Comments

zparnold commented Jul 11, 2023 • edited Loading

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Jul 11, 2023

zparnold commented Jul 11, 2023

pcwiese commented Jul 12, 2023

bastbu commented Jul 13, 2023

crobert-1 commented Aug 31, 2023

github-actions bot commented Nov 8, 2023

github-actions bot commented Jan 7, 2024

zparnold commented Jul 11, 2023 •

edited

Loading