panic: runtime error: invalid memory address or nil pointer dereference #25125

semihural-tomtom · 2023-08-09T18:29:48Z

Component(s)

receiver/filelog

What happened?

Description

I tried to deploy otel-collector via helm chart with the below configuration with daemonset mode. But the collector pod
Back-off restarting failed due to below error.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
  - repo: https://open-telemetry.github.io/opentelemetry-helm-charts
    name: opentelemetry-collector
    valuesFile: values/values.yaml
    version: 0.65.0
    includeCRDs: false
    namespace: otel-collector
    releaseName: otel-collector

Collector version

0.82.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

mode: 'daemonset'
config:
  exporters:
    logging: {}
    loki:
      endpoint: https://xxxxloki/api/v1/push
      auth:
        authenticator: basicauth/logs
  extensions:
    basicauth/logs:
      client_auth:
        username: '${LOKI_USER_ID}'
        password: '${LOKI_API_KEY}'
    health_check: {}
    memory_ballast: {}
  processors:
    batch: {}
    memory_limiter: null
    attributes:
      actions:
        - action: insert
          key: loki.attribute.labels
          value: log.file.name
    k8sattributes:
      auth_type: 'serviceAccount'
      extract:
        metadata:
          - k8s.namespace.name
          - k8s.pod.name
          - k8s.pod.start_time
          - k8s.pod.uid
          - k8s.deployment.name
          - k8s.node.name
        labels:
          - tag_name: app.kubernetes.io/component
            key: app.kubernetes.io/component
            from: pod
          - tag_name: component
            key: component
            from: pod
          - tag_name: app.kubernetes.io/instance
            key: app.kubernetes.io/instance
            from: pod
          - tag_name: app.kubernetes.io/name
            key: app.kubernetes.io/name
            from: pod
          - tag_name: label.app
            key: name
            from: pod
          - tag_name: release
            key: release
            from: pod
  receivers:
    filelog:
      exclude:
        - /var/log/pods/*/otel-collector/*.log
      include:
        - /var/log/pods/*/*/*.log
      start_at: beginning
      include_file_path: true
      include_file_name: false
      operators:
        # Find out which format is used by kubernetes
        - type: router
          id: get-format
          routes:
            - output: parser-docker
              expr: 'body matches "^\\{"'
            - output: parser-crio
              expr: 'body matches "^[^ Z]+ "'
            - output: parser-containerd
              expr: 'body matches "^[^ Z]+Z"'
        # Parse CRI-O format
        - type: regex_parser
          id: parser-crio
          regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*)
            ?(?P<log>.*)$'
          output: extract_metadata_from_filepath
          timestamp:
            parse_from: attributes.time
            layout_type: gotime
            layout: '2006-01-02T15:04:05.999999999Z07:00'
        # Parse CRI-Containerd format
        - type: regex_parser
          id: parser-containerd
          regex:
            '^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*)
            ?(?P<log>.*)$'
          output: extract_metadata_from_filepath
          timestamp:
            parse_from: attributes.time
            layout: '%Y-%m-%dT%H:%M:%S.%LZ'
        # Parse Docker format
        - type: json_parser
          id: parser-docker
          output: extract_metadata_from_filepath
          timestamp:
            parse_from: attributes.time
            layout: '%Y-%m-%dT%H:%M:%S.%LZ'
        - type: move
          from: attributes.log
          to: body
        # Extract metadata from file path
        - type: regex_parser
          id: extract_metadata_from_filepath
          regex: '^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$'
          parse_from: attributes["log.file.path"]
          cache:
            size: 128 # default maximum amount of Pods per Node is 110
        # Rename attributes
        - type: move
          from: attributes.stream
          to: attributes["log.iostream"]
        - type: move
          from: attributes.container_name
          to: resource["k8s.container.name"]
        - type: move
          from: attributes.namespace
          to: resource["k8s.namespace.name"]
        - type: move
          from: attributes.pod_name
          to: resource["k8s.pod.name"]
        - type: move
          from: attributes.restart_count
          to: resource["k8s.container.restart_count"]
        - type: move
          from: attributes.uid
          to: resource["k8s.pod.uid"]
        - type: add
          field: resource["k8s.cluster.name"]
          value: 'aks-gha-westeurope-prod'
    jaeger:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:14250
        thrift_http:
          endpoint: ${env:MY_POD_IP}:14268
        thrift_compact:
          endpoint: ${env:MY_POD_IP}:6831
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318
    prometheus:
      config:
        scrape_configs:
          - job_name: opentelemetry-collector
            scrape_interval: 10s
            static_configs:
              - targets:
                  - ${env:MY_POD_IP}:8888
    zipkin:
      endpoint: ${env:MY_POD_IP}:9411
  service:
    telemetry:
      metrics:
        address: ${env:MY_POD_IP}:8888
    extensions:
      - basicauth/logs
      - health_check
      - memory_ballast
    pipelines:
      logs:
        exporters:
          - logging
          - loki
        processors:
          - memory_limiter
          - batch
          - attributes
        receivers:
          - otlp
      metrics:
        exporters:
          - logging
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
          - prometheus
      traces:
        exporters:
          - logging
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
          - jaeger
          - zipkin
image:
  # If you want to use the core image `otel/opentelemetry-collector`, you also need to change `command.name` value to `otelcol`.
  repository: otel/opentelemetry-collector-contrib
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: ''
  # When digest is set to a non-empty value, images will be pulled by digest (regardless of tag value).
  digest: ''
imagePullSecrets: []

# OpenTelemetry Collector executable
command:
  name: otelcol-contrib
  extraArgs: []

Log output

2023-08-09T18:16:37.884Z	info	service/telemetry.go:84	Setting up own telemetry...2023-08-09T18:16:37.884Z	info	service/telemetry.go:201	Serving Prometheus metrics	{"address": "10.1.0.54:8888", "level": "Basic"}2023-08-09T18:16:37.885Z	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "traces", "name": "logging"}2023-08-09T18:16:37.886Z	info	[email protected]/memorylimiter.go:138	Using percentage memory limiter	{"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "total_memory_mib": 15990, "limit_percentage": 80, "spike_limit_percentage": 25}2023-08-09T18:16:37.886Z	info	[email protected]/memorylimiter.go:102	Memory limiter configured	{"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "limit_mib": 12792, "spike_limit_mib": 3997, "check_interval": 5}2023-08-09T18:16:37.886Z	info	kube/client.go:107	k8s filtering	{"kind": "processor", "name": "k8sattributes", "pipeline": "traces", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-default-39669127-vmss000001"}2023-08-09T18:16:37.886Z	info	[email protected]/exporter.go:41	using the new Loki exporter	{"kind": "exporter", "data_type": "logs", "name": "loki"}2023-08-09T18:16:37.886Z	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "logging"}2023-08-09T18:16:37.886Z	info	kube/client.go:107	k8s filtering	{"kind": "processor", "name": "k8sattributes", "pipeline": "metrics", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-default-39669127-vmss000001"}2023-08-09T18:16:37.886Z	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "logging"}2023-08-09T18:16:37.887Z	info	kube/client.go:107	k8s filtering	{"kind": "processor", "name": "k8sattributes", "pipeline": "logs", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-default-39669127-vmss000001"}2023-08-09T18:16:37.887Z	info	service/service.go:132	Starting otelcol-contrib...	{"Version": "0.82.0", "NumCPU": 4}2023-08-09T18:16:37.887Z	info	extensions/extensions.go:30	Starting extensions...2023-08-09T18:16:37.887Z	info	extensions/extensions.go:33	Extension is starting...	{"kind": "extension", "name": "basicauth/logs"}2023-08-09T18:16:37.887Z	info	extensions/extensions.go:37	Extension started.	{"kind": "extension", "name": "basicauth/logs"}2023-08-09T18:16:37.887Z	info	extensions/extensions.go:33	Extension is starting...	{"kind": "extension", "name": "health_check"}2023-08-09T18:16:37.887Z	info	[email protected]/healthcheckextension.go:34	Starting health_check extension	{"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}2023-08-09T18:16:37.888Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}2023-08-09T18:16:37.888Z	info	extensions/extensions.go:37	Extension started.	{"kind": "extension", "name": "health_check"}2023-08-09T18:16:37.888Z	info	extensions/extensions.go:33	Extension is starting...	{"kind": "extension", "name": "memory_ballast"}2023-08-09T18:16:37.960Z	info	[email protected]/memory_ballast.go:41	Setting memory ballast	{"kind": "extension", "name": "memory_ballast", "MiBs": 6396}2023-08-09T18:16:37.960Z	info	extensions/extensions.go:37	Extension started.	{"kind": "extension", "name": "memory_ballast"}2023-08-09T18:16:37.961Z	warn	[email protected]/processor.go:54	k8s.pod.start_time value will be changed to use RFC3339 format in v0.83.0. see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/24016 for more information. enable feature-gate k8sattr.rfc3339 to opt into this change.	{"kind": "processor", "name": "k8sattributes", "pipeline": "traces"}2023-08-09T18:16:37.961Z	warn	[email protected]/processor.go:54	k8s.pod.start_time value will be changed to use RFC3339 format in v0.83.0. see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/24016 for more information. enable feature-gate k8sattr.rfc3339 to opt into this change.	{"kind": "processor", "name": "k8sattributes", "pipeline": "metrics"}2023-08-09T18:16:37.963Z	warn	[email protected]/processor.go:54	k8s.pod.start_time value will be changed to use RFC3339 format in v0.83.0. see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/24016 for more information. enable feature-gate k8sattr.rfc3339 to opt into this change.	{"kind": "processor", "name": "k8sattributes", "pipeline": "logs"}2023-08-09T18:16:37.963Z	info	[email protected]/otlp.go:83	Starting GRPC server	{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.1.0.54:4317"}2023-08-09T18:16:37.963Z	info	[email protected]/otlp.go:101	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.1.0.54:4318"}2023-08-09T18:16:37.963Z	info	adapter/receiver.go:45	Starting stanza receiver	{"kind": "receiver", "name": "filelog", "data_type": "logs"}2023-08-09T18:16:37.964Z	info	[email protected]/metrics_receiver.go:239	Starting discovery manager	{"kind": "receiver", "name": "prometheus", "data_type": "metrics"}2023-08-09T18:16:37.964Z	info	[email protected]/metrics_receiver.go:230	Scrape job added	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "opentelemetry-collector"}2023-08-09T18:16:37.965Z	info	[email protected]/metrics_receiver.go:278	Starting scrape manager	{"kind": "receiver", "name": "prometheus", "data_type": "metrics"}2023-08-09T18:16:37.965Z	info	healthcheck/handler.go:129	Health Check state change	{"kind": "extension", "name": "health_check", "status": "ready"}2023-08-09T18:16:37.965Z	info	service/service.go:149	Everything is ready. Begin running and processing data.2023-08-09T18:16:38.165Z	info	fileconsumer/file.go:203	Started watching file	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/arc-system_az-weu-ubuntu-20.04-amd-05cpu-stage-7b4c86df-listener_48a4a601-3df7-4933-aaad-c20042f68cd7/autoscaler/0.log"}2023-08-09T18:16:38.222Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 337, "log records": 8200}panic: runtime error: invalid memory address or nil pointer dereference[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x3339d4c]goroutine 117 [running]:go.opentelemetry.io/collector/pdata/pcommon.Map.Get(...)	go.opentelemetry.io/collector/[email protected]/pcommon/map.go:54github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.getNestedAttribute({0xc000cb63a4?, 0xd?}, {0x81a1288?})	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/convert.go:117 +0x4cgithub.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.getNestedAttribute({0xc000cb63a0?, 0xc?}, {0x432bb0?})	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/convert.go:126 +0x14bgithub.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.convertAttributesToLabels({0xc191f4cf88?}, {0x41730b?})	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/convert.go:104 +0xe7github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.convertAttributesAndMerge({0x63e7a40?}, {0xc000c21980?}, 0x747f8b4?)	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/convert.go:52 +0x6efgithub.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.LogToLokiEntry({0x63edce0?}, {0xc191f4d3b8?}, {0x0?}, 0x0?)	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/logs_to_loki.go:148 +0x110github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.LogsToLokiRequests({0x1?}, 0x0?)	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/logs_to_loki.go:73 +0x4fcgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/lokiexporter.(*lokiExporter).pushLogData(0xc000c78240, {0x8234bd8, 0xc193729b30}, {0xc1933b1880?})	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/exporter.go:50 +0x58go.opentelemetry.io/collector/exporter/exporterhelper.(*logsRequest).Export(0xbc14de?, {0x8234bd8?, 0xc193729b30?})	go.opentelemetry.io/collector/[email protected]/exporterhelper/logs.go:54 +0x34go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send(0xc0004a32e8, {0x82574b0, 0xc193729b00})	go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:197 +0x96

Additional context

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2023-08-09T18:30:08Z

Pinging code owners:

receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

TylerHelmuth · 2023-08-09T18:39:24Z

@semihural-tomtom can you break the logs into separate lines?

djaglowski · 2023-08-09T18:39:31Z

I don't see any indication that there was a problem with the filelog receiver. Based on the logged error, it appears to be the loki translator.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x3339d4c] goroutine 117 [running]:
go.opentelemetry.io/collector/pdata/pcommon.Map.Get(...)
	go.opentelemetry.io/collector/[email protected]/pcommon/map.go:54
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.getNestedAttribute({0xc000cb63a4?, 0xd?}, {0x81a1288?})
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/convert.go:117 +0x4c
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.getNestedAttribute({0xc000cb63a0?, 0xc?}, {0x432bb0?})
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/convert.go:126 +0x14b
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.convertAttributesToLabels({0xc191f4cf88?}, {0x41730b?})
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/convert.go:104 +0xe7
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.convertAttributesAndMerge({0x63e7a40?}, {0xc000c21980?}, 0x747f8b4?)
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/convert.go:52 +0x6ef
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.LogToLokiEntry({0x63edce0?}, {0xc191f4d3b8?}, {0x0?}, 0x0?)
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/logs_to_loki.go:148 +0x110
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.LogsToLokiRequests({0x1?}, 0x0?)
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/[email protected]/logs_to_loki.go:73 +0x4fc
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/lokiexporter.(*lokiExporter).pushLogData(0xc000c78240, {0x8234bd8, 0xc193729b30}, {0xc1933b1880?})
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/exporter.go:50 +0x58
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsRequest).Export(0xbc14de?, {0x8234bd8?, 0xc193729b30?})
	go.opentelemetry.io/collector/[email protected]/exporterhelper/logs.go:54 +0x34
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send(0xc0004a32e8, {0x82574b0, 0xc193729b00})
	go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:197

TylerHelmuth · 2023-08-09T18:40:19Z

@djaglowski agreed

github-actions · 2023-08-09T18:40:20Z

Pinging code owners for pkg/translator/loki: @gouthamve @jpkrohling @mar4uk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2023-08-09T18:40:20Z

Pinging code owners for exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2023-08-09T18:40:21Z

Pinging code owners for translator/loki: @gouthamve @jpkrohling @mar4uk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

TylerHelmuth · 2023-08-09T18:44:05Z

Latest changes look to be:

[exporter/loki] Add setting default_labels_enabled to the config #23863
[translator/loki] fix converting attributes to labels when hint is a slice and severity_number is present in log record #23810

semihural-tomtom · 2023-08-10T08:44:18Z

Seems the issue is related to the below attributes.

  processors:
    batch: {}
    memory_limiter: null
    attributes:
      actions:
        - action: insert
          key: loki.attribute.labels
          value: log.file.name

mar4uk · 2023-08-10T08:45:27Z

@semihural-tomtom could you please enable verbosity: detailed for the logging exporter and share the Log Record that fails on loki exporter?

UPDATE
Log Record is not needed, the issue is identified

mar4uk · 2023-08-10T13:35:45Z

Thank you @semihural-tomtom for reporting this bug! The fix is on the way #25142

semihural-tomtom added bug Something isn't working needs triage New item requiring triage labels Aug 9, 2023

github-actions bot added the receiver/filelog label Aug 9, 2023

djaglowski added exporter/loki Loki Exporter translator/loki pkg/translator/loki and removed receiver/filelog needs triage New item requiring triage labels Aug 9, 2023

mar4uk mentioned this issue Aug 10, 2023

[translator/loki] Fixes a panic that occurred during the promotion of nested attributes containing dots to labels #25142

Merged

dmitryax closed this as completed Aug 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic: runtime error: invalid memory address or nil pointer dereference #25125

panic: runtime error: invalid memory address or nil pointer dereference #25125

semihural-tomtom commented Aug 9, 2023 •

edited by MovieStoreGuy

Loading

github-actions bot commented Aug 9, 2023

TylerHelmuth commented Aug 9, 2023

djaglowski commented Aug 9, 2023 •

edited

Loading

TylerHelmuth commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

TylerHelmuth commented Aug 9, 2023

semihural-tomtom commented Aug 10, 2023

mar4uk commented Aug 10, 2023 •

edited

Loading

mar4uk commented Aug 10, 2023

panic: runtime error: invalid memory address or nil pointer dereference #25125

panic: runtime error: invalid memory address or nil pointer dereference #25125

Comments

semihural-tomtom commented Aug 9, 2023 • edited by MovieStoreGuy Loading

Component(s)

What happened?

Description

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Aug 9, 2023

TylerHelmuth commented Aug 9, 2023

djaglowski commented Aug 9, 2023 • edited Loading

TylerHelmuth commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

TylerHelmuth commented Aug 9, 2023

semihural-tomtom commented Aug 10, 2023

mar4uk commented Aug 10, 2023 • edited Loading

mar4uk commented Aug 10, 2023

semihural-tomtom commented Aug 9, 2023 •

edited by MovieStoreGuy

Loading

djaglowski commented Aug 9, 2023 •

edited

Loading

mar4uk commented Aug 10, 2023 •

edited

Loading