o2c-openobserve-collector-gateway-targetallocator is null

[system51]

Component(s)

No response

What happened?

Description

2024-01-31T08:22:59.534Z        info    [email protected]/receiver.go:53       Starting shared informers and wait for initial cache sync.   {"kind": "receiver", "name": "k8s_cluster", "data_type": "metrics"}
2024-01-31T08:22:59.635Z        info    [email protected]/receiver.go:74       Completed syncing shared informer caches.       {"kind": "receiver", "name": "k8s_cluster", "data_type": "metrics"}
2024-01-31T08:23:00.535Z        error   [email protected]/metrics_receiver.go:139      Failed to retrieve job list     {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "error": "Get \"http://o2c-openobserve-collector-gateway-targetallocator:80/scrape_configs\": dial tcp 10.43.168.28:80: connect: connection refused"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).syncTargetAllocator
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/metrics_receiver.go:139
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).startTargetAllocator
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/metrics_receiver.go:108
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).Start
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/metrics_receiver.go:92
go.opentelemetry.io/collector/service/internal/graph.(*Graph).StartAll
        go.opentelemetry.io/collector/[email protected]/internal/graph/graph.go:396
go.opentelemetry.io/collector/service.(*Service).Start
        go.opentelemetry.io/collector/[email protected]/service.go:163
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
        go.opentelemetry.io/collector/[email protected]/collector.go:190
go.opentelemetry.io/collector/otelcol.(*Collector).Run
        go.opentelemetry.io/collector/[email protected]/collector.go:229
go.opentelemetry.io/collector/otelcol.NewCommand.func1
        go.opentelemetry.io/collector/[email protected]/command.go:27
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:1115
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:1039
main.runInteractive
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:27
main.run
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:20
runtime.main
        runtime/proc.go:267
2024-01-31T08:23:00.535Z        info    [email protected]/service.go:185  Starting shutdown...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x45ed740]

goroutine 1 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8seventsreceiver.(*k8seventsReceiver).Shutdown(0xc00312b9e0, {0xc00257f640, 0xc0036bf1b8})
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/receiver.go:82 +0x60
go.opentelemetry.io/collector/service/internal/graph.(*Graph).ShutdownAll(0xc0026e85a0, {0x8d77e70, 0xdd0eec0})
        go.opentelemetry.io/collector/[email protected]/internal/graph/graph.go:435 +0x1c8
go.opentelemetry.io/collector/service.(*Service).Shutdown(0xc002696500, {0x8d77e70, 0xdd0eec0})
        go.opentelemetry.io/collector/[email protected]/service.go:191 +0xd7
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents(0xc002a3d220, {0x8d77e70, 0xdd0eec0})
        go.opentelemetry.io/collector/[email protected]/collector.go:191 +0x745
go.opentelemetry.io/collector/otelcol.(*Collector).Run(0xc002a3d220, {0x8d77e70, 0xdd0eec0})
        go.opentelemetry.io/collector/[email protected]/collector.go:229 +0x52
go.opentelemetry.io/collector/otelcol.NewCommand.func1(0xc001ffef00, {0x7e783c5?, 0x7?, 0x7e70b8b?})
        go.opentelemetry.io/collector/[email protected]/command.go:27 +0x6c
github.com/spf13/cobra.(*Command).execute(0xc001ffef00, {0xc0000b40d0, 0x1, 0x1})
        github.com/spf13/[email protected]/command.go:983 +0xabc
github.com/spf13/cobra.(*Command).ExecuteC(0xc001ffef00)
        github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(0x81f88a8?)
        github.com/spf13/[email protected]/command.go:1039 +0x13
main.runInteractive({0x81f88a8, {{0x7ea34eb, 0xf}, {0x7f38cd5, 0x1f}, {0x7e74c28, 0x6}}, 0x0, {0x0, 0x0}, ...})
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:27 +0x3d
main.run(...)
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:20 +0x118

[root@ricardo-k8s-lab-1 ~]# kubectl get ep -n openobserve-collector  
NAME                                                     ENDPOINTS                                                     AGE
o2c-openobserve-collector-agent-collector                10.42.0.127:4317,10.42.1.8:4317,10.42.2.15:4317 + 3 more...   52m
o2c-openobserve-collector-agent-collector-headless       10.42.0.127:4317,10.42.1.8:4317,10.42.2.15:4317 + 3 more...   52m
o2c-openobserve-collector-agent-collector-monitoring     10.42.0.127:8888,10.42.1.8:8888,10.42.2.15:8888               52m
o2c-openobserve-collector-gateway-collector                                                                            52m
o2c-openobserve-collector-gateway-collector-headless                                                                   52m
o2c-openobserve-collector-gateway-collector-monitoring                                                                 52m
o2c-openobserve-collector-gateway-targetallocator        <none>                                                        52m

Found endpoiont is empty

Steps to Reproduce

helm --namespace openobserve-collector \
  install o2c openobserve/openobserve-collector \
  --set exporters."otlphttp/openobserve".endpoint=http://10.168.136.154:5080/api/default/  \
  --set exporters."otlphttp/openobserve".headers.Authorization="Basic cm9vdEBleGFtcGxlLmNvbTowc3N1aDdVSjViWnFmbFBI"  \
  --set exporters."otlphttp/openobserve_k8s_events".endpoint=http://10.168.136.154:5080/api/default/  \
  --set exporters."otlphttp/openobserve_k8s_events".headers.Authorization="Basic cm9vdEBleGFtcGxlLmNvbTowc3N1aDdVSjViWnFmbFBI"

Expected Result

Actual Result

Collector version

v0.8.0

Environment information

Environment

centos 7.9

ricardo-k8s-lab-1.novalocal 3.10.0-1160.80.1.el7.x86_64

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for receiver/k8sevents: @dmitryax @TylerHelmuth. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@system51 please provide more context on the issue you're facing

[system51]

I deployed openobserve but when I configured Ingestion, I got an error following the deployment process.

kubectl get ep -n o2c-openobserve-collector-gateway-targetallocator  is null  Missing backend pod

[mrpotato3]

+1

[chenpeicheng3804]

Just annotate it

# Source: openobserve-collector/templates/gateway.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: o2c-openobserve-collector-gateway
  namespace: "openobserve-collector"
  # labels:
  #   helm.sh/chart: openobserve-collector-0.3.0
  #   app.kubernetes.io/name: openobserve-collector
  #   app.kubernetes.io/instance: o2c
  #   app.kubernetes.io/version: "0.93.0"
  #   app.kubernetes.io/managed-by: Helm

Unable to create reason
opentelemetry-operator-controller-manager log

{"level":"error","ts":"2024-02-29T06:59:02.340913407Z","msg":"Reconciler error","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"o2c-openobserve-collector-gateway","namespace":"openobserve-collector"},"namespace":"openobserve-collector","name":"o2c-openobserve-collector-gateway","reconcileID":"90ca57ef-4002-4333-8e2d-602365348269","error":"failed to create objects for o2c-openobserve-collector-gateway: Deployment.apps \"o2c-openobserve-collector-gateway-targetallocator\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\"app.kubernetes.io/component\":\"opentelemetry-targetallocator\", \"app.kubernetes.io/instance\":\"openobserve-collector.o2c-openobserve-collector-gateway\", \"app.kubernetes.io/managed-by\":\"opentelemetry-operator\", \"app.kubernetes.io/name\":\"openobserve-collector\", \"app.kubernetes.io/part-of\":\"opentelemetry\", \"app.kubernetes.io/version\":\"latest\", \"helm.sh/chart\":\"openobserve-collector-0.3.0\"}: `selector` does not match template `labels`","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

[chenpeicheng3804]

The same issue occurs when installing OpenTelemetry Collector using Help

[TylerHelmuth]

I am unable to troubleshoot openobserve. Generically that error message means the endpoint is likely incorrect.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole
• receiver/k8sevents: @dmitryax @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[system51]

/close