Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

otel-collector-1 container not starting: error decoding 'exporters': unknown type: "jaeger" for id: "jaeger" #5189

Closed
avolts opened this issue Jan 19, 2024 · 2 comments
Labels

Comments

@avolts
Copy link

avolts commented Jan 19, 2024

Describe the bug

After following the installation guide for Docker Compose for SeldenCoreV2, the otel-collector-1 container cannot start due to the following error:
error decoding 'exporters': unknown type: "jaeger" for id: "jaeger" (valid values: [opensearch prometheus awsxray azuredataexplorer azuremonitor carbon cassandra mezmo sentry file honeycombmarker opencensus sumologic sapm skywalking alertmanager alibabacloud_logservice awscloudwatchlogs dataset googlecloud pulsar datadog f5cloud loadbalancing logicmonitor logzio logging clickhouse loki debug otlp awsemf coralogix instana tencentcloud_logservice dynatrace elasticsearch googlecloudpubsub prometheusremotewrite signalfx otlphttp awskinesis awss3 googlemanagedprometheus influxdb kafka splunk_hec syslog zipkin])

To reproduce

  1. Follow the installation instructions for Docker Compose https://docs.seldon.io/projects/seldon-core/en/v2/contents/getting-started/docker-installation/index.html and set CUSTOM_IMAGE_TAG=2.6.0
  2. Check running containers in Docker Desktop
  3. See that otel-collector-1 cannot start

Expected behaviour

I expect that the otel-collector-1 container is starting without issues after local deployment.
Replacing jaeger with otlp at the marked places in the otel-collector-config.yaml solves the issue.
"jaeger" is no longer in the exporters values.
image

Environment

  • SeldonCoreV2 Installation on WSL Ubuntu-22.04
  • Docker Desktop v4.24.2
  • Windows Windows 10 Enterprise, 22H2
@avolts avolts added the bug label Jan 19, 2024
@sakoush sakoush added the v2 label Jan 19, 2024
@sakoush
Copy link
Member

sakoush commented Jan 19, 2024

A potential fix is described here

a workaround for now is to use an older version of otel/jeager via setting JAEGER_IMG=jaegertracing/all-in-one:1.49

lc525 added a commit that referenced this issue Feb 10, 2024
- fixes dataflow engine logging errors by specifying the otel exporter protocol

the updates are needed because the opentelemetry-java-instrumentation library requires a http(s) URI for the otel collector endpoint, regardless of the actual protocol. As the default for us is grpc, explicitly set the OTEL_EXPORTER_OTLP_PROTOCOL environment variable on dataflow pods

- fixes otel-collector config to remove deprecated jagger exporter (jagger now supports otel directly).

add the OTEL_EXPORTED_OTLP_PROTOCOL key to the seldon-tracing configMap and update the operator, crds and helm charts to support getting the value for this key from tracing config, similarly to how OTEL_EXPORTER_OTLP_PROTOCOL is fetched

- update versions used by ansible for jagger and opentelemetry-operator
- port 14250 no longer needs to be exposed under any config
- fix dependency ordering for dataflow/gradle.

previous ordering caused kafka-streams not to be able to find the slf4j logging provider
this lead to logs produced by kafka-streams not being recorded

Fixes #
Internal issue references:

#INFRA-568 Jagger latest is crashing
#INFRA-464 Otel is not able to parse its config (deprecated exporters)

Public issues:

otel-collector-1 container not starting: #5189 
related PR with partial otel functionality: #5170
@lc525
Copy link
Member

lc525 commented Feb 10, 2024

This is believed to be fixed via #5291

@lc525 lc525 closed this as completed Feb 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants