Unable to export spans to Zipkin

[jonatan-ivanov]

Component(s)

exporter/zipkin

What happened?

Description

I'm trying to receive spans in OTLP format and export them to Zipkin (was able to do it in the past) but it seems something is broken now.

Steps to Reproduce

1. Start a collector (see version and config below)
2. Send spans in OTLP (I'm using HTPP but should not matter)
3. Check the logs and see the error

Expected Result

Spans in Zipkin

Actual Result

No Spans in Zipkin

Collector version

0.80.0, Docker image Id: 58eb1aeb91db

Environment information

Environment

OS: MacOS 13.4.1 with Docker (Desktop: 4.21.0 (113844), Engine: 24.0.2, Compose: v2.19.0)
Compiler(if manually compiled): N/A

OpenTelemetry Collector configuration

receivers:
    otlp:
        protocols:
            grpc:
            http:

processors:
    batch:
        timeout: 10s

exporters:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/zipkinexporter
    zipkin:
        endpoint: 'http://host.docker.internal:9411/api/v2/spans'
    logging:

extensions:
    health_check:
    pprof:
    zpages:

service:
    extensions: [pprof, zpages, health_check]
    pipelines:
        traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [logging, zipkin]

Log output

otel-collector  | 2023-06-30T19:45:14.371Z	info	exporterhelper/queued_retry.go:423	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "traces", "name": "zipkin", "error": "failed to push trace data via Zipkin exporter: Post \"http://host.docker.internal:9411/api/v2/spans\": EOF", "interval": "5.166770931s"}

Additional context

Please notice that seemingly the collector appends an extra character to the URL.
This is what's in my config: http://host.docker.internal:9411/api/v2/spans
And this is what's in the collector logs: http://host.docker.internal:9411/api/v2/spans\

[github-actions]

Pinging code owners:

• exporter/zipkin: @MovieStoreGuy @astencel-sumo @crobert-1

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @jonatan-ivanov, sorry for the delayed response. Are you running the collector in a docker container?

I think the extra \ character at the end of your exporter endpoint URL may be just escaping the double-quotes in the log message, just like Post \" in the same error message. I don't think that's necessarily anything to worry about here.

[jonatan-ivanov]

@crobert-1

Are you running the collector in a docker container?

Yes, see the details above (search for docker in the description for the versions).

I think the extra \ character at the end of your exporter endpoint URL may be just escaping the double-quotes in the log message, just like Post " in the same error message. I don't think that's necessarily anything to worry about here.

You are right about the escaped " but that's just a wrong guess I made, the issue is rather this is being broken, see: Exporting failed. Will retry the request after interval.

[crobert-1]

Would you be able to post more information about your setup? I ran a very basic demo using your configuration file and it worked as expected. Here's what worked for me.

Zipkin docker command:

docker run -d -p 9411:9411 openzipkin/zipkin

Collector command:

docker run -v $(pwd)/zipkin-config.yaml:/etc/otelcol-contrib/config.yaml -p 4317:4317 -p 4318:4318 otel/opentelemetry-collector-contrib:0.80.0

I also modified your config file in the following way, to make it more clear what traces the collector was receiving:

exporters:
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/zipkinexporter
  zipkin:
    endpoint: 'http://host.docker.internal:9411/api/v2/spans'
  logging:
    verbosity: detailed # This will provide more logging to see what the collector actually receives

After running these commands I just used CURL to post some sample trace data to localhost:4318/v1/traces, the HTTP endpoint for collector's OTLP receiver. The collector's output showed them coming in, and then going to localhost:9411 and searching for the trace showed that everything worked.

[jonatan-ivanov]

I tried it again with latest and it worked for me. Though the image ID for me locally (also in the description: 58eb1aeb91db) is not the same what I see on DockerHub. The Docker image tag should not be moved on DockerHub, right?

[crobert-1]

I just pulled the image and see my local image as well as digest does not match the dockerhub's value for digest either. I believe this is to be expected though. Here are a couple references.

Stack overflow question
GitHub issue discussion

Let us know if there's any other information that would be helpful! Otherwise, feel free to close the issue.

[jonatan-ivanov]

Ouch. :( Thanks for the help and sorry for the confusion!

[crobert-1]

You're welcome, happy to help!