opentelemetry-collector between aws-otel-collector Permanent error: SerializationException

[SecretSun]

The aws-otel-collector erro:
2021-06-08T08:42:04.597Z error exporterhelper/queued_retry.go:243 Exporting failed. Try enabling retry_on_failure config option. {"kind": "exporter", "name": "awsxray", "error": "Permanent error: SerializationException: \n\tstatus code: 400, request id: bb92b15b-dea5-44c8-bd4a-68deeefd7735"}

Is there some bug between OpenTelemetry and X-Ray?

Our process is as follows:

opentelemetry-collector (DaemonSet)
			|
opentelemetry-collector (Svc, Control sampling rate)
			|
aws-otel-collector (Svc, Send data to x-ray)
			|
		    aws x-ray

opentelemetry-collector (DaemonSet run) conf:

  pipelines:
    traces:
      receivers: [otlp, opencensus, zipkin]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp]

opentelemetry-collector (Svc run, Control sampling rate) conf:

  pipelines:
    traces:
      receivers: [otlp, zipkin]
      processors: [memory_limiter, resource, batch]
      exporters: [kafka, otlp]

aws-otel-collector (Svc run, Send data to x-ray) conf:

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [awsxray]

[pingleig]

Which version of otel and aws-otel-collector are you using? It seems your pipeline is distributed and only using aws-otel-collector for its xray reporter?

There might be some internal data format mismatch due to different version of core and contrib repo. Can you try using latest aws-otel-collector, or just use https://github.com/open-telemetry/opentelemetry-collector-contrib (the xray reporter is in the contrib repo, we are just importing it in aws distro). The contrib repo release is always aligned with core collector release while the aws distro may not be up to date.

[mxiamxia]

Additionally, It looks like the x-ray request generated in aws-otel-collector (Svc run, Send data to x-ray) is invalid. could you enable debug logging(see example) with "--log-level=DEBUG" and then search request: in the collector logs and send us the x-ray raw request data.

logger.Debug("request: " + input.String())

[SecretSun]

Which version of otel and aws-otel-collector are you using? It seems your pipeline is distributed and only using aws-otel-collector for its xray reporter?

There might be some internal data format mismatch due to different version of core and contrib repo. Can you try using latest aws-otel-collector, or just use https://github.com/open-telemetry/opentelemetry-collector-contrib (the xray reporter is in the contrib repo, we are just importing it in aws distro). The contrib repo release is always aligned with core collector release while the aws distro may not be up to date.

aws-otel-collector:v0.10.0
opentelemetry-collector:0.27.0

[SecretSun]

Additionally, It looks like the x-ray request generated in aws-otel-collector (Svc run, Send data to x-ray) is invalid. could you enable debug logging(see example) with "--log-level=DEBUG" and then search request: in the collector logs and send us the x-ray raw request data.

logger.Debug("request: " + input.String())

We opened Debug and found the following.

(no unique labels)
2021-06-10T03:30:09.566Z debug [email protected]/awsxray.go:65 Error translating span. {"kind": "exporter", "name": "awsxray", "error": "invalid xray traceid: f62eb7c0526ad8c857876c1a8bd8cd13"}

We think there is a problem with opentelemetry-collector forward aws-otel-collector.

[SecretSun]

w3c traceid format:https://www.w3.org/TR/trace-context/#trace-id

xray taceid: https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sendingdata.html#xray-api-traceids

It is possible to do this automatically using the AWS Tracing SDK？

[SecretSun]

The translator has problem.

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awsxrayexporter/translator/segment.go#L308

The lack of int64(binary.BigEndian.Uint32(traceIDBytes[0:4])) , w3c has no such thing.
Is this generated by AWS SDK?
But we didn't use it? How to fix it?

[mxiamxia]

How was you trace data generated in your service? It looks like your service is not generating the trace data with x-ray format of trace ID and the x-ray exporter failed to process it. X-Ray Exporter only takes the x-ray format of trace id for now. And we had provided xrayidgenerator you can enable it in your service if you're using OTel SDK. Or you can directly use aws-otel-java-instrumentation to send trace data with x-ray traceid.

[SecretSun]

How was you trace data generated in your service? It looks like your service is not generating the trace data with x-ray format of trace ID and the x-ray exporter failed to process it. X-Ray Exporter only takes the x-ray format of trace id for now. And we had provided xrayidgenerator you can enable it in your service if you're using OTel SDK. Or you can directly use aws-otel-java-instrumentation to send trace data with x-ray traceid.

The aws-otel-collector does not convert, We have to use the X-Ray SDK?

[anuraaga]

This seems to be a duplicate of open-telemetry/opentelemetry-collector#3405

Are you not able to use the aws-otel components to enable the ID generator? It's not recommended to use the X-Ray SDK over those. But the collector cannot do the conversion by itself since IDs are generated and propagated with an app.

[SecretSun]

This seems to be a duplicate of open-telemetry/opentelemetry-collector#3405

Are you not able to use the aws-otel components to enable the ID generator? It's not recommended to use the X-Ray SDK over those. But the collector cannot do the conversion by itself since IDs are generated and propagated with an app.

Thank you so much...