load-balancing exporter not route some traceID to same collector

[Liubey]

Component(s)

exporter/loadbalancing

What happened?

Description

I use sidecar-->collector cluster--> jaeger mode to collector span.
In sidecar, I use load balance to route traceID to some collector, so the config is:

    exporters:
      logging:
        loglevel: info
      loadbalancing:
        routing_key: "traceID"
        protocol:
          otlp:
            timeout: 10s
            compression: gzip
            tls:
              insecure: true
        resolver:
          dns:
            hostname: tracing-opentelemetry-collector.tracing.svc.cluster.local

By the way, I used resolver by dns before, But I got some result(same traceID not route to same collector).

In collector, I use tail_sampling to collector the trace I want, and use resourcedetection to tag collector id to test, the config is:

      tail_sampling:
        decision_wait: 30s
        num_traces: 5000
        expected_new_traces_per_sec: 500
        policies:
          [
            {
              name: error,
              type: boolean_attribute,
              boolean_attribute: { key: error, value: true }
            },
            {
              name: status_code,
              type: numeric_attribute,
              numeric_attribute: { key: http.status_code, min_value: 400, max_value: 400 }
            },
            {
              name: status_code,
              type: numeric_attribute,
              numeric_attribute: { key: http.status_code, min_value: 402, max_value: 504 }
            },
            {
              name: latency,
              type: latency,
              latency: { threshold_ms: 1 } ####### It was config 2000, but modified to 1ms to test.
            },
          ]

But I got the trace like:

You can see , ### the same traceID'spans reported by two collector, It's weird.
By the way, Server and Task are two servcie, Task published by Server, and execute by Task, Very typical asynchronous architecture.

Steps to Reproduce

Expected Result

I want same traceID route to the same collector, So I can use tail-sampling better to collector the interesting trace.

Actual Result

same traceID spans data route to different collector.

Collector version

0.85.0

Environment information

Environment

K8S
debian

OpenTelemetry Collector configuration

No response

Log output

2023-09-20T05:59:23.457Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel created {"grpc_log": true}
2023-09-20T05:59:23.459Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] original dial target is: "10.100.194.91:4317" {"grpc_log": true}
2023-09-20T05:59:23.459Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] dial target "10.100.194.91:4317" parse failed: parse "10.100.194.91:4317": first path segment in URL cannot contain colon {"grpc_log": true}
2023-09-20T05:59:23.459Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] fallback to scheme "passthrough" {"grpc_log": true}
2023-09-20T05:59:23.459Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] parsed dial target is: {URL:{Scheme:passthrough Opaque: User: Host: Path:/10.100.194.91:4317 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} {"grpc_log": true}
2023-09-20T05:59:23.459Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel authority set to "10.100.194.91:4317" {"grpc_log": true}
2023-09-20T05:59:23.461Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Resolver state updated: {
"Addresses": [
{
"Addr": "10.100.194.91:4317",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Metadata": null
}
],
"Endpoints": [
{
"Addresses": [
{
"Addr": "10.100.194.91:4317",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Metadata": null
}
],
"Attributes": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses) {"grpc_log": true}
2023-09-20T05:59:23.461Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel switches to new LB policy "pick_first" {"grpc_log": true}
2023-09-20T05:59:23.461Z info zapgrpc/zapgrpc.go:178 [core] [pick-first-lb 0xc002b4da70] Received new config {
"shuffleAddressList": false
}, resolver state {
"Addresses": [
{
"Addr": "10.100.194.91:4317",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Metadata": null
}
],
"Endpoints": [
{
"Addresses": [
{
"Addr": "10.100.194.91:4317",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Metadata": null
}
],
"Attributes": null
}
],
"ServiceConfig": null,
"Attributes": null
} {"grpc_log": true}
2023-09-20T05:59:23.461Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel created {"grpc_log": true}
2023-09-20T05:59:23.462Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel Connectivity change to CONNECTING {"grpc_log": true}
2023-09-20T05:59:23.464Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING {"grpc_log": true}
2023-09-20T05:59:23.465Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel picks a new address "10.100.194.91:4317" to connect {"grpc_log": true}

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Liubey]

In my logs, 10.100.194.91 is K8S' Service IP, Is it right?
Maybe It should resolve the real IPs of collector?
log created by dns mode.

[jpkrohling]

tracing-opentelemetry-collector.tracing.svc.cluster.local

Is this a headless service? Or a regular service?

[Liubey]

regular service. But in dns mode, parse to get Service IP(10.100.194.91) , not real server IP list.
Maybe cause by some dns server config by my k8s cluster?

[jpkrohling]

A regular service will yield a single and different IP during every DNS query. Take a look at the metrics for the collector, and share the ones you see related to the load balancing.

As mentioned in the documentation, you should use a headless service.

[Liubey]

@jpkrohling thank you for your advice, I will check the document and adjust config.
Thank you so much.

[Liubey]

@jpkrohling by the way, 😂the documentation mention 'headless' on https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter?
I can not find it.
Sorry for that.

[jpkrohling]

Sorry about that! It's buried in another doc. Would you mind adding this to the readme?

https://opentelemetry.io/docs/collector/scaling/#scaling-stateful-collectors

[Liubey]

@jpkrohling OK, my pleasure.

[jpkrohling]

@Liubey , if you added a link to that page, could you please add a link here as well, so that other folks would find it when reading this issue?

[Liubey]

@jpkrohling sorry for my late action.
the pull request is #28687.