[exporter/sapm] Token pass-through not working

[strk1204]

Component(s)

exporter/sapm

What happened?

Description.

When using the following setup, tokens provided via the downstream agents, over OTLP, do not get passed on to SOC(SignalFX), resulting in a 401 being returned at the gateway/collector level.

Example Setup:

Java Application -> Java (Splunk Dist) OTel Agent -> (Splunk Dist) OTel Collector -> SplunkOC

The Java OTel agent is configured with the collector as both the SignalFX (for metrics) and OTLP (for traces) endpoints. A valid token is provided, and in this configuration, metrics are being gathered from the application successfully. The collector is configured with the correct setup to forward this data to SOC. The collector/exporter level token, is "NULL", with access_token_passthrough enabled. include_metadata: true has also been enabled for the OTLP receiver, to ensure that all values are being retained before passing to the SAPM exporter. It is expected that, as there is a valid token in the payload, it will use this instead of the exporters token, as other exporters behave.

      sapm:
        access_token: "NULL"
        access_token_passthrough: true
        disable_compression: false # Force enable GZIP
        endpoint: "https://ingest.${SPLUNK_O11Y_REALM}.signalfx.com/v2/trace"
        sending_queue:
          num_consumers: 32

I can confirm that the token IS being passed to the collector, via the "X-SF-TOKEN" header, however this is not being used by the SAPM exporter.

As a work around, I've added the following config, which has enabled this setup to work correctly.

        actions:
          - key: "com.splunk.signalfx.access_token"
            from_context: "x-sf-token"
            action: insert
      transform/retain-sfx-token:
        error_mode: silent
        trace_statements:
          - context: span
            statements:
              - set(resource.attributes["com.splunk.signalfx.access_token"], attributes["com.splunk.signalfx.access_token"])
              - delete_key(attributes, "com.splunk.signalfx.access_token")

I also can't use OTLP -> OTLPHTTP due to headers not carrying over, and the lack of the headers_setter ext in the Splunk dist of the collector; IDEA

I'm unsure if here, or the agent level itself is the best place to raise this, however given that it appears that the agents are exporting the tokens as expected, in-line with the Splunk provided documentation, the fault lies in the handling of the token, within the headers, by the SAPM exporter.

Steps to Reproduce

1. Deploy example application with OTel agent
2. Export OTLP metrics, with valid token, to a collector, configured as outlined above.
3. Check logs
4. Export fail.

Expected Result

Token passed through correctly to SOC, auth completed and traces ingested.

Actual Result

401 returns on traces.

P.S, all references to agent above, refer to an application agent, NOT a collector deployed in, what Splunk coin, "agent mode".

Collector version

0.100.0

Environment information

Environment

Splunk Distribution - OTel Collector - 0.100.0 - Deployed via Helm to Kubernetes. Configuration as outlined.

OpenTelemetry Collector configuration

...
    receivers:
      otlp: # Metrics, Traces, Logs
        protocols:
          http:
            endpoint: "${SPLUNK_LISTEN_INTERFACE}:4318"
            include_metadata: true
      sapm: # Metrics & Traces
        endpoint: "${SPLUNK_LISTEN_INTERFACE}:7276"
        access_token_passthrough: true
...
    processors:
      attributes/retain-sfx-token:
        actions:
          - key: "com.splunk.signalfx.access_token"
            from_context: "x-sf-token"
            action: insert
      transform/retain-sfx-token:
        error_mode: silent
        trace_statements:
          - context: span
            statements:
              - set(resource.attributes["com.splunk.signalfx.access_token"], attributes["com.splunk.signalfx.access_token"])
              - delete_key(attributes, "com.splunk.signalfx.access_token")
...
    exporters:
      sapm:
        access_token: "NULL"
        access_token_passthrough: true
        disable_compression: false # Force enable GZIP
        endpoint: "https://ingest.${SPLUNK_O11Y_REALM}.signalfx.com/v2/trace"
        sending_queue:
          num_consumers: 32
...
pipelines:
        traces:
          receivers: [sapm, otlp]
          processors:
            - memory_limiter
            # - batch # Disable due to breaking this fix
            ## The following are required to handle OTLP Tokens
            - attributes/retain-sfx-token
            - transform/retain-sfx-token
          exporters: [sapm]
...

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/sapm: @dmitryax @atoulme

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

I have a working example here that is a bit simpler since we use the resource processor.
Here is the docker-compose.yml to generate this:

version: "3"
services:
  telemetrygen:
    image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest
    container_name: telemetrygen
    command:
      - "traces"
      - "--otlp-endpoint"
      - "otelcollector:4317"
      - "--otlp-insecure"
      - "--duration"
      - "10m"
      - "--rate"
      - "1000"
      - "--workers"
      - "3"
      - "--otlp-attributes"
      - 'deployment.environment="testpassthrough"'
      - "--otlp-header"
      - Content-Type="application/x-protobuf"
      - "--otlp-header"
      - X-SF-Token="<REPLACE WITH YOUR TOKEN>"
    depends_on:
      - otelcollector
  # OpenTelemetry Collector
  otelcollector:
    image:  quay.io/signalfx/splunk-otel-collector:latest
    container_name: otelcollector
    command: ["--config=/etc/otel-collector-config.yml", "--set=service.telemetry.logs.level=debug"]
    volumes:
      - ./otel-collector-config.yml:/etc/otel-collector-config.yml

Here is the collector configuration:

receivers:
    otlp:
      protocols:
        http:
          include_metadata: true
          endpoint: 0.0.0.0:4318
        grpc:
          include_metadata: true
          endpoint: 0.0.0.0:4317

exporters:
    sapm:
      access_token: "NULL"
      access_token_passthrough: true
      disable_compression: false # Force enable GZIP
      endpoint: "https://ingest.us0.signalfx.com/v2/trace"
      sending_queue:
        num_consumers: 32

processors:
    batch:
    resource:
      attributes:
        - action: insert
          from_context: X-SF-Token
          key: com.splunk.signalfx.access_token

extensions:
    health_check:
      endpoint: 0.0.0.0:13133
    pprof:
      endpoint: :1888
    zpages:
      endpoint: :55679

service:
    extensions: [pprof, zpages, health_check]
    pipelines:
      traces:
        receivers: [otlp]
        processors: [resource]
        exporters: [sapm]

From what I understand, you expect the collector to automatically extract the token from the request headers and set if on the response. That's not possible out of the box. You need to do the extraction and mapping to a resource attribute yourself.

[strk1204]

Yes that's correct, that was the underlying expectation here.

Is there any plan on adding this functionality to the SAPM exporter?

In-light of posts relating to the possible deprecation of SAPM in favour of OTLP, assuming we retain include_metadata, can we expect traces OTLP -> OTLP to retain X-SF-TOKEN, and this be accepted by the appropriate OTLP endpoint on the signalfx platform? Thus this would remove the need for this work around, should the behavior work as expected.

Thanks!

[atoulme]

No, the same approach is taken. We cannot pass around tokens implicitly or we might be disclosing secrets.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/sapm: @dmitryax @atoulme

See Adding Labels via Comments if you do not have permissions to add labels yourself.