datadog exporter ignores attributes #2066

deweyjose · 2022-11-08T20:47:36Z

Describe the bug
The Datadog exporter does not apply attributes configured in Router.yaml to spans. Our team needs to set the version tag at the root of all traces exported to Datadog for Deployment tracking.

To Reproduce
Steps to reproduce the behavior:

  tracing:
    trace_config:
      service_name: "${env.DD_SERVICE:-rusty-router}"
      attributes:
        version: "${env.DD_VERSION:-test}"
    propagation:
      datadog: true

Expected behavior
We expect to all traces to include a version tag at the root of each span named version.

Output
If applicable, add output to help explain your problem.
We do not see the version tag.

Desktop (please complete the following information):

OS: [e.g. iOS]
Version [e.g. 22]

Additional context

As a workaround we added a supergraph plugin that adds a span explicitly for this purpose, but it seems wasteful. It also does not help when Router receives invalid gql request that cause 400s as those short circuit any plugins we right - our version spans will not be applied.

The text was updated successfully, but these errors were encountered:

bnjjj · 2022-11-09T10:17:58Z

related to #1162

BrynCooke · 2022-11-16T09:22:45Z

This is fixed in the otel upgrade branch but we need a new otel release.

lennyburdette · 2023-01-19T16:15:16Z

The fix for the DD exporter is in an unreleased version of open-telemetry-datadog, but if you're building a custom binary you can apply a patch in your Cargo.toml:

router/Cargo.toml

Lines 43 to 49 in 3bb8a67

    
           # TODO: to delete 
        
           # [patch.crates-io] 
        
           # opentelemetry = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"} 
        
           # opentelemetry-http = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"} 
        
           # opentelemetry-jaeger = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"} 
        
           # opentelemetry-zipkin = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"} 
        
           # opentelemetry-datadog = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"}

I confirmed that the patches work.

An alternative is to configure your DD agent to listen for OTLP traces.

# datadog agent config
otlp_config:
  receiver:
    protocols:
      grpc:
      endpoint: 0.0.0.0:4317

# router config
telemetry:
  tracing:
    trace_config:
      attributes:
        version: "2"
        env: "dev"
        test: "42"
    otlp:
      endpoint: 0.0.0.0:4317
      protocol: grpc

abernix · 2023-03-21T08:23:00Z

I'll put the same comment here as I put on #1162 (comment):

This is now waiting on open-telemetry/opentelemetry-rust#965, which we're hoping comes soon.

(We are led to believe it's coming very soon. There's just maybe a patch in there somewhere that might be very relevant for Datadog, specifically.)

abernix · 2023-03-29T07:05:21Z

The fix for this is purportedly is now in https://github.com/open-telemetry/opentelemetry-rust/releases/tag/v0.19.0. I'll see about getting the integration of that upgrade prioritized on our side.

jmvtrinidad · 2023-05-02T12:18:11Z

The fix for the DD exporter is in an unreleased version of open-telemetry-datadog, but if you're building a custom binary you can apply a patch in your Cargo.toml:

router/Cargo.toml

Lines 43 to 49 in 3bb8a67

# TODO: to delete

# [patch.crates-io]

# opentelemetry = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"}

# opentelemetry-http = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"}

# opentelemetry-jaeger = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"}

# opentelemetry-zipkin = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"}

# opentelemetry-datadog = { git = "https://github.com/open-telemetry/opentelemetry-rust.git", rev = "e5ef3552efab2bdbf2f838023c37461cd799ab2c"}

I confirmed that the patches work.

An alternative is to configure your DD agent to listen for OTLP traces.
# datadog agent config
otlp_config:
  receiver:
    protocols:
      grpc:
      endpoint: 0.0.0.0:4317
# router config
telemetry:
  tracing:
    trace_config:
      attributes:
        version: "2"
        env: "dev"
        test: "42"
    otlp:
      endpoint: 0.0.0.0:4317
      protocol: grpc

@lennyburdette do you mean to uncomment the patch.crates-io and build? Thanks

lennyburdette · 2023-05-02T13:33:43Z

Yeah, though this might be pretty out-of-date now so I can't guarantee that it will still work.

abernix · 2023-05-23T18:18:16Z

We are unfortunately still blocked on the OpenTelemetry update because of upstream dependencies. It is perhaps getting close though.

abernix · 2023-05-31T10:05:39Z

Ok, in theory, this is fixed by https://github.com/tokio-rs/tracing-opentelemetry/releases/tag/v0.19.0 and we have an umbrella ticket #2878 that tracks that need (since it unblocks many constraints — like this one — now, at least in theory.) We're proritizing making that update in the next week or two.

garypen · 2023-06-01T16:25:32Z

I have verified that this is fixed in my opentelemetry 0.19.0 PR: #3196

The update requires a change to the implementation and test update as follows: - In otel 0.18.0, processor factories had a `with_memory(bool)` method which we were using when building our prometheus exporter. AFAICT, this used to be a mechanism for controlling how metrics handled stale gauges. In 0.19.0, [this method was removed](open-telemetry/opentelemetry-rust#946) and now gauges are all assumed to be as though they were created with `false`. We had been providing `true` on our call. I'm not 100% certain of the impact of this change, but it appears that we can ignore it. We may need to consider it more carefully if problems arise. - There are now two standard OTEL attributes: ```otel_scope_name="apollo/router",otel_scope_version=""``` added to output and a number of tests had to be updated to accommodate that change. - One of our tests appeared to be searching for `apollo_router_cache_hit_count` (and this was working) when it should have been searching for `apollo_router_cache_hit_count_total` (likewise for miss). I've updated the test and think this is the correct thing to do. It looks like a bug was fixed in otel and this change matches the fix. The upgrade fixes many of the outstanding issues related to opentelemetry and various APM vendors: Fixes: #2878 Fixes: #2066 Fixes: #2959 Fixes: #2225 Fixes: #1520  **Checklist** Complete the checklist (and note appropriate exceptions) before a final PR is raised. - [x] Changes are compatible[^1] - [x] Documentation[^2] completed - [x] Performance impact assessed and acceptable - Tests added and passing[^3] - [x] Unit Tests - [x] Integration Tests - [ ] Manual Tests **Exceptions** *Note any exceptions here* **Notes** [^1]. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. [^2]. Configuration is an important part of many changes. Where applicable please try to document configuration examples. [^3]. Tick whichever testing boxes are applicable. If you are adding Manual Tests: - please document the manual testing (extensively) in the Exceptions. - please raise a separate issue to automate the test and label it (or ask for it to be labeled) as `manual test`

o0Ignition0o · 2023-06-15T11:36:47Z

Reopening the issue since we have to revert the upgrade until they release a patch. See #3242

The update requires a change to the implementation and test update as follows: - In otel 0.18.0, processor factories had a `with_memory(bool)` method which we were using when building our prometheus exporter. AFAICT, this used to be a mechanism for controlling how metrics handled stale gauges. In 0.19.0, [this method was removed](open-telemetry/opentelemetry-rust#946) and now gauges are all assumed to be as though they were created with `false`. We had been providing `true` on our call. I'm not 100% certain of the impact of this change, but it appears that we can ignore it. We may need to consider it more carefully if problems arise. - There are now two standard OTEL attributes: ```otel_scope_name="apollo/router",otel_scope_version=""``` added to output and a number of tests had to be updated to accommodate that change. - One of our tests appeared to be searching for `apollo_router_cache_hit_count` (and this was working) when it should have been searching for `apollo_router_cache_hit_count_total` (likewise for miss). I've updated the test and think this is the correct thing to do. It looks like a bug was fixed in otel and this change matches the fix. Regarding that last point. The prometheus spec mandates naming format and the change was part of the compliance with that spec. This PR made the change: open-telemetry/opentelemetry-rust#952 The two affected counters in the router were: apollo_router_cache_hit_count -> apollo_router_cache_hit_count_total apollo_router_cache_miss_count -> apollo_router_cache_miss_count_total It's good that our prometheus metrics are now spec compliant, but we should note this in the release notes and (if possible) somewhere in our documentation. I'll add it to the changeset at least. The upgrade fixes many of the outstanding issues related to opentelemetry and various APM vendors: Fixes: #2878 Fixes: #2066 Fixes: #2959 Fixes: #2225 Fixes: #1520  **Checklist** Complete the checklist (and note appropriate exceptions) before a final PR is raised. - [x] Changes are compatible[^1] - [x] Documentation[^2] completed - [x] Performance impact assessed and acceptable - Tests added and passing[^3] - [x] Unit Tests - [x] Integration Tests - [ ] Manual Tests **Exceptions** *Note any exceptions here* **Notes** [^1]. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. [^2]. Configuration is an important part of many changes. Where applicable please try to document configuration examples. [^3]. Tick whichever testing boxes are applicable. If you are adding Manual Tests: - please document the manual testing (extensively) in the Exceptions. - please raise a separate issue to automate the test and label it (or ask for it to be labeled) as `manual test`

deweyjose added raised by user triage labels Nov 8, 2022

bnjjj added the bug label Nov 9, 2022

bnjjj mentioned this issue Dec 14, 2022

Unable to send Env tag through configuration for Datadog #1520

Closed

abernix removed the triage label Mar 21, 2023

abernix mentioned this issue Mar 29, 2023

Update OpenTelemetry to 0.19.0 #2878

Closed

garypen self-assigned this Jun 1, 2023

garypen mentioned this issue Jun 1, 2023

update opentelemetry to 0.19.0 #3196

Merged

6 tasks

garypen closed this as completed in #3196 Jun 5, 2023

o0Ignition0o reopened this Jun 15, 2023

garypen mentioned this issue Jul 12, 2023

otel 0.19.0 second try #3421

Merged

6 tasks

garypen closed this as completed in #3421 Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

datadog exporter ignores attributes #2066

datadog exporter ignores attributes #2066

deweyjose commented Nov 8, 2022

bnjjj commented Nov 9, 2022

BrynCooke commented Nov 16, 2022

lennyburdette commented Jan 19, 2023

abernix commented Mar 21, 2023

abernix commented Mar 29, 2023

jmvtrinidad commented May 2, 2023

lennyburdette commented May 2, 2023

abernix commented May 23, 2023 •

edited

Loading

abernix commented May 31, 2023

garypen commented Jun 1, 2023

o0Ignition0o commented Jun 15, 2023

datadog exporter ignores attributes #2066

datadog exporter ignores attributes #2066

Comments

deweyjose commented Nov 8, 2022

bnjjj commented Nov 9, 2022

BrynCooke commented Nov 16, 2022

lennyburdette commented Jan 19, 2023

abernix commented Mar 21, 2023

abernix commented Mar 29, 2023

jmvtrinidad commented May 2, 2023

lennyburdette commented May 2, 2023

abernix commented May 23, 2023 • edited Loading

abernix commented May 31, 2023

garypen commented Jun 1, 2023

o0Ignition0o commented Jun 15, 2023

abernix commented May 23, 2023 •

edited

Loading