feat(opentelemetry): implement unified service tagging #26118

wdhif · 2024-05-30T08:32:05Z

What does this PR do?

OpenTelemetry implements an equivalent to Datadog's Unified Service Tags

We want to support the OpenTelemetry SDK resource attributes the same as Unified Service Tags.

Motivation

This is needed to improve feature parity between Datadog and OpenTelemetry.

Additional notes

Note that OpenTelemetry SDK resource attributes are incompatible with pod/deployment level Unified Service Tags since they are applied at the container level.

Describe how to test/QA your changes

Deploy the following manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy-nginx-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dummy-nginx-app
  template:
    metadata:
      labels:
        app: dummy-nginx-app
        admission.datadoghq.com/enabled: "false"
    spec:
      containers:
      - name: dummy-nginx-app
        image: nginx
        env:
        - name: DD_AGENT_HOST
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: OTEL_SERVICE_NAME
          value: "Service from environment variable"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=Service from resource attributes,service.version=0.0.1,deployment.environment=Development"

When running the tagger-list command, you should see Unified Service Tags for your container in both the Agent and Process Agent.

➜  kubectl exec -it daemonsets/datadog-agent -c agent -- agent tagger-list | grep dummy-nginx-app
=Tags: [kube_deployment:dummy-nginx-app kube_namespace:default kube_ownerref_kind:replicaset kube_ownerref_name:dummy-nginx-app-f8dd44f76 kube_qos:BestEffort kube_replica_set:dummy-nginx-app-f8dd44f76 pod_name:dummy-nginx-app-f8dd44f76-np9q9 pod_phase:running]
=Tags: [container_id:ec02cca65cb13a23d409a4463124e6d8a541d2edae41f30b827c0b80dd5d22a4 container_name:dummy-nginx-app docker_image:nginx:latest env:Development image_id:nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c image_name:nginx image_tag:latest service:Service from environment variable service:Service from resource attributes short_image:nginx version:0.0.1]
=Tags: [container_id:ec02cca65cb13a23d409a4463124e6d8a541d2edae41f30b827c0b80dd5d22a4 display_container_name:dummy-nginx-app_dummy-nginx-app-f8dd44f76-np9q9 env:Development image_id:nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c image_name:nginx image_tag:latest kube_container_name:dummy-nginx-app kube_deployment:dummy-nginx-app kube_namespace:default kube_ownerref_kind:replicaset kube_ownerref_name:dummy-nginx-app-f8dd44f76 kube_qos:BestEffort kube_replica_set:dummy-nginx-app-f8dd44f76 pod_name:dummy-nginx-app-f8dd44f76-np9q9 pod_phase:running service:Service from environment variable service:Service from resource attributes short_image:nginx version:0.0.1]

➜  kubectl exec -it daemonsets/datadog-agent -c process-agent -- process-agent tagger-list | grep dummy-nginx-app
=Tags: [kube_deployment:dummy-nginx-app kube_namespace:default kube_ownerref_kind:replicaset kube_ownerref_name:dummy-nginx-app-f8dd44f76 kube_qos:BestEffort kube_replica_set:dummy-nginx-app-f8dd44f76 pod_name:dummy-nginx-app-f8dd44f76-np9q9 pod_phase:running]
=Tags: [container_id:ec02cca65cb13a23d409a4463124e6d8a541d2edae41f30b827c0b80dd5d22a4 container_name:dummy-nginx-app docker_image:nginx:latest env:Development image_id:nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c image_name:nginx image_tag:latest service:Service from environment variable service:Service from resource attributes short_image:nginx version:0.0.1]
=Tags: [container_id:ec02cca65cb13a23d409a4463124e6d8a541d2edae41f30b827c0b80dd5d22a4 display_container_name:dummy-nginx-app_dummy-nginx-app-f8dd44f76-np9q9 env:Development image_id:nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c image_name:nginx image_tag:latest kube_container_name:dummy-nginx-app kube_deployment:dummy-nginx-app kube_namespace:default kube_ownerref_kind:replicaset kube_ownerref_name:dummy-nginx-app-f8dd44f76 kube_qos:BestEffort kube_replica_set:dummy-nginx-app-f8dd44f76 pod_name:dummy-nginx-app-f8dd44f76-np9q9 pod_phase:running service:Service from environment variable service:Service from resource attributes short_image:nginx version:0.0.1]

You can also see Unified Service Tags on metrics

comp/core/tagger/taggerimpl/collectors/workloadmeta_extract.go

davidor · 2024-05-30T09:06:34Z

We should add a test case for this in workloadmeta_test.go.
Also, the release note is missing.

pr-commenter · 2024-05-30T09:07:30Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=35469451 --os-family=ubuntu

codecov · 2024-05-30T09:15:32Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 50.24%. Comparing base (78e4822) to head (94b9526).
Report is 16 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #26118       +/-   ##
===========================================
+ Coverage   45.08%   50.24%    +5.15%     
===========================================
  Files        2340     1587      -753     
  Lines      269884   137053   -132831     
===========================================
- Hits       121683    68861    -52822     
+ Misses     138579    64292    -74287     
+ Partials     9622     3900     -5722

Flag	Coverage Δ
amzn_aarch64	`50.57% <100.00%> (+4.66%)`	⬆️
centos_x86_64	`50.43% <100.00%> (+4.61%)`	⬆️
ubuntu_aarch64	`50.57% <100.00%> (+4.66%)`	⬆️
ubuntu_x86_64	`50.57% <100.00%> (+4.66%)`	⬆️
windows_amd64	`54.25% <100.00%> (+2.89%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

wdhif · 2024-05-30T09:21:30Z

We should add a test case for this in workloadmeta_test.go. Also, the release note is missing.

Done 👍

comp/core/tagger/taggerimpl/collectors/workloadmeta_test.go

pr-commenter · 2024-05-30T10:59:07Z

Regression Detector

Regression Detector Results

Run ID: 68cd3260-dbf4-466e-be67-0957499d1433
Baseline: 78e4822
Comparison: 1b68351

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	tcp_syslog_to_blackhole	ingress throughput	+6.47	[-15.33, +28.27]
➖	pycheck_1000_100byte_tags	% cpu utilization	+0.79	[-3.68, +5.26]
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+0.58	[-2.35, +3.52]
➖	uds_dogstatsd_to_api	ingress throughput	+0.02	[-0.19, +0.22]
➖	trace_agent_msgpack	ingress throughput	+0.00	[-0.00, +0.00]
➖	otel_to_otel_logs	ingress throughput	-0.01	[-0.37, +0.34]
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.01	[-0.05, +0.02]
➖	trace_agent_json	ingress throughput	-0.02	[-0.04, +0.01]
➖	idle	memory utilization	-0.26	[-0.30, -0.22]
➖	file_tree	memory utilization	-0.63	[-0.73, -0.52]
➖	basic_py_check	% cpu utilization	-3.13	[-5.93, -0.33]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

Signed-off-by: Wassim DHIF <[email protected]>

wdhif · 2024-05-30T14:51:22Z

/merge

dd-devflow · 2024-05-30T14:51:35Z

🚂 MergeQueue

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

maycmlee

Just a couple of suggestions, but approving.

releasenotes/notes/opentelemetry-resource-attributes-ust-e379136c3a5c8046.yaml

dd-devflow · 2024-05-30T14:54:42Z

🚂 MergeQueue

Pull request added to the queue.

There are 2 builds ahead! (estimated merge in less than 1h)

Use /merge -c to cancel this operation!

…36c3a5c8046.yaml Co-authored-by: May Lee <[email protected]>

dd-devflow · 2024-05-30T14:57:06Z

❌ MergeQueue

This PR is rejected because it was updated

If you need support, contact us on Slack #devflow with those details!

wdhif · 2024-05-30T14:57:20Z

/merge

dd-devflow · 2024-05-30T14:57:28Z

🚂 MergeQueue

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

dd-devflow · 2024-05-30T15:46:15Z

🚂 MergeQueue

Pull request added to the queue.

There are 6 builds ahead! (estimated merge in less than 3h)

Use /merge -c to cancel this operation!

wdhif added this to the 7.55.0 milestone May 30, 2024

github-actions bot added the team/container-platform The Container Platform Team label May 30, 2024

wdhif added the team/opentelemetry OpenTelemetry team label May 30, 2024

davidor reviewed May 30, 2024

View reviewed changes

comp/core/tagger/taggerimpl/collectors/workloadmeta_extract.go Show resolved Hide resolved

comp/core/tagger/taggerimpl/collectors/workloadmeta_extract.go Show resolved Hide resolved

wdhif force-pushed the CONTP-218/wassim.dhif/otel-ust branch from 9db91e2 to b8b22fe Compare May 30, 2024 09:20

wdhif force-pushed the CONTP-218/wassim.dhif/otel-ust branch from b8b22fe to f741960 Compare May 30, 2024 09:21

davidor reviewed May 30, 2024

View reviewed changes

comp/core/tagger/taggerimpl/collectors/workloadmeta_test.go Show resolved Hide resolved

feat(opentelemetry): implement unified service tagging

1b68351

Signed-off-by: Wassim DHIF <[email protected]>

wdhif force-pushed the CONTP-218/wassim.dhif/otel-ust branch from 0875218 to 1b68351 Compare May 30, 2024 11:37

wdhif marked this pull request as ready for review May 30, 2024 13:12

wdhif requested review from a team as code owners May 30, 2024 13:12

wdhif requested a review from davidor May 30, 2024 13:17

davidor approved these changes May 30, 2024

View reviewed changes

sblumenthal approved these changes May 30, 2024

View reviewed changes

dineshg13 approved these changes May 30, 2024

View reviewed changes

maycmlee approved these changes May 30, 2024

View reviewed changes

releasenotes/notes/opentelemetry-resource-attributes-ust-e379136c3a5c8046.yaml Outdated Show resolved Hide resolved

Update releasenotes/notes/opentelemetry-resource-attributes-ust-e3791…

94b9526

…36c3a5c8046.yaml Co-authored-by: May Lee <[email protected]>

dineshg13 approved these changes May 30, 2024

View reviewed changes

dd-mergequeue bot merged commit 7afbdee into main May 30, 2024
217 of 221 checks passed

dd-mergequeue bot deleted the CONTP-218/wassim.dhif/otel-ust branch May 30, 2024 16:26

This was referenced Jun 6, 2024

fix(opentelemetry): add OTEL_RESOURCE_ATTRIBUTES spaces trimming #26233

Merged

fix(opentelemetry): add OTEL_RESOURCE_ATTRIBUTES key/pair check #26427

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(opentelemetry): implement unified service tagging #26118

feat(opentelemetry): implement unified service tagging #26118

wdhif commented May 30, 2024 •

edited

Loading

davidor commented May 30, 2024

pr-commenter bot commented May 30, 2024 •

edited

Loading

codecov bot commented May 30, 2024 •

edited

Loading

wdhif commented May 30, 2024

pr-commenter bot commented May 30, 2024 •

edited

Loading

Fine details of change detection per experiment

Explanation

wdhif commented May 30, 2024

dd-devflow bot commented May 30, 2024

maycmlee left a comment

dd-devflow bot commented May 30, 2024

dd-devflow bot commented May 30, 2024

wdhif commented May 30, 2024

dd-devflow bot commented May 30, 2024

dd-devflow bot commented May 30, 2024

feat(opentelemetry): implement unified service tagging #26118

feat(opentelemetry): implement unified service tagging #26118

Conversation

wdhif commented May 30, 2024 • edited Loading

What does this PR do?

Motivation

Additional notes

Describe how to test/QA your changes

davidor commented May 30, 2024

pr-commenter bot commented May 30, 2024 • edited Loading

Test changes on VM

codecov bot commented May 30, 2024 • edited Loading

Codecov Report

wdhif commented May 30, 2024

pr-commenter bot commented May 30, 2024 • edited Loading

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Explanation

wdhif commented May 30, 2024

dd-devflow bot commented May 30, 2024

maycmlee left a comment

Choose a reason for hiding this comment

dd-devflow bot commented May 30, 2024

dd-devflow bot commented May 30, 2024

wdhif commented May 30, 2024

dd-devflow bot commented May 30, 2024

dd-devflow bot commented May 30, 2024

wdhif commented May 30, 2024 •

edited

Loading

pr-commenter bot commented May 30, 2024 •

edited

Loading

codecov bot commented May 30, 2024 •

edited

Loading

pr-commenter bot commented May 30, 2024 •

edited

Loading