APMSP-1241 Directly import trace-agent stats code for client-side stats #2817

ajgajg1134 · 2024-08-13T20:15:59Z

WIP: This PR should be updated to point to the released v0.58 version of the trace-agent when it is available and then should be ready for merge. However this code is very unlikely to change other than changing the go.mod version so it should be safe to review now with minimal disruption.

What does this PR do?

Begin using the trace-agent's Concentrator instead of our own (mostly copied) implementation from the trace-agent.

Unfortunately there's some additional changes here needed like updating to go 1.22 as that's the minimum supported version of the trace-agent (but this is aligned with dd-trace-go's published version support policy). Additionally the google.golang.org/api contrib needed to be updated to support the more recent versions required by the trace-agent. I believe these imports / upgrades are ok but if we need to target a smaller import and modularize some more of the trace-agent to reduce impact here just let me know!

Motivation

Some new features introduced in the trace-agent concentrator have been missed here over time and this brings back feature parity when using client side stats (e.g. peer tags aggregation, marking trace root spans). We will eventually be enabling client stats by default (as it's more efficient!) and this is a required step to ensure that we don't break important stats features.

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
Add an appropriate team label so this PR gets put in the right place for the release notes.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

datadog-datadog-prod-us1 · 2024-08-13T20:16:35Z

go.mod

-	google.golang.org/grpc v1.57.1
-	google.golang.org/protobuf v1.33.0
+	google.golang.org/api v0.169.0
+	google.golang.org/grpc v1.64.0


🟡 Library Vulnerability

google.golang.org/grpc → 1.64.0

Private tokens could appear in logs if context containing gRPC metadata is logged in github.com/grpc/grpc-go (...read more)

Impact

This issue represents a potential PII concern. If applications were printing or logging a context containing gRPC metadata, the affected versions will contain all the metadata, which may include private information.

Patches

The issue first appeared in 1.64.0 and is patched in 1.64.1 and 1.65.0

Workarounds

If using an affected version and upgrading is not possible, ensuring you do not log or print contexts will avoid the problem.

datadog-datadog-prod-us1 · 2024-08-13T20:16:36Z

go.mod

-	google.golang.org/grpc v1.57.1
-	google.golang.org/protobuf v1.33.0
+	google.golang.org/api v0.169.0
+	google.golang.org/grpc v1.64.0


🟡 Library Vulnerability

google.golang.org/grpc → 1.64.0

Private tokens could appear in logs if context containing gRPC metadata is logged in github.com/grpc/grpc-go (...read more)

Impact

This issue represents a potential PII concern. If applications were printing or logging a context containing gRPC metadata, the affected versions will contain all the metadata, which may include private information.

Patches

The issue first appeared in 1.64.0 and is patched in 1.64.1 and 1.65.0

Workarounds

If using an affected version and upgrading is not possible, ensuring you do not log or print contexts will avoid the problem.

pr-commenter · 2024-08-13T20:53:16Z

Benchmarks

Benchmark execution time: 2024-10-31 09:31:26

Comparing candidate commit 9c2328e in PR branch andrew.glaude/spanConcentrator2 with baseline commit 0bd0a8c in branch main.

Found 4 performance improvements and 1 performance regressions! Performance is the same for 53 metrics, 1 unstable metrics.

scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24

🟩 execution_time [-197.789ns; -162.611ns] or [-4.095%; -3.367%]

scenario:BenchmarkOTelApiWithCustomTags/otel_api-24

🟩 allocations [-1; -1] or [-2.273%; -2.273%]
🟩 execution_time [-407.377ns; -351.423ns] or [-5.529%; -4.770%]

scenario:BenchmarkSetTagStringer-24

🟥 execution_time [+4.686ns; +8.854ns] or [+3.371%; +6.370%]

scenario:BenchmarkStartSpanConcurrent-24

🟩 execution_time [-2.405µs; -2.256µs] or [-31.744%; -29.774%]

datadog-datadog-prod-us1 · 2024-08-14T14:57:58Z

internal/apps/go.mod

+	golang.org/x/text v0.16.0 // indirect
+	golang.org/x/tools v0.23.0 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240520151616-dc85e6b867a5 // indirect
+	google.golang.org/grpc v1.64.0 // indirect


🟡 Library Vulnerability

google.golang.org/grpc → 1.64.0

Private tokens could appear in logs if context containing gRPC metadata is logged in github.com/grpc/grpc-go (...read more)

Impact

This issue represents a potential PII concern. If applications were printing or logging a context containing gRPC metadata, the affected versions will contain all the metadata, which may include private information.

Patches

The issue first appeared in 1.64.0 and is patched in 1.64.1 and 1.65.0

Workarounds

If using an affected version and upgrading is not possible, ensuring you do not log or print contexts will avoid the problem.

datadog-datadog-prod-us1 · 2024-08-14T14:57:59Z

internal/exectracetest/go.mod

 	golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
-	google.golang.org/protobuf v1.33.0 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240520151616-dc85e6b867a5 // indirect
+	google.golang.org/grpc v1.64.0 // indirect


🟡 Library Vulnerability

google.golang.org/grpc → 1.64.0

Private tokens could appear in logs if context containing gRPC metadata is logged in github.com/grpc/grpc-go (...read more)

Impact

This issue represents a potential PII concern. If applications were printing or logging a context containing gRPC metadata, the affected versions will contain all the metadata, which may include private information.

Patches

The issue first appeared in 1.64.0 and is patched in 1.64.1 and 1.65.0

Workarounds

If using an affected version and upgrading is not possible, ensuring you do not log or print contexts will avoid the problem.

…an agent env

…agent The dd-agent and google.golang.org/api both rely on google.golang.org/grpc, the old version of the api package forces a downgrade of the datadog-agent which breaks the build.

This reverts commit a279fd2.

This reverts commit 9e135c8.

eliottness

Approving on behalf of appsec only for appsec code. Thanks for starting out the migration to rolling out go 1.21 👍

go.mod

github-actions · 2024-09-26T01:57:43Z

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

darccio · 2024-10-09T19:18:03Z

@ajgajg1134 Was the agent released? When will this able to be merged?

ajgajg1134 · 2024-10-16T15:03:50Z

@ajgajg1134 Was the agent released? When will this able to be merged?

The current ETA for agent release is TOMORROW! 🎉

initial support for agent SpanConcentrator wip

dc418a5

datadog-datadog-prod-us1 bot reviewed Aug 13, 2024

View reviewed changes

ajgajg1134 added 2 commits August 14, 2024 10:34

Add fallback env for stats to avoid panic

63bf1de

go mod tidy

cd72859

datadog-datadog-prod-us1 bot reviewed Aug 14, 2024

View reviewed changes

ajgajg1134 added 6 commits August 14, 2024 13:38

add testEnv for tests asserting on logs, in real usage there will be …

b81bc9c

…an agent env

bump k8s client go to match k8s api version

e187e48

add some temp logs

704acc7

more logs

21aa1ce

bump to dev version of agent to fix parametric tests (temporary)

e3796bd

Clone net http request to avoid race

cfe668d

ajgajg1134 force-pushed the andrew.glaude/spanConcentrator2 branch from ed5d824 to cfe668d Compare August 15, 2024 18:04

ajgajg1134 added 3 commits August 15, 2024 14:25

stats need accurate origin for synthetics

c64efa3

Bump google.golang.org/api to newer version to avoid downgrade of dd-…

0520899

…agent The dd-agent and google.golang.org/api both rely on google.golang.org/grpc, the old version of the api package forces a downgrade of the datadog-agent which breaks the build.

bump go minimum version to 1.22

a279fd2

ajgajg1134 added the team:apm-go label Aug 16, 2024

ajgajg1134 added 8 commits August 16, 2024 14:11

Merge branch 'main' into andrew.glaude/spanConcentrator2

8d6425e

Merge branch 'main' into andrew.glaude/spanConcentrator2

04aa447

Merge branch 'main' into andrew.glaude/spanConcentrator2

521f7d4

Revert "bump go minimum version to 1.22"

9e135c8

This reverts commit a279fd2.

Revert "Revert "bump go minimum version to 1.22""

4de2dfd

This reverts commit 9e135c8.

tidy up a bit

50f6db6

more cleanup

68e056f

Merge branch 'main' into andrew.glaude/spanConcentrator2

cb851e8

github-actions bot added the apm:ecosystem contrib/* related feature requests or bugs label Sep 4, 2024

ajgajg1134 changed the title ~~initial support for agent SpanConcentrator WIP~~ APMSP-1241 Directly import trace-agent stats code for client-side stats [Waiting Agent Release] Sep 4, 2024

ajgajg1134 marked this pull request as ready for review September 4, 2024 19:16

ajgajg1134 requested a review from a team as a code owner September 4, 2024 19:16

ajgajg1134 requested review from a team as code owners September 4, 2024 19:16

copyright job needs go installed

1ca882e

eliottness approved these changes Sep 5, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

no go mod toolchain

bc30973

ajgajg1134 mentioned this pull request Sep 11, 2024

Regenerate stats payload to add is_trace_root #2856

Merged

6 tasks

github-actions bot added the stale Stuck for more than 1 month label Sep 26, 2024

ajgajg1134 removed the stale Stuck for more than 1 month label Sep 26, 2024

Merge branch 'main' into andrew.glaude/spanConcentrator2

bb54dd9

ajgajg1134 changed the title ~~APMSP-1241 Directly import trace-agent stats code for client-side stats [Waiting Agent Release]~~ APMSP-1241 Directly import trace-agent stats code for client-side stats Oct 22, 2024

ajgajg1134 and others added 3 commits October 22, 2024 11:14

Merge branch 'main' into andrew.glaude/spanConcentrator2

fc53ad9

ignore seelog goroutine leak

2bd6411

Merge branch 'main' into andrew.glaude/spanConcentrator2

117a018

darccio approved these changes Oct 30, 2024

View reviewed changes

ajgajg1134 and others added 2 commits October 30, 2024 10:49

Merge branch 'main' into andrew.glaude/spanConcentrator2

13a3305

Merge branch 'main' into andrew.glaude/spanConcentrator2

9c2328e

darccio enabled auto-merge (squash) October 31, 2024 10:11

Merge branch 'main' into andrew.glaude/spanConcentrator2

3cbae65

darccio merged commit c16be07 into main Oct 31, 2024
170 of 171 checks passed

darccio deleted the andrew.glaude/spanConcentrator2 branch October 31, 2024 10:16

darccio mentioned this pull request Nov 14, 2024

chore(go.mod): reduce blast radius in go.mod from #2817 #2979

Merged

6 tasks

darccio added a commit that referenced this pull request Nov 14, 2024

chore(go.mod): reduce blast radius in go.mod from #2817 (#2979)

0698107

darccio added a commit that referenced this pull request Nov 14, 2024

chore(go.mod): reduce blast radius in go.mod from #2817 (#2979)

432060b

darccio mentioned this pull request Nov 14, 2024

chore(go.mod): reduce blast radius in go.mod from #2817 (#2979) #2980

Merged

6 tasks

felixge mentioned this pull request Nov 26, 2024

[BUG] unexpected goroutines remain in service after upgrading from 1.69.1 to 1.70.1 #2987

Open

darccio added a commit that referenced this pull request Nov 29, 2024

chore(go.mod): reduce blast radius in go.mod from #2817 (#2979) (#2980)

697219e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

APMSP-1241 Directly import trace-agent stats code for client-side stats #2817

APMSP-1241 Directly import trace-agent stats code for client-side stats #2817

ajgajg1134 commented Aug 13, 2024 •

edited

Loading

datadog-datadog-prod-us1 bot Aug 13, 2024

datadog-datadog-prod-us1 bot Aug 13, 2024

pr-commenter bot commented Aug 13, 2024 •

edited

Loading

datadog-datadog-prod-us1 bot Aug 14, 2024

datadog-datadog-prod-us1 bot Aug 14, 2024

eliottness left a comment

github-actions bot commented Sep 26, 2024

darccio commented Oct 9, 2024

ajgajg1134 commented Oct 16, 2024

APMSP-1241 Directly import trace-agent stats code for client-side stats #2817

APMSP-1241 Directly import trace-agent stats code for client-side stats #2817

Conversation

ajgajg1134 commented Aug 13, 2024 • edited Loading

What does this PR do?

Motivation

Reviewer's Checklist

datadog-datadog-prod-us1 bot Aug 13, 2024

Choose a reason for hiding this comment

🟡 Library Vulnerability

Impact

Patches

Workarounds

datadog-datadog-prod-us1 bot Aug 13, 2024

Choose a reason for hiding this comment

🟡 Library Vulnerability

Impact

Patches

Workarounds

pr-commenter bot commented Aug 13, 2024 • edited Loading

Benchmarks

scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24

scenario:BenchmarkOTelApiWithCustomTags/otel_api-24

scenario:BenchmarkSetTagStringer-24

scenario:BenchmarkStartSpanConcurrent-24

datadog-datadog-prod-us1 bot Aug 14, 2024

Choose a reason for hiding this comment

🟡 Library Vulnerability

Impact

Patches

Workarounds

datadog-datadog-prod-us1 bot Aug 14, 2024

Choose a reason for hiding this comment

🟡 Library Vulnerability

Impact

Patches

Workarounds

eliottness left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 26, 2024

darccio commented Oct 9, 2024

ajgajg1134 commented Oct 16, 2024

ajgajg1134 commented Aug 13, 2024 •

edited

Loading

pr-commenter bot commented Aug 13, 2024 •

edited

Loading