ddtrace/tracer: add integration tag to spans_started/finished

[hannahkm]

What does this PR do?

Add a integration tag to the existing datadog.tracer.spans_started and datadog.tracer.spans_finished metrics. The value of the tag will be the name of the component from which the span was started. For example, for a contrib, it will be the name of the contrib package (chi, net/http, etc). For spans that were created manually, the tag will say manual.

Motivation

We want to know, in addition to when a span is started, where the span originated from. This could be a contrib or a manual implementation.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[datadog-datadog-prod-us1]

Datadog Report

Branch report: apm-rd/span-source-health-metric
Commit report: 308cb58
Test service: dd-trace-go

✅ 0 Failed, 5124 Passed, 70 Skipped, 2m 36.53s Total Time

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-12-19 21:50:56

Comparing candidate commit 5cf2e2e in PR branch apm-rd/span-source-health-metric with baseline commit 822d81b in branch main.

Found 0 performance improvements and 20 performance regressions! Performance is the same for 37 metrics, 2 unstable metrics.

scenario:BenchmarkHttpServeTrace-24

• 🟥 allocations [+2; +2] or [+2.532%; +2.532%]

scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24

• 🟥 allocations [+2; +2] or [+6.667%; +6.667%]
• 🟥 execution_time [+197.498ns; +237.102ns] or [+4.190%; +5.030%]

scenario:BenchmarkOTelApiWithCustomTags/otel_api-24

• 🟥 allocations [+2; +2] or [+4.651%; +4.651%]
• 🟥 execution_time [+232.516ns; +286.484ns] or [+3.277%; +4.038%]

scenario:BenchmarkPartialFlushing/Disabled-24

• 🟥 allocations [+200253; +203602] or [+6.122%; +6.224%]
• 🟥 execution_time [+10.373ms; +12.976ms] or [+3.852%; +4.819%]

scenario:BenchmarkPartialFlushing/Enabled-24

• 🟥 allocations [+198073; +200064] or [+6.035%; +6.096%]
• 🟥 execution_time [+11.771ms; +14.672ms] or [+4.307%; +5.368%]

scenario:BenchmarkSingleSpanRetention/no-rules-24

• 🟥 allocations [+202; +202] or [+14.175%; +14.175%]
• 🟥 execution_time [+10.303µs; +12.449µs] or [+4.352%; +5.258%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24

• 🟥 allocations [+202; +202] or [+14.175%; +14.175%]
• 🟥 execution_time [+9.700µs; +12.324µs] or [+4.069%; +5.170%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24

• 🟥 allocations [+202; +202] or [+14.175%; +14.175%]
• 🟥 execution_time [+9.282µs; +11.354µs] or [+3.876%; +4.741%]

scenario:BenchmarkStartSpan-24

• 🟥 allocations [+1; +1] or [+6.250%; +6.250%]
• 🟥 execution_time [+153.600ns; +180.800ns] or [+6.923%; +8.149%]

scenario:BenchmarkStartSpanConcurrent-24

• 🟥 allocations [+10; +10] or [+6.250%; +6.250%]
• 🟥 execution_time [+4.318µs; +4.453µs] or [+80.819%; +83.342%]

scenario:BenchmarkTracerAddSpans-24

• 🟥 allocations [+2; +2] or [+8.000%; +8.000%]

[mtoffl01]

Ok, so you're reporting spansStarted/spansFinished on span.Start/span.Finished if the integration is not empty, and leaving the chunk reporting to any spans that are manual... I understand why you did this but not totally sure about the approach.

span.Start and span.Stop are typically called quite frequently, so if a majority of the spans are from automatic integrations, this will be very noisy (and defeats the purpose of reporting the metrics at a specified interval, to reduce noise)

One alternative idea:
Change the way we track spansStarted and spansFinished to be some kind of counter map that includes the integration name, e.g. map[string]uint32 where the key is the integration name and the value is the count of spans started/finished that integration name. Then, in this goroutine, we'll have to iterate over the map and report the spans started/finished per integration
(or some other idea I haven't thought of?)

[hannahkm]

@mtoffl01 Good points! A map would probably work better; I was hesitant at first since I didn't want to change too much of what already exists, but knowing that these metrics are pretty old... I'm more down to change it up now.