Update version conflict test #6226

andrewlock · 2024-11-01T17:03:59Z

Summary of changes

Change the version conflict test to something sane

Reason for change

Since 3.x, this test doesn't really make sense - it's meant to test the behaviour of the app when the manual version is higher than the automatic version. Since 3.x, we don't ship Datadog.Trace.dll in the NuGet package, so this scenario, where the manual version is higher than the automatic, can't happen in practice.

What we can still have is a manual version 2.x with automatic 3.x, so changed the test to check that scenario instead.

Implementation details

Stop using the magic "future" NuGet and use the existing latest 2.x line.
The behaviour should be the same for both .NET FX and .NET Core now

Test coverage

Kinda the same, kinda different. We're testing auto > manual instead of manual > auto now.

Other details

This is particularly necessarily for supporting .NET 9, seeing as the existing 255.x.x package won't support it, so we would need to create a new one, which isn't worth the hassle seeing as it can't happen.

andrewlock · 2024-11-01T17:35:39Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6226) - mean (71ms)  : 68, 74
     .   : milestone, 71,
    master - mean (71ms)  : 68, 74
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (6226) - mean (1,120ms)  : 1097, 1144
     .   : milestone, 1120,
    master - mean (1,117ms)  : 1097, 1138
     .   : milestone, 1117,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6226) - mean (110ms)  : 107, 113
     .   : milestone, 110,
    master - mean (110ms)  : 105, 115
     .   : milestone, 110,

    section CallTarget+Inlining+NGEN
    This PR (6226) - mean (779ms)  : 760, 798
     .   : milestone, 779,
    master - mean (780ms)  : 764, 796
     .   : milestone, 780,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6226) - mean (93ms)  : 91, 95
     .   : milestone, 93,
    master - mean (93ms)  : 91, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (6226) - mean (735ms)  : 718, 752
     .   : milestone, 735,
    master - mean (734ms)  : 721, 747
     .   : milestone, 734,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6226) - mean (190ms)  : 186, 193
     .   : milestone, 190,
    master - mean (190ms)  : 186, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6226) - mean (1,219ms)  : 1196, 1243
     .   : milestone, 1219,
    master - mean (1,221ms)  : 1199, 1243
     .   : milestone, 1221,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6226) - mean (276ms)  : 271, 281
     .   : milestone, 276,
    master - mean (277ms)  : 270, 284
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6226) - mean (942ms)  : 925, 960
     .   : milestone, 942,
    master - mean (949ms)  : 929, 969
     .   : milestone, 949,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6226) - mean (264ms)  : 260, 268
     .   : milestone, 264,
    master - mean (264ms)  : 261, 268
     .   : milestone, 264,

    section CallTarget+Inlining+NGEN
    This PR (6226) - mean (928ms)  : 909, 947
     .   : milestone, 928,
    master - mean (932ms)  : 911, 952
     .   : milestone, 932,

datadog-ddstaging · 2024-11-01T17:36:17Z

Datadog Report

Branch report: andrew/update-version-conflict-tests
Commit report: 83d902b
Test service: dd-trace-dotnet

✅ 0 Failed, 367186 Passed, 2084 Skipped, 15h 2m 32.41s Total Time

andrewlock · 2024-11-01T18:06:46Z

Benchmarks Report for tracer 🐌

Benchmarks for #6226 compared to master:

1 benchmarks are faster, with geometric mean 1.344
1 benchmarks are slower, with geometric mean 1.158
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.34μs	47.2ns	343ns	0.0168	0.00841	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	56.4ns	339ns	0.0147	0.00491	0	5.8 KB
master	`StartStopWithChild`	net472	16.3μs	44.3ns	172ns	1.05	0.31	0.106	6.21 KB
#6226	`StartStopWithChild`	net6.0	8.15μs	45.3ns	290ns	0.0207	0.00828	0	5.61 KB
#6226	`StartStopWithChild`	netcoreapp3.1	10μs	50.5ns	220ns	0.0242	0.00968	0	5.8 KB
#6226	`StartStopWithChild`	net472	16.3μs	40.1ns	155ns	1.05	0.314	0.0965	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	463μs	414ns	1.55μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	641μs	482ns	1.87μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	839μs	410ns	1.53μs	0.417	3.3 KB
#6226	`WriteAndFlushEnrichedTraces`	net6.0	484μs	654ns	2.45μs	0	2.7 KB
#6226	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	636μs	173ns	647ns	0	2.7 KB
#6226	`WriteAndFlushEnrichedTraces`	net472	845μs	545ns	2.11μs	0.419	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	216μs	1.25μs	10.7μs	0.202	18.73 KB
master	`SendRequest`	netcoreapp3.1	240μs	1.39μs	11.6μs	0.244	20.89 KB
master	`SendRequest`	net472	0.000262ns	0.000152ns	0.000568ns	0	0 b
#6226	`SendRequest`	net6.0	216μs	1.2μs	11μs	0.217	18.73 KB
#6226	`SendRequest`	netcoreapp3.1	237μs	1.36μs	11.1μs	0.227	20.89 KB
#6226	`SendRequest`	net472	0.00154ns	0.000549ns	0.00213ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	584μs	3.32μs	23μs	0.548	0	0	41.61 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	711μs	3.99μs	25.6μs	0.347	0	0	41.94 KB
master	`WriteAndFlushEnrichedTraces`	net472	904μs	4.52μs	20.2μs	8.8	2.64	0.44	53.34 KB
#6226	`WriteAndFlushEnrichedTraces`	net6.0	582μs	3.1μs	17.2μs	0.568	0	0	41.61 KB
#6226	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	712μs	3.67μs	17.6μs	0.357	0	0	41.84 KB
#6226	`WriteAndFlushEnrichedTraces`	net472	867μs	4μs	16μs	8.36	2.64	0.44	53.3 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.18μs	1.83ns	7.08ns	0.0143	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.72μs	1.44ns	5.56ns	0.0136	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.12μs	1.5ns	5.6ns	0.156	0.00106	987 B
#6226	`ExecuteNonQuery`	net6.0	1.2μs	0.938ns	3.38ns	0.0144	0	1.02 KB
#6226	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.55ns	5.99ns	0.0132	0	1.02 KB
#6226	`ExecuteNonQuery`	net472	2.11μs	1.89ns	7.07ns	0.156	0.00105	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.28μs	0.688ns	2.66ns	0.0136	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.52μs	0.844ns	3.27ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.56μs	2.05ns	7.92ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.22μs	0.62ns	2.32ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.58μs	0.676ns	2.53ns	0.0135	1.02 KB
master	`CallElasticsearchAsync`	net472	2.56μs	0.557ns	1.93ns	0.167	1.05 KB
#6226	`CallElasticsearch`	net6.0	1.18μs	2.03ns	7.58ns	0.0135	976 B
#6226	`CallElasticsearch`	netcoreapp3.1	1.52μs	4.04ns	15.7ns	0.013	976 B
#6226	`CallElasticsearch`	net472	2.69μs	4.76ns	18.5ns	0.158	995 B
#6226	`CallElasticsearchAsync`	net6.0	1.27μs	0.859ns	3.33ns	0.0133	952 B
#6226	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	0.73ns	2.53ns	0.0138	1.02 KB
#6226	`CallElasticsearchAsync`	net472	2.66μs	3.87ns	15ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.27μs	4.54ns	17.6ns	0.0132	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.58μs	0.908ns	3.4ns	0.0127	952 B
master	`ExecuteAsync`	net472	1.73μs	0.322ns	1.21ns	0.145	915 B
#6226	`ExecuteAsync`	net6.0	1.35μs	0.722ns	2.7ns	0.0129	952 B
#6226	`ExecuteAsync`	netcoreapp3.1	1.61μs	0.686ns	2.57ns	0.0129	952 B
#6226	`ExecuteAsync`	net472	1.8μs	0.401ns	1.5ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.24μs	1.86ns	6.97ns	0.0319	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.23μs	2.11ns	8.17ns	0.0367	2.85 KB
master	`SendAsync`	net472	7.38μs	1.28ns	4.96ns	0.494	3.12 KB
#6226	`SendAsync`	net6.0	4.36μs	2.4ns	9.28ns	0.0325	2.31 KB
#6226	`SendAsync`	netcoreapp3.1	5.29μs	1.79ns	6.94ns	0.0371	2.85 KB
#6226	`SendAsync`	net472	7.45μs	4.07ns	15.8ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.57μs	1.11ns	3.86ns	0.0231	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.29μs	1.36ns	5.27ns	0.0218	1.64 KB
master	`EnrichedLog`	net472	2.7μs	2.73ns	10.6ns	0.25	1.57 KB
#6226	`EnrichedLog`	net6.0	1.47μs	1.71ns	5.92ns	0.023	1.64 KB
#6226	`EnrichedLog`	netcoreapp3.1	2.25μs	3.19ns	11.9ns	0.022	1.64 KB
#6226	`EnrichedLog`	net472	2.69μs	5.06ns	19.6ns	0.25	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	130ns	485ns	0.0578	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	122μs	191ns	739ns	0	0	4.28 KB
master	`EnrichedLog`	net472	149μs	161ns	624ns	0.668	0.223	4.46 KB
#6226	`EnrichedLog`	net6.0	117μs	212ns	792ns	0	0	4.28 KB
#6226	`EnrichedLog`	netcoreapp3.1	123μs	361ns	1.35μs	0	0	4.28 KB
#6226	`EnrichedLog`	net472	153μs	102ns	395ns	0.685	0.228	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3μs	0.746ns	2.89ns	0.0299	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.25μs	1.34ns	4.81ns	0.0298	2.2 KB
master	`EnrichedLog`	net472	4.87μs	2ns	7.75ns	0.321	2.02 KB
#6226	`EnrichedLog`	net6.0	3.09μs	1.51ns	5.66ns	0.0311	2.2 KB
#6226	`EnrichedLog`	netcoreapp3.1	4.08μs	0.926ns	3.59ns	0.0304	2.2 KB
#6226	`EnrichedLog`	net472	4.82μs	1.3ns	5.02ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.37μs	0.811ns	3.03ns	0.0164	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.74μs	1.08ns	3.75ns	0.0155	1.14 KB
master	`SendReceive`	net472	2.14μs	1.5ns	5.8ns	0.183	1.16 KB
#6226	`SendReceive`	net6.0	1.42μs	0.761ns	2.95ns	0.0163	1.14 KB
#6226	`SendReceive`	netcoreapp3.1	1.85μs	1.06ns	4.11ns	0.0157	1.14 KB
#6226	`SendReceive`	net472	2.09μs	0.762ns	2.85ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.72μs	0.873ns	3.27ns	0.0218	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.81μs	2.11ns	8.16ns	0.0209	1.65 KB
master	`EnrichedLog`	net472	4.33μs	3.1ns	12ns	0.324	2.04 KB
#6226	`EnrichedLog`	net6.0	2.71μs	0.759ns	2.94ns	0.0217	1.6 KB
#6226	`EnrichedLog`	netcoreapp3.1	3.91μs	1.57ns	5.66ns	0.0217	1.65 KB
#6226	`EnrichedLog`	net472	4.31μs	2.15ns	8.34ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6226

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.344	644.21	479.39

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	408ns	0.297ns	1.15ns	0.00812	576 B
master	`StartFinishSpan`	netcoreapp3.1	566ns	0.468ns	1.69ns	0.00776	576 B
master	`StartFinishSpan`	net472	736ns	0.851ns	3.3ns	0.0918	578 B
master	`StartFinishScope`	net6.0	644ns	0.427ns	1.65ns	0.00972	696 B
master	`StartFinishScope`	netcoreapp3.1	703ns	0.348ns	1.3ns	0.00939	696 B
master	`StartFinishScope`	net472	868ns	0.541ns	2.1ns	0.104	658 B
#6226	`StartFinishSpan`	net6.0	399ns	0.185ns	0.691ns	0.00804	576 B
#6226	`StartFinishSpan`	netcoreapp3.1	576ns	0.252ns	0.942ns	0.00755	576 B
#6226	`StartFinishSpan`	net472	696ns	0.262ns	0.943ns	0.0918	578 B
#6226	`StartFinishScope`	net6.0	479ns	0.113ns	0.421ns	0.00964	696 B
#6226	`StartFinishScope`	netcoreapp3.1	768ns	0.294ns	1.1ns	0.0092	696 B
#6226	`StartFinishScope`	net472	914ns	0.658ns	2.55ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6226

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.158	649.81	752.23

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	650ns	0.213ns	0.795ns	0.0097	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	998ns	0.185ns	0.669ns	0.00899	696 B
master	`RunOnMethodBegin`	net472	1.19μs	0.613ns	2.37ns	0.104	658 B
#6226	`RunOnMethodBegin`	net6.0	752ns	0.416ns	1.61ns	0.00979	696 B
#6226	`RunOnMethodBegin`	netcoreapp3.1	979ns	0.441ns	1.65ns	0.00932	696 B
#6226	`RunOnMethodBegin`	net472	1.19μs	0.95ns	3.56ns	0.104	658 B

lucaspimentel · 2024-11-01T18:07:32Z

This is particularly necessarily for supporting .NET, seeing as the existing 255.x.x package won't support it

Did you mean ".NET 9"?

Update version conflict behaviour

83d902b

andrewlock added the area:tests unit tests, integration tests label Nov 1, 2024

andrewlock requested review from a team as code owners November 1, 2024 17:04

lucaspimentel approved these changes Nov 1, 2024

View reviewed changes

kevingosse approved these changes Nov 4, 2024

View reviewed changes

andrewlock merged commit bf5333d into master Nov 4, 2024
76 of 77 checks passed

andrewlock deleted the andrew/update-version-conflict-tests branch November 4, 2024 13:19

github-actions bot added this to the vNext-v3 milestone Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update version conflict test #6226

Update version conflict test #6226

andrewlock commented Nov 1, 2024 •

edited

Loading

andrewlock commented Nov 1, 2024

datadog-ddstaging bot commented Nov 1, 2024 •

edited

Loading

andrewlock commented Nov 1, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6226

Raw results

Slower ⚠️ in #6226

Raw results

lucaspimentel commented Nov 1, 2024

Update version conflict test #6226

Update version conflict test #6226

Conversation

andrewlock commented Nov 1, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Nov 1, 2024

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Nov 1, 2024 • edited Loading

Datadog Report

andrewlock commented Nov 1, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6226

Raw results

Slower ⚠️ in #6226

Raw results

lucaspimentel commented Nov 1, 2024

andrewlock commented Nov 1, 2024 •

edited

Loading

datadog-ddstaging bot commented Nov 1, 2024 •

edited

Loading