[ASM] Fix TestExternalWafHeaders snapshot netcore 2.1 #6202

NachoEchevarria · 2024-10-25T10:25:46Z

Summary of changes

This PR fixes a snapshots that was not updated with the event rules tag for netcore 2.1

Reason for change

Implementation details

Test coverage

Other details

github-actions · 2024-10-25T10:26:33Z

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

1 occurrences of :

+      _dd.appsec.event_rules.version: 1.13.1,

datadog-ddstaging · 2024-10-25T10:47:42Z

Datadog Report

Branch report: nacho/Fix2.1SnapshotsRulesVersion
Commit report: c911b26
Test service: dd-trace-dotnet

❌ 3 Failed (0 Known Flaky), 370074 Passed, 2123 Skipped, 24h 7m 51.74s Total Time

❌ Failed Tests (3)

AzureFunctionsTests+InProcessRuntimeV4.SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests - Details
Expand for error
```
 The sample did not exit in 600000ms. Memory dump taken: True. Killing process.
```
AzureFunctionsTests+IsolatedRuntimeV4.SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests - Details
Expand for error
```
 The sample did not exit in 600000ms. Memory dump taken: True. Killing process.
```
AzureFunctionsTests+IsolatedRuntimeV4.SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests - Details
Expand for error
```
 Expected exit code: -1, actual exit code: 1.
```

andrewlock · 2024-10-25T10:56:47Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6202) - mean (70ms)  : 67, 73
     .   : milestone, 70,
    master - mean (70ms)  : 68, 73
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6202) - mean (1,114ms)  : 1091, 1137
     .   : milestone, 1114,
    master - mean (1,114ms)  : 1092, 1135
     .   : milestone, 1114,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6202) - mean (109ms)  : 106, 112
     .   : milestone, 109,
    master - mean (109ms)  : 106, 113
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6202) - mean (772ms)  : 749, 794
     .   : milestone, 772,
    master - mean (772ms)  : 755, 788
     .   : milestone, 772,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6202) - mean (93ms)  : 90, 96
     .   : milestone, 93,
    master - mean (93ms)  : 90, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (6202) - mean (728ms)  : 715, 742
     .   : milestone, 728,
    master - mean (726ms)  : 709, 744
     .   : milestone, 726,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6202) - mean (190ms)  : 187, 193
     .   : milestone, 190,
    master - mean (190ms)  : 186, 193
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6202) - mean (1,201ms)  : 1174, 1228
     .   : milestone, 1201,
    master - mean (1,200ms)  : 1169, 1231
     .   : milestone, 1200,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6202) - mean (275ms)  : 271, 280
     .   : milestone, 275,
    master - mean (277ms)  : 272, 283
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6202) - mean (943ms)  : 924, 962
     .   : milestone, 943,
    master - mean (942ms)  : 922, 962
     .   : milestone, 942,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6202) - mean (264ms)  : 258, 269
     .   : milestone, 264,
    master - mean (265ms)  : 261, 268
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (6202) - mean (930ms)  : 910, 949
     .   : milestone, 930,
    master - mean (927ms)  : 907, 947
     .   : milestone, 927,

andrewlock · 2024-10-25T11:47:25Z

Benchmarks Report for tracer 🐌

Benchmarks for #6202 compared to master:

1 benchmarks are slower, with geometric mean 1.198
1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.6μs	41.6ns	328ns	0.0175	0.00699	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	9.88μs	48.9ns	201ns	0.02	0.00499	0	5.62 KB
master	`StartStopWithChild`	net472	16μs	46.4ns	180ns	1.03	0.332	0.103	6.06 KB
#6202	`StartStopWithChild`	net6.0	7.95μs	45.6ns	335ns	0.0154	0.00772	0	5.43 KB
#6202	`StartStopWithChild`	netcoreapp3.1	10.1μs	36.5ns	141ns	0.015	0.005	0	5.62 KB
#6202	`StartStopWithChild`	net472	16.1μs	48.6ns	188ns	1.03	0.316	0.103	6.05 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	479μs	272ns	1.05μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	632μs	216ns	780ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	847μs	417ns	1.56μs	0.446	3.3 KB
#6202	`WriteAndFlushEnrichedTraces`	net6.0	503μs	398ns	1.49μs	0	2.7 KB
#6202	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	631μs	630ns	2.44μs	0	2.7 KB
#6202	`WriteAndFlushEnrichedTraces`	net472	841μs	442ns	1.59μs	0.419	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	200μs	1.17μs	11.1μs	0.213	18.45 KB
master	`SendRequest`	netcoreapp3.1	221μs	1.26μs	8.81μs	0.216	20.61 KB
master	`SendRequest`	net472	1.9E‑05ns	1.9E‑05ns	7.11E‑05ns	0	0 b
#6202	`SendRequest`	net6.0	195μs	1.1μs	7.1μs	0.189	18.45 KB
#6202	`SendRequest`	netcoreapp3.1	221μs	1.28μs	10.7μs	0.223	20.61 KB
#6202	`SendRequest`	net472	0.00357ns	0.000949ns	0.00368ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6202

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.74 KB	41.38 KB	-363 B	-0.87%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	563μs	2.86μs	14μs	0.553	0	0	41.74 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	696μs	3.66μs	17.9μs	0.332	0	0	41.84 KB
master	`WriteAndFlushEnrichedTraces`	net472	882μs	2.79μs	10.8μs	8.13	2.57	0.428	53.32 KB
#6202	`WriteAndFlushEnrichedTraces`	net6.0	554μs	1.67μs	6.23μs	0.573	0	0	41.38 KB
#6202	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	685μs	3.27μs	13.9μs	0.326	0	0	41.82 KB
#6202	`WriteAndFlushEnrichedTraces`	net472	851μs	2.74μs	10.2μs	8.13	2.57	0.428	53.3 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.31μs	0.934ns	3.49ns	0.0144	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.7μs	3ns	11.6ns	0.0137	1.02 KB
master	`ExecuteNonQuery`	net472	2.08μs	2.3ns	8.92ns	0.156	987 B
#6202	`ExecuteNonQuery`	net6.0	1.25μs	1.27ns	4.93ns	0.0143	1.02 KB
#6202	`ExecuteNonQuery`	netcoreapp3.1	1.76μs	2.51ns	9.73ns	0.0131	1.02 KB
#6202	`ExecuteNonQuery`	net472	2.03μs	2.26ns	8.44ns	0.156	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.17μs	0.735ns	2.55ns	0.0138	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.61μs	3.27ns	12.7ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.4μs	1.53ns	5.73ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.23μs	0.565ns	2.11ns	0.0129	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.67μs	0.92ns	3.32ns	0.0138	1.02 KB
master	`CallElasticsearchAsync`	net472	2.47μs	1.87ns	6.99ns	0.167	1.05 KB
#6202	`CallElasticsearch`	net6.0	1.2μs	0.492ns	1.9ns	0.0138	976 B
#6202	`CallElasticsearch`	netcoreapp3.1	1.63μs	0.651ns	2.52ns	0.0131	976 B
#6202	`CallElasticsearch`	net472	2.5μs	1.32ns	5.12ns	0.158	995 B
#6202	`CallElasticsearchAsync`	net6.0	1.31μs	0.519ns	1.94ns	0.0132	952 B
#6202	`CallElasticsearchAsync`	netcoreapp3.1	1.61μs	0.626ns	2.34ns	0.0135	1.02 KB
#6202	`CallElasticsearchAsync`	net472	2.63μs	0.771ns	2.88ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.31μs	0.725ns	2.71ns	0.0132	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.69μs	1.51ns	5.87ns	0.0126	952 B
master	`ExecuteAsync`	net472	1.81μs	2.07ns	8ns	0.145	915 B
#6202	`ExecuteAsync`	net6.0	1.19μs	0.478ns	1.79ns	0.0136	952 B
#6202	`ExecuteAsync`	netcoreapp3.1	1.67μs	0.641ns	2.4ns	0.0127	952 B
#6202	`ExecuteAsync`	net472	1.74μs	0.39ns	1.51ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.19μs	0.869ns	3.36ns	0.0309	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.19μs	2.58ns	9.98ns	0.036	2.76 KB
master	`SendAsync`	net472	7.67μs	1.51ns	5.64ns	0.498	3.15 KB
#6202	`SendAsync`	net6.0	4.16μs	3.05ns	11.8ns	0.0312	2.22 KB
#6202	`SendAsync`	netcoreapp3.1	5.05μs	7.32ns	28.4ns	0.0352	2.76 KB
#6202	`SendAsync`	net472	7.77μs	1.67ns	6.02ns	0.499	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.46μs	0.573ns	2.15ns	0.0228	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.27μs	1.28ns	4.79ns	0.0216	1.64 KB
master	`EnrichedLog`	net472	2.6μs	1.34ns	5.17ns	0.249	1.57 KB
#6202	`EnrichedLog`	net6.0	1.44μs	0.556ns	2.08ns	0.023	1.64 KB
#6202	`EnrichedLog`	netcoreapp3.1	2.13μs	0.84ns	3.25ns	0.0221	1.64 KB
#6202	`EnrichedLog`	net472	2.67μs	1.09ns	4.09ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	288ns	1.11μs	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	122μs	140ns	486ns	0	0	4.28 KB
master	`EnrichedLog`	net472	152μs	186ns	695ns	0.686	0.229	4.46 KB
#6202	`EnrichedLog`	net6.0	115μs	115ns	430ns	0	0	4.28 KB
#6202	`EnrichedLog`	netcoreapp3.1	123μs	221ns	856ns	0	0	4.28 KB
#6202	`EnrichedLog`	net472	152μs	152ns	589ns	0.686	0.229	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.08μs	1.04ns	3.9ns	0.0311	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.32μs	1.96ns	7.6ns	0.0302	2.2 KB
master	`EnrichedLog`	net472	5μs	5.86ns	22.7ns	0.319	2.02 KB
#6202	`EnrichedLog`	net6.0	3.24μs	0.824ns	3.08ns	0.0308	2.2 KB
#6202	`EnrichedLog`	netcoreapp3.1	4.39μs	1.08ns	4.17ns	0.0304	2.2 KB
#6202	`EnrichedLog`	net472	4.67μs	1.09ns	4.09ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.35μs	0.767ns	2.87ns	0.016	0	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.75μs	0.511ns	1.84ns	0.0158	0	1.14 KB
master	`SendReceive`	net472	2.16μs	0.876ns	3.39ns	0.184	0	1.16 KB
#6202	`SendReceive`	net6.0	1.3μs	0.54ns	2.02ns	0.0162	0	1.14 KB
#6202	`SendReceive`	netcoreapp3.1	1.78μs	0.715ns	2.77ns	0.0151	0	1.14 KB
#6202	`SendReceive`	net472	2.16μs	0.982ns	3.8ns	0.183	0.00108	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.73μs	0.475ns	1.84ns	0.0218	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	4.01μs	1.81ns	7.02ns	0.022	1.65 KB
master	`EnrichedLog`	net472	4.37μs	1.34ns	5.03ns	0.322	2.04 KB
#6202	`EnrichedLog`	net6.0	2.67μs	0.881ns	3.41ns	0.0227	1.6 KB
#6202	`EnrichedLog`	netcoreapp3.1	3.9μs	1.15ns	4.29ns	0.0215	1.65 KB
#6202	`EnrichedLog`	net472	4.38μs	1.95ns	7.29ns	0.324	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	396ns	0.139ns	0.539ns	0.00802	576 B
master	`StartFinishSpan`	netcoreapp3.1	655ns	0.41ns	1.59ns	0.00754	576 B
master	`StartFinishSpan`	net472	696ns	0.395ns	1.53ns	0.0915	578 B
master	`StartFinishScope`	net6.0	530ns	0.295ns	1.11ns	0.00982	696 B
master	`StartFinishScope`	netcoreapp3.1	694ns	0.279ns	1.04ns	0.00942	696 B
master	`StartFinishScope`	net472	896ns	0.41ns	1.53ns	0.105	658 B
#6202	`StartFinishSpan`	net6.0	412ns	0.164ns	0.591ns	0.00807	576 B
#6202	`StartFinishSpan`	netcoreapp3.1	651ns	0.85ns	3.29ns	0.00787	576 B
#6202	`StartFinishSpan`	net472	692ns	1.4ns	5.43ns	0.0917	578 B
#6202	`StartFinishScope`	net6.0	478ns	0.116ns	0.418ns	0.00972	696 B
#6202	`StartFinishScope`	netcoreapp3.1	748ns	0.476ns	1.84ns	0.00916	696 B
#6202	`StartFinishScope`	net472	883ns	0.489ns	1.89ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6202

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.198	589.77	706.36

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	590ns	0.19ns	0.736ns	0.00975	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	926ns	0.411ns	1.59ns	0.00937	696 B
master	`RunOnMethodBegin`	net472	1.14μs	0.322ns	1.21ns	0.104	658 B
#6202	`RunOnMethodBegin`	net6.0	706ns	0.176ns	0.683ns	0.00959	696 B
#6202	`RunOnMethodBegin`	netcoreapp3.1	898ns	0.362ns	1.4ns	0.00929	696 B
#6202	`RunOnMethodBegin`	net472	1.15μs	0.244ns	0.946ns	0.104	658 B

e-n-0

LGTM!

Fix snapshot

c911b26

github-actions bot added the area:tests unit tests, integration tests label Oct 25, 2024

NachoEchevarria added the area:asm label Oct 25, 2024

NachoEchevarria changed the title ~~Fix snapshot~~ [ASM] Fix snapshot netcore 2.1 Oct 25, 2024

NachoEchevarria changed the title ~~[ASM] Fix snapshot netcore 2.1~~ [ASM] Fix TestExternalWafHeaders snapshot netcore 2.1 Oct 25, 2024

NachoEchevarria marked this pull request as ready for review October 25, 2024 10:33

NachoEchevarria requested a review from a team as a code owner October 25, 2024 10:33

andrewlock approved these changes Oct 25, 2024

View reviewed changes

e-n-0 approved these changes Oct 25, 2024

View reviewed changes

NachoEchevarria merged commit 47a2861 into master Oct 25, 2024
73 of 75 checks passed

NachoEchevarria deleted the nacho/Fix2.1SnapshotsRulesVersion branch October 25, 2024 12:00

github-actions bot added this to the vNext-v3 milestone Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ASM] Fix TestExternalWafHeaders snapshot netcore 2.1 #6202

[ASM] Fix TestExternalWafHeaders snapshot netcore 2.1 #6202

NachoEchevarria commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024

datadog-ddstaging bot commented Oct 25, 2024 •

edited

Loading

andrewlock commented Oct 25, 2024

andrewlock commented Oct 25, 2024

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6202

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6202

Raw results

e-n-0 left a comment

[ASM] Fix TestExternalWafHeaders snapshot netcore 2.1 #6202

[ASM] Fix TestExternalWafHeaders snapshot netcore 2.1 #6202

Conversation

NachoEchevarria commented Oct 25, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

github-actions bot commented Oct 25, 2024

Snapshots difference summary

datadog-ddstaging bot commented Oct 25, 2024 • edited Loading

Datadog Report

❌ Failed Tests (3)

andrewlock commented Oct 25, 2024

Execution-Time Benchmarks Report ⏱️

andrewlock commented Oct 25, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6202

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6202

Raw results

e-n-0 left a comment

Choose a reason for hiding this comment

NachoEchevarria commented Oct 25, 2024 •

edited

Loading

datadog-ddstaging bot commented Oct 25, 2024 •

edited

Loading