[ASM] Fix one flakiness in rcm asm data integration tests and simplify some asm rcm code #6119

anna-git · 2024-10-04T16:11:06Z

Summary of changes

Problem was: we send asm features.asm.enabled=true at the same time as we sent some asm_data. But, we need to rememeber that when asm is off, there is NO asm_data, asm products rc subscriptions. So we're not expecting these products hence a common warning in the logs like:

Received config RemoteConfigurationPath { Path = datadog/2/ASM_DATA/AspNetCore5AsmDataSecurityEnabledBlockingRequestIpOneClick8f9d4f72-ea07-478c-ae2f-4cf25117a1e7/config

It was flaky and still working because as soon as asm_features.asm.enabled = true is applied, we apply other configurations that continue to be sent right after (as the mock agent is setup with these files and will give the same response if unchanged). So most of the time, the 2 rc sets would be applied quickly and the request sent right after would show a security event.

But, we weren't waiting for the right state, as we were waiting for only one rc response (with asm_features and asm_data together) and we know they can't be applied at the same time (as the asm_data subscription is removed). It's applied right after. That's why, most of the time, it worked. But it shouldn't really work as we should have been waiting for another round of rc polling. So now, we separate both requests.

So conclusion: beware of what subtle flake can turning on/off subscriptions cause.
We still want to keep behaving the same though, we dont want to have useless products subscriptions if appsec is disabled, even if that means a creation then later an update of the waf..

Reason for change

Simplify some code as well, no need for the fallback to embedded rule set flag, just test if "rules" is in the dictionary, if not add it from the embedded set.

Implementation details

Test coverage

Other details

datadog-ddstaging · 2024-10-04T16:32:47Z

Datadog Report

Branch report: anna/asm/fix-flakiness-asmdata
Commit report: 6a39674
Test service: dd-trace-dotnet

✅ 0 Failed, 368282 Passed, 2087 Skipped, 16h 14m 2.01s Total Time

andrewlock · 2024-10-04T16:42:08Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6119) - mean (71ms)  : 68, 74
     .   : milestone, 71,
    master - mean (71ms)  : 68, 74
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (6119) - mean (1,116ms)  : 1089, 1144
     .   : milestone, 1116,
    master - mean (1,109ms)  : 1084, 1134
     .   : milestone, 1109,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6119) - mean (111ms)  : 107, 115
     .   : milestone, 111,
    master - mean (109ms)  : 107, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6119) - mean (782ms)  : 762, 802
     .   : milestone, 782,
    master - mean (777ms)  : 762, 792
     .   : milestone, 777,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6119) - mean (93ms)  : 90, 96
     .   : milestone, 93,
    master - mean (93ms)  : 92, 95
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (6119) - mean (731ms)  : 718, 745
     .   : milestone, 731,
    master - mean (731ms)  : 716, 747
     .   : milestone, 731,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6119) - mean (192ms)  : 188, 197
     .   : milestone, 192,
    master - mean (191ms)  : 188, 194
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6119) - mean (1,199ms)  : 1167, 1231
     .   : milestone, 1199,
    master - mean (1,199ms)  : 1171, 1227
     .   : milestone, 1199,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6119) - mean (280ms)  : 273, 288
     .   : milestone, 280,
    master - mean (276ms)  : 271, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6119) - mean (945ms)  : 925, 966
     .   : milestone, 945,
    master - mean (939ms)  : 921, 957
     .   : milestone, 939,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6119) - mean (264ms)  : 260, 268
     .   : milestone, 264,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (6119) - mean (927ms)  : 904, 949
     .   : milestone, 927,
    master - mean (927ms)  : 908, 947
     .   : milestone, 927,

andrewlock · 2024-10-04T17:07:56Z

Benchmarks Report for appsec 🐌

Benchmarks for #6119 compared to master:

1 benchmarks are faster, with geometric mean 1.199
1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6119

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑net472	1.199	4,672.82	3,897.74

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	71.4μs	78.9ns	306ns	0.072	0	6 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	62.9μs	40.9ns	148ns	0.0941	0	6.95 KB
master	`AllCycleSimpleBody`	net472	48.5μs	25ns	93.6ns	1.31	0	8.34 KB
master	`AllCycleMoreComplexBody`	net6.0	76.9μs	71ns	266ns	0.115	0	9.51 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	69μs	64.2ns	249ns	0.138	0	10.36 KB
master	`AllCycleMoreComplexBody`	net472	54.7μs	26.7ns	99.8ns	1.86	0.0274	11.85 KB
master	`ObjectExtractorSimpleBody`	net6.0	140ns	0.129ns	0.483ns	0.00392	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	195ns	0.334ns	1.29ns	0.0037	0	272 B
master	`ObjectExtractorSimpleBody`	net472	164ns	0.29ns	1.12ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.15μs	1.12ns	4.05ns	0.0536	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.89μs	3.27ns	12.7ns	0.0506	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	4.67μs	3.58ns	13.9ns	0.603	0.0046	3.8 KB
#6119	`AllCycleSimpleBody`	net6.0	73.7μs	362ns	1.88μs	0.0727	0	6 KB
#6119	`AllCycleSimpleBody`	netcoreapp3.1	63.6μs	90.7ns	351ns	0.095	0	6.95 KB
#6119	`AllCycleSimpleBody`	net472	48.5μs	74.5ns	289ns	1.32	0	8.34 KB
#6119	`AllCycleMoreComplexBody`	net6.0	76.8μs	73.3ns	264ns	0.115	0	9.51 KB
#6119	`AllCycleMoreComplexBody`	netcoreapp3.1	69.8μs	95.2ns	356ns	0.14	0	10.37 KB
#6119	`AllCycleMoreComplexBody`	net472	55μs	105ns	408ns	1.87	0.0274	11.85 KB
#6119	`ObjectExtractorSimpleBody`	net6.0	142ns	0.245ns	0.947ns	0.00393	0	280 B
#6119	`ObjectExtractorSimpleBody`	netcoreapp3.1	208ns	0.23ns	0.891ns	0.00376	0	272 B
#6119	`ObjectExtractorSimpleBody`	net472	166ns	0.0865ns	0.312ns	0.0446	0	281 B
#6119	`ObjectExtractorMoreComplexBody`	net6.0	3.1μs	2.63ns	9.83ns	0.0541	0	3.78 KB
#6119	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.03μs	2.66ns	9.94ns	0.0506	0	3.69 KB
#6119	`ObjectExtractorMoreComplexBody`	net472	3.89μs	2.97ns	11.1ns	0.603	0.00583	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	38.1μs	19ns	73.6ns	0.455	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	54.4μs	17.8ns	69ns	0.433	0	32.4 KB
master	`EncodeArgs`	net472	65.8μs	37.3ns	144ns	5.14	0.0659	32.5 KB
master	`EncodeLegacyArgs`	net6.0	75.8μs	418ns	2.51μs	0	0	2.14 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	104μs	426ns	1.65μs	0	0	2.15 KB
master	`EncodeLegacyArgs`	net472	151μs	103ns	400ns	0.3	0	2.15 KB
#6119	`EncodeArgs`	net6.0	36.7μs	17.8ns	69.1ns	0.461	0	32.4 KB
#6119	`EncodeArgs`	netcoreapp3.1	54.4μs	15.9ns	57.5ns	0.434	0	32.4 KB
#6119	`EncodeArgs`	net472	65.6μs	18.8ns	70.5ns	5.16	0.0657	32.5 KB
#6119	`EncodeLegacyArgs`	net6.0	70.6μs	44.4ns	172ns	0	0	2.14 KB
#6119	`EncodeLegacyArgs`	netcoreapp3.1	107μs	244ns	943ns	0	0	2.15 KB
#6119	`EncodeLegacyArgs`	net472	153μs	364ns	1.36μs	0.299	0	2.15 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	186μs	142ns	548ns	0	2.44 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	198μs	261ns	1.01μs	0	2.39 KB
master	`RunWafRealisticBenchmark`	net472	211μs	168ns	651ns	0.311	2.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	122μs	108ns	404ns	0	1.47 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	130μs	57.4ns	222ns	0	1.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	139μs	32.7ns	127ns	0.208	1.49 KB
#6119	`RunWafRealisticBenchmark`	net6.0	186μs	65.6ns	245ns	0	2.44 KB
#6119	`RunWafRealisticBenchmark`	netcoreapp3.1	199μs	229ns	885ns	0	2.39 KB
#6119	`RunWafRealisticBenchmark`	net472	210μs	194ns	750ns	0.311	2.46 KB
#6119	`RunWafRealisticBenchmarkWithAttack`	net6.0	122μs	72ns	279ns	0	1.47 KB
#6119	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	129μs	109ns	409ns	0	1.46 KB
#6119	`RunWafRealisticBenchmarkWithAttack`	net472	139μs	33.5ns	130ns	0.208	1.49 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6119

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	265.54 KB	254.52 KB	-11.02 KB	-4.15%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	60μs	782ns	7.78μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	60μs	802ns	7.98μs	42.64 KB
master	`StringConcatBenchmark`	net472	37.3μs	50ns	180ns	59.07 KB
master	`StringConcatAspectBenchmark`	net6.0	306μs	1.63μs	9.92μs	253.55 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	339μs	1μs	3.47μs	265.54 KB
master	`StringConcatAspectBenchmark`	net472	276μs	5.47μs	53.3μs	278.53 KB
#6119	`StringConcatBenchmark`	net6.0	58.8μs	795ns	7.91μs	43.44 KB
#6119	`StringConcatBenchmark`	netcoreapp3.1	60.4μs	722ns	7.22μs	42.64 KB
#6119	`StringConcatBenchmark`	net472	36.9μs	54.5ns	196ns	59.07 KB
#6119	`StringConcatAspectBenchmark`	net6.0	315μs	1.77μs	13μs	254.78 KB
#6119	`StringConcatAspectBenchmark`	netcoreapp3.1	334μs	1.52μs	5.49μs	254.52 KB
#6119	`StringConcatAspectBenchmark`	net472	279μs	6.12μs	59.6μs	278.53 KB

andrewlock · 2024-10-04T17:49:40Z

Benchmarks Report for tracer 🐌

Benchmarks for #6119 compared to master:

1 benchmarks are faster, with geometric mean 1.187
1 benchmarks are slower, with geometric mean 1.170
1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.83μs	43.8ns	270ns	0.0149	0.00744	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	9.81μs	54.2ns	347ns	0.0189	0.00945	0	5.61 KB
master	`StartStopWithChild`	net472	16.8μs	75ns	291ns	1.02	0.311	0.0864	6.06 KB
#6119	`StartStopWithChild`	net6.0	7.75μs	43.3ns	287ns	0.0157	0.00784	0	5.42 KB
#6119	`StartStopWithChild`	netcoreapp3.1	10.2μs	51.5ns	268ns	0.0145	0.00484	0	5.62 KB
#6119	`StartStopWithChild`	net472	16.7μs	76.2ns	295ns	1.02	0.316	0.0972	6.06 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	499μs	216ns	778ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	648μs	730ns	2.83μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	826μs	214ns	828ns	0.414	3.3 KB
#6119	`WriteAndFlushEnrichedTraces`	net6.0	465μs	619ns	2.4μs	0	2.7 KB
#6119	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	645μs	407ns	1.57μs	0	2.7 KB
#6119	`WriteAndFlushEnrichedTraces`	net472	840μs	587ns	2.27μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	192μs	1.07μs	6.66μs	0.147	18.45 KB
master	`SendRequest`	netcoreapp3.1	218μs	1.18μs	8.24μs	0.217	20.61 KB
master	`SendRequest`	net472	0.00176ns	0.000702ns	0.00272ns	0	0 b
#6119	`SendRequest`	net6.0	207μs	1.21μs	11.8μs	0.186	18.45 KB
#6119	`SendRequest`	netcoreapp3.1	216μs	1.21μs	7.92μs	0.213	20.61 KB
#6119	`SendRequest`	net472	0.0199ns	0.00192ns	0.00744ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6119

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.75 KB	41.5 KB	-253 B	-0.61%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	576μs	1.64μs	6.12μs	0.553	0	0	41.75 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	699μs	3.53μs	16.6μs	0.336	0	0	41.73 KB
master	`WriteAndFlushEnrichedTraces`	net472	852μs	3.34μs	12.9μs	8.45	2.53	0.422	53.28 KB
#6119	`WriteAndFlushEnrichedTraces`	net6.0	561μs	2.98μs	14.6μs	0.558	0	0	41.5 KB
#6119	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	680μs	2.8μs	10.8μs	0.338	0	0	41.83 KB
#6119	`WriteAndFlushEnrichedTraces`	net472	865μs	3.86μs	14.4μs	8.13	2.57	0.428	53.33 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.36μs	1.25ns	4.84ns	0.014	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.75μs	1.7ns	6.58ns	0.0139	1.02 KB
master	`ExecuteNonQuery`	net472	2.15μs	1.75ns	6.57ns	0.157	987 B
#6119	`ExecuteNonQuery`	net6.0	1.33μs	1.36ns	5.26ns	0.014	1.02 KB
#6119	`ExecuteNonQuery`	netcoreapp3.1	1.76μs	0.869ns	3.36ns	0.0141	1.02 KB
#6119	`ExecuteNonQuery`	net472	1.99μs	2.28ns	8.84ns	0.157	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.17μs	0.644ns	2.41ns	0.0135	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.49μs	1.09ns	4.22ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.5μs	1.29ns	5ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.28μs	0.913ns	3.42ns	0.0134	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.6μs	0.532ns	1.99ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.57μs	1.07ns	4.01ns	0.167	1.05 KB
#6119	`CallElasticsearch`	net6.0	1.26μs	0.941ns	3.64ns	0.0139	976 B
#6119	`CallElasticsearch`	netcoreapp3.1	1.5μs	0.886ns	3.43ns	0.0128	976 B
#6119	`CallElasticsearch`	net472	2.47μs	1.55ns	5.99ns	0.158	995 B
#6119	`CallElasticsearchAsync`	net6.0	1.27μs	0.474ns	1.83ns	0.0133	952 B
#6119	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	0.712ns	2.67ns	0.0136	1.02 KB
#6119	`CallElasticsearchAsync`	net472	2.64μs	2.59ns	10ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.25μs	2.17ns	8.41ns	0.0131	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.61μs	1.36ns	5.07ns	0.0129	952 B
master	`ExecuteAsync`	net472	1.9μs	0.723ns	2.8ns	0.144	915 B
#6119	`ExecuteAsync`	net6.0	1.27μs	1.03ns	4ns	0.013	952 B
#6119	`ExecuteAsync`	netcoreapp3.1	1.52μs	1.72ns	6.43ns	0.013	952 B
#6119	`ExecuteAsync`	net472	1.82μs	3.3ns	12.8ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.11μs	2.8ns	10.5ns	0.0308	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.16μs	3.69ns	14.3ns	0.0361	2.76 KB
master	`SendAsync`	net472	7.75μs	5.08ns	19ns	0.498	3.15 KB
#6119	`SendAsync`	net6.0	4.19μs	1.7ns	6.35ns	0.0313	2.22 KB
#6119	`SendAsync`	netcoreapp3.1	5.04μs	5.86ns	22.7ns	0.0355	2.76 KB
#6119	`SendAsync`	net472	7.84μs	7.48ns	29ns	0.499	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.44μs	0.626ns	2.34ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.22μs	1.67ns	6.24ns	0.0223	1.64 KB
master	`EnrichedLog`	net472	2.54μs	1.09ns	4.07ns	0.249	1.57 KB
#6119	`EnrichedLog`	net6.0	1.56μs	0.782ns	3.03ns	0.0227	1.64 KB
#6119	`EnrichedLog`	netcoreapp3.1	2.36μs	0.726ns	2.81ns	0.0224	1.64 KB
#6119	`EnrichedLog`	net472	2.57μs	0.677ns	2.53ns	0.25	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	128ns	495ns	0.0567	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	166ns	644ns	0.0588	0	4.28 KB
master	`EnrichedLog`	net472	146μs	134ns	519ns	0.654	0.218	4.46 KB
#6119	`EnrichedLog`	net6.0	115μs	274ns	1.06μs	0	0	4.28 KB
#6119	`EnrichedLog`	netcoreapp3.1	118μs	136ns	510ns	0.0589	0	4.28 KB
#6119	`EnrichedLog`	net472	146μs	121ns	454ns	0.654	0.218	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.06μs	1.91ns	7.39ns	0.0305	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.3μs	1.99ns	7.69ns	0.0302	2.2 KB
master	`EnrichedLog`	net472	4.85μs	2ns	7.49ns	0.32	2.02 KB
#6119	`EnrichedLog`	net6.0	3.08μs	3.12ns	12.1ns	0.0309	2.2 KB
#6119	`EnrichedLog`	netcoreapp3.1	4.17μs	2.52ns	9.77ns	0.0291	2.2 KB
#6119	`EnrichedLog`	net472	4.86μs	1.25ns	4.85ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.35μs	0.772ns	2.89ns	0.0161	0	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.83μs	0.869ns	3.37ns	0.0155	0	1.14 KB
master	`SendReceive`	net472	2.14μs	1.73ns	6.7ns	0.183	0.00108	1.16 KB
#6119	`SendReceive`	net6.0	1.34μs	1.16ns	4.49ns	0.016	0	1.14 KB
#6119	`SendReceive`	netcoreapp3.1	1.88μs	1.33ns	5.14ns	0.015	0	1.14 KB
#6119	`SendReceive`	net472	2.08μs	0.89ns	3.45ns	0.183	0.00104	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.67μs	1.16ns	4.48ns	0.0214	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.81μs	1.71ns	6.61ns	0.0211	1.65 KB
master	`EnrichedLog`	net472	4.46μs	2.37ns	8.87ns	0.322	2.04 KB
#6119	`EnrichedLog`	net6.0	2.97μs	0.741ns	2.77ns	0.0221	1.6 KB
#6119	`EnrichedLog`	netcoreapp3.1	3.87μs	0.59ns	2.21ns	0.0213	1.65 KB
#6119	`EnrichedLog`	net472	4.54μs	1.99ns	7.72ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6119

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.170	485.03	567.45

Faster 🎉 in #6119

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.187	722.58	608.69

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	413ns	0.329ns	1.27ns	0.00812	576 B
master	`StartFinishSpan`	netcoreapp3.1	649ns	1.79ns	6.92ns	0.0077	576 B
master	`StartFinishSpan`	net472	722ns	0.204ns	0.764ns	0.0915	578 B
master	`StartFinishScope`	net6.0	485ns	0.356ns	1.38ns	0.00974	696 B
master	`StartFinishScope`	netcoreapp3.1	733ns	1.47ns	5.67ns	0.00939	696 B
master	`StartFinishScope`	net472	844ns	0.555ns	2ns	0.104	658 B
#6119	`StartFinishSpan`	net6.0	398ns	0.158ns	0.613ns	0.00804	576 B
#6119	`StartFinishSpan`	netcoreapp3.1	588ns	0.423ns	1.58ns	0.00799	576 B
#6119	`StartFinishSpan`	net472	609ns	0.764ns	2.96ns	0.0916	578 B
#6119	`StartFinishScope`	net6.0	567ns	0.283ns	1.1ns	0.00969	696 B
#6119	`StartFinishScope`	netcoreapp3.1	779ns	0.756ns	2.93ns	0.00932	696 B
#6119	`StartFinishScope`	net472	836ns	0.643ns	2.49ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	591ns	0.29ns	1.12ns	0.00966	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	903ns	0.496ns	1.92ns	0.00919	696 B
master	`RunOnMethodBegin`	net472	1.07μs	0.313ns	1.13ns	0.105	658 B
#6119	`RunOnMethodBegin`	net6.0	593ns	0.155ns	0.561ns	0.00976	696 B
#6119	`RunOnMethodBegin`	netcoreapp3.1	942ns	0.529ns	1.98ns	0.00917	696 B
#6119	`RunOnMethodBegin`	net472	1.1μs	0.89ns	3.45ns	0.104	658 B

andrewlock · 2024-10-04T19:12:34Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6119) (11.110M)   : 0, 11110019
    master (10.947M)   : 0, 10947067
    benchmarks/2.9.0 (11.081M)   : 0, 11080577

    section Automatic
    This PR (6119) (7.259M)   : 0, 7259040
    master (7.357M)   : 0, 7357178
    benchmarks/2.9.0 (7.732M)   : 0, 7732233

    section Trace stats
    master (7.658M)   : 0, 7658023

    section Manual
    master (11.170M)   : 0, 11169713

    section Manual + Automatic
    This PR (6119) (6.736M)   : 0, 6736159
    master (6.794M)   : 0, 6793735

    section DD_TRACE_ENABLED=0
    master (10.165M)   : 0, 10164915

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6119) (9.461M)   : 0, 9461066
    master (9.407M)   : 0, 9406571
    benchmarks/2.9.0 (9.798M)   : 0, 9798067

    section Automatic
    This PR (6119) (6.423M)   : 0, 6422520
    master (6.543M)   : 0, 6542618

    section Trace stats
    master (6.888M)   : 0, 6887750

    section Manual
    master (9.380M)   : 0, 9379533

    section Manual + Automatic
    This PR (6119) (6.126M)   : 0, 6125955
    master (6.148M)   : 0, 6147954

    section DD_TRACE_ENABLED=0
    master (8.559M)   : 0, 8559117

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6119) (10.539M)   : 0, 10538775
    master (9.869M)   : 0, 9868639
    benchmarks/2.9.0 (10.067M)   : 0, 10067315

    section Automatic
    This PR (6119) (6.853M)   : 0, 6852857
    master (6.673M)   : 0, 6672534
    benchmarks/2.9.0 (7.552M)   : 0, 7552193

    section Trace stats
    master (7.379M)   : 0, 7378820

    section Manual
    master (10.188M)   : 0, 10188482

    section Manual + Automatic
    This PR (6119) (6.364M)   : 0, 6363526
    master (6.068M)   : 0, 6067649

    section DD_TRACE_ENABLED=0
    master (9.316M)   : 0, 9315564

daniel-romano-DD · 2024-10-08T15:05:52Z

tracer/src/Datadog.Trace/AppSec/Waf/ReturnTypes.Managed/InitResult.cs

@@ -22,7 +22,6 @@ private InitResult(ushort failedToLoadRules, ushort loadedRules, string ruleFile
        {
            HasErrors = errors.Count > 0;
            Errors = errors;
-            EmbeddedRules = embeddedRules;


This argument seems not to be used anymore

daniel-romano-DD

LGTM with a minor NIT

NachoEchevarria

LGTM!

github-actions bot added the area:asm label Oct 4, 2024

anna-git added 3 commits October 8, 2024 15:06

Renames and simplify syntax

ad4434a

Fix the flakiness first

2a3a05d

Simplify rcm updates

25166aa

anna-git force-pushed the anna/asm/fix-flakiness-asmdata branch from 277cefe to 25166aa Compare October 8, 2024 13:07

anna-git marked this pull request as ready for review October 8, 2024 14:42

anna-git requested review from a team as code owners October 8, 2024 14:42

daniel-romano-DD reviewed Oct 8, 2024

View reviewed changes

daniel-romano-DD approved these changes Oct 8, 2024

View reviewed changes

NachoEchevarria approved these changes Oct 8, 2024

View reviewed changes

answer to comment

6a39674

anna-git merged commit 0653d23 into master Oct 9, 2024
75 checks passed

anna-git deleted the anna/asm/fix-flakiness-asmdata branch October 9, 2024 16:35

github-actions bot added this to the vNext-v3 milestone Oct 9, 2024

andrewlock added area:tests unit tests, integration tests type:cleanup Minor code clean up labels Oct 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ASM] Fix one flakiness in rcm asm data integration tests and simplify some asm rcm code #6119

[ASM] Fix one flakiness in rcm asm data integration tests and simplify some asm rcm code #6119

anna-git commented Oct 4, 2024

datadog-ddstaging bot commented Oct 4, 2024 •

edited

Loading

andrewlock commented Oct 4, 2024 •

edited

Loading

andrewlock commented Oct 4, 2024 •

edited

Loading

Faster 🎉 in #6119

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6119

Raw results

andrewlock commented Oct 4, 2024 •

edited

Loading

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6119

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6119

Faster 🎉 in #6119

Raw results

Raw results

andrewlock commented Oct 4, 2024 •

edited

Loading

daniel-romano-DD Oct 8, 2024

daniel-romano-DD left a comment

NachoEchevarria left a comment

[ASM] Fix one flakiness in rcm asm data integration tests and simplify some asm rcm code #6119

[ASM] Fix one flakiness in rcm asm data integration tests and simplify some asm rcm code #6119

Conversation

anna-git commented Oct 4, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Oct 4, 2024 • edited Loading

Datadog Report

andrewlock commented Oct 4, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Oct 4, 2024 • edited Loading

Benchmarks Report for appsec 🐌

Benchmark details

Faster 🎉 in #6119

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6119

Raw results

andrewlock commented Oct 4, 2024 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6119

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6119

Faster 🎉 in #6119

Raw results

Raw results

andrewlock commented Oct 4, 2024 • edited Loading

Throughput/Crank Report ⚡

daniel-romano-DD Oct 8, 2024

Choose a reason for hiding this comment

daniel-romano-DD left a comment

Choose a reason for hiding this comment

NachoEchevarria left a comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Oct 4, 2024 •

edited

Loading

andrewlock commented Oct 4, 2024 •

edited

Loading

andrewlock commented Oct 4, 2024 •

edited

Loading

andrewlock commented Oct 4, 2024 •

edited

Loading

andrewlock commented Oct 4, 2024 •

edited

Loading