-
Notifications
You must be signed in to change notification settings - Fork 142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add SSI denylist and tests #5928
Conversation
Datadog ReportBranch report: ✅ 0 Failed, 379572 Passed, 3091 Skipped, 27h 26m 4.33s Total Time |
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5928) - mean (70ms) : 67, 72
. : milestone, 70,
master - mean (70ms) : 67, 72
. : milestone, 70,
section CallTarget+Inlining+NGEN
This PR (5928) - mean (1,113ms) : 1088, 1137
. : milestone, 1113,
master - mean (1,106ms) : 1086, 1127
. : milestone, 1106,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5928) - mean (108ms) : 106, 111
. : milestone, 108,
master - mean (109ms) : 106, 112
. : milestone, 109,
section CallTarget+Inlining+NGEN
This PR (5928) - mean (770ms) : 754, 787
. : milestone, 770,
master - mean (772ms) : 755, 788
. : milestone, 772,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5928) - mean (92ms) : 89, 95
. : milestone, 92,
master - mean (92ms) : 89, 94
. : milestone, 92,
section CallTarget+Inlining+NGEN
This PR (5928) - mean (724ms) : 709, 740
. : milestone, 724,
master - mean (726ms) : 707, 746
. : milestone, 726,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5928) - mean (191ms) : 188, 194
. : milestone, 191,
master - mean (191ms) : 188, 194
. : milestone, 191,
section CallTarget+Inlining+NGEN
This PR (5928) - mean (1,201ms) : 1177, 1225
. : milestone, 1201,
master - mean (1,202ms) : 1175, 1230
. : milestone, 1202,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5928) - mean (276ms) : 272, 281
. : milestone, 276,
master - mean (276ms) : 271, 280
. : milestone, 276,
section CallTarget+Inlining+NGEN
This PR (5928) - mean (942ms) : 928, 956
. : milestone, 942,
master - mean (947ms) : 929, 964
. : milestone, 947,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5928) - mean (265ms) : 261, 269
. : milestone, 265,
master - mean (265ms) : 261, 268
. : milestone, 265,
section CallTarget+Inlining+NGEN
This PR (5928) - mean (927ms) : 905, 948
. : milestone, 927,
master - mean (932ms) : 910, 953
. : milestone, 932,
|
Benchmarks Report for tracer 🐌Benchmarks for #5928 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Slower
|
Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0 | 1.133 | 1,229.48 | 1,392.69 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | ExecuteAsync |
net6.0 | 1.23μs | 0.6ns | 2.33ns | 0.0135 | 0 | 0 | 952 B |
master | ExecuteAsync |
netcoreapp3.1 | 1.59μs | 0.764ns | 2.86ns | 0.0128 | 0 | 0 | 952 B |
master | ExecuteAsync |
net472 | 1.76μs | 0.656ns | 2.46ns | 0.145 | 0 | 0 | 915 B |
#5928 | ExecuteAsync |
net6.0 | 1.39μs | 0.613ns | 2.29ns | 0.0133 | 0 | 0 | 952 B |
#5928 | ExecuteAsync |
netcoreapp3.1 | 1.68μs | 0.897ns | 3.36ns | 0.0127 | 0 | 0 | 952 B |
#5928 | ExecuteAsync |
net472 | 1.8μs | 0.881ns | 3.41ns | 0.145 | 0 | 0 | 915 B |
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | SendAsync |
net6.0 | 4.2μs | 1.88ns | 7.29ns | 0.0315 | 0 | 0 | 2.22 KB |
master | SendAsync |
netcoreapp3.1 | 5.07μs | 3.93ns | 15.2ns | 0.0356 | 0 | 0 | 2.76 KB |
master | SendAsync |
net472 | 7.8μs | 2.37ns | 9.19ns | 0.5 | 0 | 0 | 3.15 KB |
#5928 | SendAsync |
net6.0 | 4.15μs | 1.34ns | 5.01ns | 0.0312 | 0 | 0 | 2.22 KB |
#5928 | SendAsync |
netcoreapp3.1 | 5.1μs | 1.93ns | 7.21ns | 0.0357 | 0 | 0 | 2.76 KB |
#5928 | SendAsync |
net472 | 7.72μs | 3.86ns | 14.9ns | 0.498 | 0 | 0 | 3.15 KB |
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 1.57μs | 0.854ns | 3.19ns | 0.0233 | 0 | 0 | 1.64 KB |
master | EnrichedLog |
netcoreapp3.1 | 2.18μs | 1.27ns | 4.9ns | 0.0226 | 0 | 0 | 1.64 KB |
master | EnrichedLog |
net472 | 2.62μs | 0.91ns | 3.4ns | 0.25 | 0 | 0 | 1.57 KB |
#5928 | EnrichedLog |
net6.0 | 1.5μs | 0.616ns | 2.39ns | 0.0233 | 0 | 0 | 1.64 KB |
#5928 | EnrichedLog |
netcoreapp3.1 | 2.33μs | 1.07ns | 4.14ns | 0.0221 | 0 | 0 | 1.64 KB |
#5928 | EnrichedLog |
net472 | 2.64μs | 0.677ns | 2.62ns | 0.249 | 0 | 0 | 1.57 KB |
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 118μs | 177ns | 684ns | 0.059 | 0 | 0 | 4.28 KB |
master | EnrichedLog |
netcoreapp3.1 | 119μs | 155ns | 599ns | 0.0598 | 0 | 0 | 4.28 KB |
master | EnrichedLog |
net472 | 151μs | 275ns | 1.06μs | 0.679 | 0.226 | 0 | 4.46 KB |
#5928 | EnrichedLog |
net6.0 | 117μs | 172ns | 667ns | 0.0574 | 0 | 0 | 4.28 KB |
#5928 | EnrichedLog |
netcoreapp3.1 | 122μs | 292ns | 1.13μs | 0 | 0 | 0 | 4.28 KB |
#5928 | EnrichedLog |
net472 | 150μs | 104ns | 403ns | 0.677 | 0.226 | 0 | 4.46 KB |
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 3.24μs | 3.65ns | 14.1ns | 0.0309 | 0 | 0 | 2.2 KB |
master | EnrichedLog |
netcoreapp3.1 | 4.26μs | 2.53ns | 9.8ns | 0.0298 | 0 | 0 | 2.2 KB |
master | EnrichedLog |
net472 | 4.82μs | 2.05ns | 7.93ns | 0.318 | 0 | 0 | 2.02 KB |
#5928 | EnrichedLog |
net6.0 | 3.16μs | 0.57ns | 2.13ns | 0.0301 | 0 | 0 | 2.2 KB |
#5928 | EnrichedLog |
netcoreapp3.1 | 4.12μs | 1.4ns | 5.23ns | 0.0289 | 0 | 0 | 2.2 KB |
#5928 | EnrichedLog |
net472 | 4.83μs | 1.66ns | 6.43ns | 0.32 | 0 | 0 | 2.02 KB |
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | SendReceive |
net6.0 | 1.35μs | 0.615ns | 2.38ns | 0.0164 | 0 | 0 | 1.14 KB |
master | SendReceive |
netcoreapp3.1 | 1.76μs | 0.901ns | 3.49ns | 0.0149 | 0 | 0 | 1.14 KB |
master | SendReceive |
net472 | 2.08μs | 1.04ns | 3.89ns | 0.183 | 0.00104 | 0 | 1.16 KB |
#5928 | SendReceive |
net6.0 | 1.35μs | 0.96ns | 3.72ns | 0.0162 | 0 | 0 | 1.14 KB |
#5928 | SendReceive |
netcoreapp3.1 | 1.73μs | 0.659ns | 2.46ns | 0.0155 | 0 | 0 | 1.14 KB |
#5928 | SendReceive |
net472 | 2.14μs | 1.1ns | 4.27ns | 0.183 | 0.00107 | 0 | 1.16 KB |
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 2.64μs | 0.576ns | 2.15ns | 0.0224 | 0 | 0 | 1.6 KB |
master | EnrichedLog |
netcoreapp3.1 | 3.88μs | 1.21ns | 4.38ns | 0.0213 | 0 | 0 | 1.65 KB |
master | EnrichedLog |
net472 | 4.45μs | 2.08ns | 8.06ns | 0.322 | 0 | 0 | 2.04 KB |
#5928 | EnrichedLog |
net6.0 | 2.65μs | 0.512ns | 1.92ns | 0.0225 | 0 | 0 | 1.6 KB |
#5928 | EnrichedLog |
netcoreapp3.1 | 3.93μs | 2.11ns | 8.17ns | 0.0216 | 0 | 0 | 1.65 KB |
#5928 | EnrichedLog |
net472 | 4.28μs | 2.5ns | 9.69ns | 0.324 | 0 | 0 | 2.04 KB |
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️
Faster 🎉 in #5928
Benchmark
base/diff
Base Median (ns)
Diff Median (ns)
Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0
1.169
467.80
400.24
Benchmark | base/diff | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 | 1.169 | 467.80 | 400.24 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | StartFinishSpan |
net6.0 | 468ns | 0.176ns | 0.682ns | 0.00802 | 0 | 0 | 576 B |
master | StartFinishSpan |
netcoreapp3.1 | 615ns | 1.44ns | 5.58ns | 0.00771 | 0 | 0 | 576 B |
master | StartFinishSpan |
net472 | 634ns | 1.1ns | 4.28ns | 0.0917 | 0 | 0 | 578 B |
master | StartFinishScope |
net6.0 | 494ns | 0.224ns | 0.868ns | 0.00985 | 0 | 0 | 696 B |
master | StartFinishScope |
netcoreapp3.1 | 766ns | 1.47ns | 5.1ns | 0.00953 | 0 | 0 | 696 B |
master | StartFinishScope |
net472 | 898ns | 0.377ns | 1.41ns | 0.104 | 0 | 0 | 658 B |
#5928 | StartFinishSpan |
net6.0 | 400ns | 0.216ns | 0.835ns | 0.00805 | 0 | 0 | 576 B |
#5928 | StartFinishSpan |
netcoreapp3.1 | 583ns | 0.254ns | 0.983ns | 0.0079 | 0 | 0 | 576 B |
#5928 | StartFinishSpan |
net472 | 685ns | 0.878ns | 3.4ns | 0.0915 | 0 | 0 | 578 B |
#5928 | StartFinishScope |
net6.0 | 477ns | 0.246ns | 0.951ns | 0.00979 | 0 | 0 | 696 B |
#5928 | StartFinishScope |
netcoreapp3.1 | 732ns | 0.585ns | 2.27ns | 0.00959 | 0 | 0 | 696 B |
#5928 | StartFinishScope |
net472 | 907ns | 0.909ns | 3.52ns | 0.104 | 0 | 0 | 658 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️
Faster 🎉 in #5928
Benchmark
base/diff
Base Median (ns)
Diff Median (ns)
Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0
1.150
685.72
596.18
Benchmark | base/diff | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 | 1.150 | 685.72 | 596.18 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | RunOnMethodBegin |
net6.0 | 686ns | 2.3ns | 8.9ns | 0.00977 | 0 | 0 | 696 B |
master | RunOnMethodBegin |
netcoreapp3.1 | 964ns | 2.07ns | 8.01ns | 0.00961 | 0 | 0 | 696 B |
master | RunOnMethodBegin |
net472 | 1.17μs | 1.12ns | 4.36ns | 0.104 | 0 | 0 | 658 B |
#5928 | RunOnMethodBegin |
net6.0 | 596ns | 0.389ns | 1.51ns | 0.00984 | 0 | 0 | 696 B |
#5928 | RunOnMethodBegin |
netcoreapp3.1 | 986ns | 1.37ns | 5.29ns | 0.00928 | 0 | 0 | 696 B |
#5928 | RunOnMethodBegin |
net472 | 1.17μs | 1.29ns | 5ns | 0.104 | 0 | 0 | 658 B |
499b7ee
to
b0acedd
Compare
b0acedd
to
9cc1910
Compare
df45e2b
to
6b2b0ce
Compare
aac8631
to
eb5bcb0
Compare
Throughput/Crank Report ⚡Throughput results for AspNetCoreSimpleController comparing the following branches/commits: Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red. Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards! gantt
title Throughput Linux x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5928) (11.135M) : 0, 11135451
master (11.127M) : 0, 11126752
benchmarks/2.9.0 (11.081M) : 0, 11080577
section Automatic
This PR (5928) (7.455M) : 0, 7454973
master (7.377M) : 0, 7376563
benchmarks/2.9.0 (7.732M) : 0, 7732233
section Trace stats
master (7.612M) : 0, 7612091
section Manual
master (11.042M) : 0, 11042306
section Manual + Automatic
This PR (5928) (6.885M) : 0, 6885258
master (6.743M) : 0, 6742844
section DD_TRACE_ENABLED=0
master (10.176M) : 0, 10175955
gantt
title Throughput Linux arm64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5928) (9.407M) : 0, 9407418
master (9.601M) : 0, 9600942
benchmarks/2.9.0 (9.798M) : 0, 9798067
section Automatic
This PR (5928) (6.388M) : 0, 6388490
master (6.478M) : 0, 6477825
section Trace stats
master (6.824M) : 0, 6824216
section Manual
master (9.656M) : 0, 9655746
section Manual + Automatic
This PR (5928) (6.152M) : 0, 6151550
master (6.006M) : 0, 6006388
section DD_TRACE_ENABLED=0
master (8.643M) : 0, 8642890
gantt
title Throughput Windows x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5928) (10.161M) : 0, 10160901
master (10.217M) : 0, 10217051
benchmarks/2.9.0 (10.067M) : 0, 10067315
section Automatic
This PR (5928) (6.587M) : 0, 6587005
master (6.825M) : 0, 6825492
benchmarks/2.9.0 (7.552M) : 0, 7552193
section Trace stats
master (7.437M) : 0, 7437330
section Manual
master (9.999M) : 0, 9998601
section Manual + Automatic
This PR (5928) (6.317M) : 0, 6316857
master (6.024M) : 0, 6023637
section DD_TRACE_ENABLED=0
master (9.286M) : 0, 9286300
|
f2d5d45
to
da096bf
Compare
ee513c5
to
b3bff46
Compare
c37ad7d
to
7019ce6
Compare
This reverts commit 605e22b.
7019ce6
to
67bda22
Compare
## Summary of changes Adds the SSI denylist and initial testing ## Reason for change We want to bail out early if we don't support the platform. ## Implementation details Add a requirements.json file that specifies supported hosts and some of the patterns we expect to block or allow injection on ## Test coverage This includes a `requirements_json_test` stage. This will be ported to the one-pipeline once the `auto_inject` code is merged, this is a PoC that proves it works atm ## Other details Will backport this to 2.x as well
## Summary of changes Adds the SSI denylist and initial testing ## Reason for change We want to bail out early if we don't support the platform. ## Implementation details Add a requirements.json file that specifies supported hosts and some of the patterns we expect to block or allow injection on ## Test coverage This includes a `requirements_json_test` stage. This will be ported to the one-pipeline once the `auto_inject` code is merged, this is a PoC that proves it works atm ## Other details Backport of #5928
Summary of changes
Adds the SSI denylist and initial testing
Reason for change
We want to bail out early if we don't support the platform.
Implementation details
Add a requirements.json file that specifies supported hosts and some of the patterns we expect to block or allow injection on
Test coverage
This includes a
requirements_json_test
stage. This will be ported to the one-pipeline once theauto_inject
code is merged, this is a PoC that proves it works atmOther details
Will backport this to 2.x as well