Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Changes at 5/30/2022 5:18:40 PM #5868

Closed
performanceautofiler bot opened this issue Jun 7, 2022 · 1 comment
Closed

[Perf] Changes at 5/30/2022 5:18:40 PM #5868

performanceautofiler bot opened this issue Jun 7, 2022 · 1 comment
Assignees
@AndyAyersMS
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jun 7, 2022 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.TryAddGiventSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 9.87 μs 11.00 μs 1.11 0.02 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryAddGiventSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddGiventSize<String>.Dictionary(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.99983260233918 > 10.379600335817404.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -27.985454026071398 (T) = (0 -11161.806655564626) / Math.Sqrt((18171.264563411456 / (21)) + (62396.81664550035 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.12941891050354545 = (9882.787114471275 - 11161.806655564626) / 9882.787114471275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 108.82 ns 124.07 ns 1.14 0.06 False
IsMatch - Duration of single invocation 69.41 ns 82.95 ns 1.20 0.11 False
IsMatch - Duration of single invocation 118.61 ns 139.03 ns 1.17 0.05 False
IsMatch - Duration of single invocation 114.24 ns 129.23 ns 1.13 0.06 False
IsMatch - Duration of single invocation 124.22 ns 137.47 ns 1.11 0.07 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 10, Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 124.07029333694665 > 112.18285357223827.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/15/2022 3:27:46 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -17.659561207622726 (T) = (0 -125.78850456044614) / Math.Sqrt((11.069921722261116 / (21)) + (26.65810365859347 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.16801605427731614 = (107.69415719912769 - 125.78850456044614) / 107.69415719912769 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 1, Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 82.94520972092253 > 73.08364853189485.
IsChangePoint: Marked as a change because one of 4/22/2022 7:11:21 PM, 5/9/2022 11:56:12 AM, 5/11/2022 9:09:01 PM, 5/18/2022 4:10:31 AM, 5/27/2022 6:33:44 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -5.1530953800046 (T) = (0 -85.95155262244992) / Math.Sqrt((81.60278929021568 / (21)) + (4.037012106153998 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.13556524854349644 = (75.69054506792408 - 85.95155262244992) / 75.69054506792408 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 139.02711273942995 > 124.77740986557704.
IsChangePoint: Marked as a change because one of 5/11/2022 9:09:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -14.528949200403131 (T) = (0 -139.5461346234724) / Math.Sqrt((26.038615258660815 / (21)) + (13.25445868812534 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.146142049397908 = (121.75291421929668 - 139.5461346234724) / 121.75291421929668 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 9, Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.2276694595754 > 119.90458456311633.
IsChangePoint: Marked as a change because one of 5/11/2022 9:09:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -19.881590939517157 (T) = (0 -129.93844950740427) / Math.Sqrt((0.602214722474547 / (21)) + (29.928751821323413 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.13641773496881968 = (114.34039218947022 - 129.93844950740427) / 114.34039218947022 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.46837103762078 > 130.7152144980496.
IsChangePoint: Marked as a change because one of 5/10/2022 1:42:57 AM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -24.0720493565256 (T) = (0 -140.60649842535577) / Math.Sqrt((4.633622733854685 / (21)) + (7.967504829284167 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.11744425971045498 = (125.82864621971285 - 140.60649842535577) / 125.82864621971285 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 115.29 ns 137.06 ns 1.19 0.07 False
Parse - Duration of single invocation 117.15 ns 142.74 ns 1.22 0.07 False
TryParse - Duration of single invocation 111.68 ns 143.84 ns 1.29 0.12 False
TryParse - Duration of single invocation 52.00 ns 66.79 ns 1.28 0.07 False
TryParse - Duration of single invocation 105.78 ns 139.11 ns 1.32 0.12 False
Parse - Duration of single invocation 52.43 ns 59.04 ns 1.13 0.03 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.Parse(value: "1.7976931348623157e+308")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.0568498266847 > 118.42904804825866.
IsChangePoint: Marked as a change because one of 4/11/2022 1:47:52 PM, 4/14/2022 8:50:18 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -10.334340350362416 (T) = (0 -131.80169251558272) / Math.Sqrt((40.321951925641216 / (21)) + (25.28302716541836 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.1387831577717734 = (115.73906025575191 - 131.80169251558272) / 115.73906025575191 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.Parse(value: "-1.7976931348623157e+308")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 142.74121511431122 > 120.99850435845686.
IsChangePoint: Marked as a change because one of 3/30/2022 6:37:35 PM, 4/11/2022 1:47:52 PM, 4/14/2022 8:50:18 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -14.275495274950462 (T) = (0 -139.68790711519569) / Math.Sqrt((40.29464348616143 / (21)) + (16.917087763661282 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.18107050910384775 = (118.27228437122324 - 139.68790711519569) / 118.27228437122324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 143.83730024294363 > 116.80401521030954.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -13.652462853232594 (T) = (0 -140.33222749282402) / Math.Sqrt((61.162080317859434 / (21)) + (26.22889934670685 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.2196501308672507 = (115.05941248334771 - 140.33222749282402) / 115.05941248334771 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.TryParse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 66.79212207249705 > 54.53303994018309.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -19.83714901356965 (T) = (0 -60.52895437722216) / Math.Sqrt((0.16938005838816028 / (21)) + (9.845877897051679 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.1722997763865099 = (51.63265881001553 - 60.52895437722216) / 51.63265881001553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.TryParse(value: "1.7976931348623157e+308")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 139.10875162319203 > 111.68270757256214.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -22.877109968049588 (T) = (0 -134.3378070688334) / Math.Sqrt((17.450951657834228 / (21)) + (25.889819900769293 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.24537840120063906 = (107.86906769807601 - 134.3378070688334) / 107.86906769807601 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.Parse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.04179914094483 > 55.24612904387697.
IsChangePoint: Marked as a change because one of 5/11/2022 9:09:01 PM, 5/20/2022 10:18:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -24.738934628181553 (T) = (0 -58.10612080608749) / Math.Sqrt((0.4033465286847956 / (21)) + (1.1662041490321697 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.09568655713110737 = (53.031700013030864 - 58.10612080608749) / 53.031700013030864 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 8.07 ns 9.38 ns 1.16 0.03 False
TryParse - Duration of single invocation 20.36 ns 26.02 ns 1.28 0.04 False
TryParse - Duration of single invocation 15.75 ns 21.29 ns 1.35 0.03 False
ParseHex - Duration of single invocation 17.38 ns 18.38 ns 1.06 0.05 False
TryParse - Duration of single invocation 19.17 ns 28.61 ns 1.49 0.04 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.TryFormat(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.37799856842049 > 8.498350861161882.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -17.724318634580737 (T) = (0 -8.919489304094956) / Math.Sqrt((0.0004444891519090147 / (21)) + (0.1115883659989248 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.10302164759379544 = (8.086413647050827 - 8.919489304094956) / 8.086413647050827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryParse(value: "2147483647")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.017896092048296 > 21.365472259432316.
IsChangePoint: Marked as a change because one of 4/8/2022 8:59:46 PM, 4/9/2022 9:54:15 AM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -40.85155008877778 (T) = (0 -25.79118472708775) / Math.Sqrt((0.13984427597728366 / (67)) + (0.08814828878702677 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -0.28602654588064363 = (20.05493961979338 - 25.79118472708775) / 20.05493961979338 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryParse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.290688502785713 > 16.553673520973263.
IsChangePoint: Marked as a change because one of 3/30/2022 6:37:35 PM, 4/28/2022 4:25:37 PM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -29.32840935389612 (T) = (0 -20.88541396048852) / Math.Sqrt((0.24087354605105246 / (67)) + (0.17797818274137664 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -0.38503233096067585 = (15.079369263533462 - 20.88541396048852) / 15.079369263533462 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.ParseHex(value: "7FFFFFFF")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.38267020153511 > 18.260850227222154.
IsChangePoint: Marked as a change because one of 4/25/2022 5:27:54 PM, 4/28/2022 4:25:37 PM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -10.065469884070335 (T) = (0 -18.748892800050733) / Math.Sqrt((0.13664069342588522 / (67)) + (0.08526251015378862 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -0.08012283860872224 = (17.35811162386001 - 18.748892800050733) / 17.35811162386001 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryParse(value: "-2147483648")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.606607283299482 > 20.818654764652777.
IsChangePoint: Marked as a change because one of 3/30/2022 12:01:24 PM, 5/3/2022 11:42:39 AM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -17.791275883165387 (T) = (0 -28.143571248773707) / Math.Sqrt((0.35942667907018855 / (67)) + (0.9290557314094756 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -0.3819865555001683 = (20.364576729610796 - 28.143571248773707) / 20.364576729610796 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Decimal

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 75.22 ns 86.88 ns 1.16 0.06 False
Parse - Duration of single invocation 69.16 ns 84.46 ns 1.22 0.07 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Decimal*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Decimal.TryParse(value: "123456.789")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.88126959512908 > 77.02922650063351.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -13.264538653424081 (T) = (0 -86.42769899558687) / Math.Sqrt((13.735816049828893 / (21)) + (16.80500262717185 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.17953956201292576 = (73.27240372344524 - 86.42769899558687) / 73.27240372344524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Decimal.Parse(value: "123456.789")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 84.45614574277016 > 71.40289225034365.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -25.353179878127523 (T) = (0 -81.27025313299572) / Math.Sqrt((1.8328652754728065 / (21)) + (7.075182700492638 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.17413267063858984 = (69.21726578717401 - 81.27025313299572) / 69.21726578717401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Distinct - Duration of single invocation 2.12 μs 2.39 μs 1.12 0.03 True
Count - Duration of single invocation 256.41 ns 285.33 ns 1.11 0.19 False
ToDictionary - Duration of single invocation 1.88 μs 2.28 μs 1.21 0.18 False
Except - Duration of single invocation 2.73 μs 3.05 μs 1.12 0.13 False
Intersect - Duration of single invocation 3.31 μs 3.62 μs 1.09 0.09 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.Distinct(input: IEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3851745810055864 > 2.240086085214206.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -34.51877392427077 (T) = (0 -2426.1893046715304) / Math.Sqrt((268.2467991658867 / (21)) + (3033.478339031773 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.13757536552800953 = (2132.7723667305386 - 2426.1893046715304) / 2132.7723667305386 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Count(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 285.3330527840488 > 269.24427716325647.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -16.601026435680915 (T) = (0 -296.96140209897493) / Math.Sqrt((0.10455203912235275 / (21)) + (301.69007842752325 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.1574385101341319 = (256.5677567308185 - 296.96140209897493) / 256.5677567308185 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToDictionary(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2768355278276857 > 1.9744065370017274.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -68.68267413778204 (T) = (0 -2252.493686408211) / Math.Sqrt((217.01272602105442 / (21)) + (1001.2959460703461 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.20036395357270662 = (1876.5089368970096 - 2252.493686408211) / 1876.5089368970096 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Except(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0540807682563815 > 2.868999124943706.
IsChangePoint: Marked as a change because one of 5/31/2022 6:06:22 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -15.6746404826657 (T) = (0 -3044.42224242225) / Math.Sqrt((5971.528708813805 / (29)) + (3857.198357257714 / (43))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (43) - 2, .025) and -0.0971207152152939 = (2774.920024935293 - 3044.42224242225) / 2774.920024935293 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Intersect(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.6226016893554216 > 3.477522068783281.
IsChangePoint: Marked as a change because one of 6/1/2022 7:21:14 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -11.158577653665644 (T) = (0 -3585.9630530058885) / Math.Sqrt((9699.321246184028 / (33)) + (3477.4937798221026 / (39))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (39) - 2, .025) and -0.06485465055320434 = (3367.5610574109237 - 3585.9630530058885) / 3367.5610574109237 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 7, 2022 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ObjrefValueTypeObj - Duration of single invocation 278.56 μs 306.30 μs 1.10 0.41 False
CheckArrayIsInterfaceYes - Duration of single invocation 362.17 μs 573.62 μs 1.58 0.36 False
CheckArrayIsInterfaceNo - Duration of single invocation 362.20 μs 691.28 μs 1.91 0.32 False
CheckArrayIsArrayByVariance - Duration of single invocation 4.39 ns 9.57 ns 2.18 0.36 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.ObjrefValueTypeObj

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 306.30367647058824 > 302.6687016225962.
IsChangePoint: Marked as a change because one of 4/16/2022 10:12:41 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -8.204815192656149 (T) = (0 -351582.93497380876) / Math.Sqrt((492643492.6619276 / (21)) + (560538147.3100281 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.158713001625547 = (303425.3818508781 - 351582.93497380876) / 303425.3818508781 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.CheckArrayIsInterfaceYes

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 573.62285 > 496.993330965909.
IsChangePoint: Marked as a change because one of 5/25/2022 1:42:32 AM, 5/25/2022 12:28:02 PM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -3.71260354133692 (T) = (0 -882037.8551399332) / Math.Sqrt((6713049768.019545 / (67)) + (73593877615.0624 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -1.0508179056868012 = (430090.7714400642 - 882037.8551399332) / 430090.7714400642 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 691.2814285714286 > 380.3315170454546.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/16/2022 12:27:40 PM, 6/4/2022 9:47:11 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -5.501585362077305 (T) = (0 -713796.0848403106) / Math.Sqrt((25065816873.71303 / (58)) + (19438582000.17202 / (14))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (58) + (14) - 2, .025) and -0.49002664847320127 = (479049.2073223807 - 713796.0848403106) / 479049.2073223807 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.CheckArrayIsArrayByVariance

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.573037742190312 > 4.026671220021515.
IsChangePoint: Marked as a change because one of 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -13.289115928150755 (T) = (0 -9.049889008475782) / Math.Sqrt((0.7839697342367882 / (67)) + (0.684697740984232 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -1.3048750878269886 = (3.926411915453483 - 9.049889008475782) / 3.926411915453483 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 14.07 μs 15.23 μs 1.08 0.05 False
IDictionary - Duration of single invocation 15.27 μs 16.88 μs 1.11 0.06 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<String>.Dictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.232415358194306 > 14.772190939654704.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -22.621840728291144 (T) = (0 -15158.208052217049) / Math.Sqrt((39501.550908917125 / (21)) + (75343.47661924634 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.09467330463444142 = (13847.243728373396 - 15158.208052217049) / 13847.243728373396 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.88184594062947 > 15.949866279737485.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -13.207038789146917 (T) = (0 -16869.280055114785) / Math.Sqrt((190613.1134855347 / (21)) + (282441.3815096337 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.10454146304730527 = (15272.654417675136 - 16869.280055114785) / 15272.654417675136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.IO.Tests.StreamReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLineAsync - Duration of single invocation 27.58 μs 33.59 μs 1.22 0.24 False
ReadLineAsync - Duration of single invocation 21.17 μs 23.65 μs 1.12 0.15 False
ReadLineAsync - Duration of single invocation 20.18 μs 27.91 μs 1.38 0.19 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 33, 128])

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.58951086956522 > 29.171114914809912.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 5/20/2022 10:18:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -20.813521394062043 (T) = (0 -34197.97462720035) / Math.Sqrt((459952.24024780653 / (21)) + (3604544.111505334 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.2272708719807115 = (27865.058487054048 - 34197.97462720035) / 27865.058487054048 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.64529125755287 > 22.12938946558622.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/25/2022 11:05:49 AM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -18.685346381555075 (T) = (0 -26948.950264263098) / Math.Sqrt((6011.272542998564 / (21)) + (5027432.283948461 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.2787891883693917 = (21073.802085100684 - 26948.950264263098) / 21073.802085100684 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.91082105864945 > 21.19849181569689.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 5/20/2022 10:18:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -12.650783817362939 (T) = (0 -24032.872775865366) / Math.Sqrt((4126.2549541590515 / (21)) + (4697763.96955042 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.19037961518853935 = (20189.250949209927 - 24032.872775865366) / 20189.250949209927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 676.06 μs 717.77 μs 1.06 0.15 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 717.7744973776224 > 710.0656927083332.
IsChangePoint: Marked as a change because one of 3/31/2022 1:45:25 PM, 4/17/2022 3:25:40 PM, 5/11/2022 9:09:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -8.171651695104924 (T) = (0 -738442.8063496478) / Math.Sqrt((556096320.3473651 / (21)) + (632034986.5059437 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.07410861110515576 = (687493.6097848245 - 738442.8063496478) / 687493.6097848245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 9.89 μs 11.07 μs 1.12 0.02 True
IDictionary - Duration of single invocation 11.08 μs 12.13 μs 1.09 0.03 True

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.Dictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.068681473323542 > 10.365152327286724.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -30.08971798424474 (T) = (0 -11171.074858912123) / Math.Sqrt((16992.968838300032 / (21)) + (56778.13673461569 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.13391709764066437 = (9851.756254628952 - 11171.074858912123) / 9851.756254628952 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.AddGivenSize&lt;String&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.125055686582808 > 11.538787626478586.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -30.532904793271886 (T) = (0 -12505.514311610506) / Math.Sqrt((18205.6451115703 / (21)) + (81070.00358446411 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.13767208441146123 = (10992.195803134116 - 12505.514311610506) / 10992.195803134116 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.TryAddDefaultSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 13.50 μs 14.48 μs 1.07 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryAddDefaultSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddDefaultSize<String>.Dictionary(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.484335346619831 > 14.036467226416.
IsChangePoint: Marked as a change because one of 4/8/2022 8:59:46 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -26.78430523942629 (T) = (0 -14666.22142041102) / Math.Sqrt((24776.56176000642 / (21)) + (66736.89300193748 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.10023209273117992 = (13330.11599762017 - 14666.22142041102) / 13330.11599762017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@AndyAyersMS AndyAyersMS mentioned this issue Jun 7, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AndyAyersMS AndyAyersMS

Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AndyAyersMS