Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 11 Regressions on 1/15/2023 6:48:05 PM #11808

Open
performanceautofiler bot opened this issue Jan 17, 2023 · 3 comments
Open

[Perf] Linux/x64: 11 Regressions on 1/15/2023 6:48:05 PM #11808

performanceautofiler bot opened this issue Jan 17, 2023 · 3 comments
Labels
arch-wasm arch-x64 branch-refs/heads/main kind-micro os-linux perf-regression untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 17, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 359.97 ns 492.86 ns 1.37 0.17 False
ToString - Duration of single invocation 651.85 ns 729.21 ns 1.12 0.07 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.TryFormat(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 492.8608440749241 > 375.545925450121.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -25.84457571190944 (T) = (0 -502.44390338617205) / Math.Sqrt((983.1173759194976 / (47)) + (178.30350063196926 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.4709948723134669 = (341.56740641520196 - 502.44390338617205) / 341.56740641520196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.ToString(value: -2147483648)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 729.2063155758293 > 684.3506826283758.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/15/2023 7:15:22 AM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -10.20566597903779 (T) = (0 -722.3251669528804) / Math.Sqrt((5026.290572474745 / (46)) + (141.95231641833035 / (10))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (10) - 2, .025) and -0.18622758522786528 = (608.9262936961015 - 722.3251669528804) / 608.9262936961015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf8 - Duration of single invocation 71.44 ms 90.50 ms 1.27 0.05 False
WriteBasicUtf8 - Duration of single invocation 52.14 ms 75.38 ms 1.45 0.05 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 90.49589283928572 > 73.034914943125.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -43.82252103494706 (T) = (0 -91814303.24142857) / Math.Sqrt((1044661612834.847 / (46)) + (2194022181497.7998 / (10))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (10) - 2, .025) and -0.30693774754127945 = (70251474.038574 - 91814303.24142857) / 70251474.038574 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 75.38498208928571 > 54.62824496634616.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -65.4121511111436 (T) = (0 -74875062.99024725) / Math.Sqrt((1754074128119.2312 / (46)) + (757653307401.3975 / (10))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (10) - 2, .025) and -0.41810652770680706 = (52799321.86147276 - 74875062.99024725) / 52799321.86147276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat2 - Duration of single invocation 760.17 ns 1.12 μs 1.47 0.12 False
ToString2 - Duration of single invocation 1.10 μs 1.43 μs 1.30 0.04 False
ToString3 - Duration of single invocation 1.47 μs 1.90 μs 1.30 0.05 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.TryFormat2

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1184515228533525 > 724.2589986636451.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -86.24907913695705 (T) = (0 -1087.2573937011782) / Math.Sqrt((226.79804288115042 / (46)) + (159.77626131237326 / (10))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (10) - 2, .025) and -0.5691834273522646 = (692.881007248301 - 1087.2573937011782) / 692.881007248301 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4267705436460205 > 1.1659301484824813.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -28.963582231678842 (T) = (0 -1402.8259962134505) / Math.Sqrt((4483.3668800814485 / (46)) + (341.5761509150203 / (10))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (10) - 2, .025) and -0.3103953822855898 = (1070.5364313529885 - 1402.8259962134505) / 1070.5364313529885 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9020946846902582 > 1.5187384945824778.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -46.43041809205259 (T) = (0 -1902.88170383561) / Math.Sqrt((5319.790771938327 / (47)) + (79.24576453853257 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.3670995365588348 = (1391.9116003984668 - 1902.88170383561) / 1391.9116003984668 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterInt64 - Duration of single invocation 777.56 ns 981.08 ns 1.26 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 9223372036854775807)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 981.0816089963674 > 798.1077434198894.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -32.89379017734937 (T) = (0 -998.0796408524504) / Math.Sqrt((133.1485354529759 / (47)) + (450.97475587610467 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.2956183978865858 = (770.3500062059315 - 998.0796408524504) / 770.3500062059315 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf16 - Duration of single invocation 175.98 ms 200.27 ms 1.14 0.05 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 200.27485742857144 > 188.79953076923078.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 12/9/2022 1:52:32 PM, 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -21.08812700573864 (T) = (0 -202543721.80666667) / Math.Sqrt((12700212976706.117 / (47)) + (10186352047534.66 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.1340451753872421 = (178602868.9179019 - 202543721.80666667) / 178602868.9179019 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringD - Duration of single invocation 659.78 ns 1.03 μs 1.56 0.13 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: 123)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0287410134526986 > 712.3003201155551.
IsChangePoint: Marked as a change because one of 12/1/2022 11:48:13 PM, 12/9/2022 1:52:32 PM, 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -24.12521454824905 (T) = (0 -1013.3107592956) / Math.Sqrt((3672.7741414237175 / (47)) + (792.2529223974778 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.425858241708403 = (710.6672526446205 - 1013.3107592956) / 710.6672526446205 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 388.84 ns 566.44 ns 1.46 0.16 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.TryFormat(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 566.4432835672932 > 407.215650687202.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/15/2023 4:08:29 PM, 1/17/2023 11:08:46 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 11:08:46 AM.
IsRegressionStdDev: Marked as regression because -31.97460420023956 (T) = (0 -551.4386234994943) / Math.Sqrt((664.1440875497849 / (47)) + (155.45488516080795 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.4617164935130571 = (377.254156976897 - 551.4386234994943) / 377.254156976897 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@lewing
Copy link
Member

lewing commented Jan 17, 2023

there have been some changes that may not be visible in the diff but dotnet/runtime#80668 may actually be the cause of the this regression

@lewing
Copy link
Member

lewing commented Jan 17, 2023

cc @EgorBo

@EgorBo
Copy link
Member

EgorBo commented Jan 17, 2023

I filed dotnet/runtime#80745 for this one, although, I can't repro it locally yet 😕

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-wasm arch-x64 branch-refs/heads/main kind-micro os-linux perf-regression untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lewing @EgorBo