Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf -138%] System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.ProducerConsumer #39031

Closed
DrewScoggins opened this issue Jul 9, 2020 · 9 comments
Closed

[Perf -138%] System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.ProducerConsumer #39031

DrewScoggins opened this issue Jul 9, 2020 · 9 comments
Labels
arch-x64 arch-x86 area-System.Buffers os-linux Linux OS (any supported distro) os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
Future

Comments

@DrewScoggins
Copy link
Member

DrewScoggins commented Jul 9, 2020 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.18362
Changes diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests

Benchmark Baseline Test Test/Base Modality Baseline Outlier
ProducerConsumer 227.31 ns 567.92 ns 2.50 False

graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests<Byte>*';

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

[158.326 ; 189.781) | @
[189.781 ; 253.830) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[253.830 ; 311.112) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[311.112 ; 370.078) | @@@@@@@@@@@@@@@
[370.078 ; 431.574) | @@@@@@@@@@@@@
[431.574 ; 489.706) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[489.706 ; 546.987) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[546.987 ; 593.916) | @@@@@@@@@
[593.916 ; 651.198) | 
[651.198 ; 705.890) | 
[705.890 ; 763.172) | @
[763.172 ; 783.237) | 
[783.237 ; 840.519) | @

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added area-System.Buffers untriaged New issue has not been triaged by the area owner labels Jul 9, 2020
@ghost
Copy link

ghost commented Jul 9, 2020

Tagging subscribers to this area: @tannergooding, @pgovind
Notify danmosemsft if you want to be subscribed.

@DrewScoggins DrewScoggins added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark and removed tenet-performance-benchmarks Issue from performance benchmark labels Jul 9, 2020
@stephentoub
Copy link
Member

For reference, this is what I get on my machine currently:
dotnet run -c Release -f netcoreapp3.1 --runtimes netcoreapp3.1 netcoreapp5.0 --filter="System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer"

Method Job Runtime Toolchain RentalSize ManipulateArray Async UseSharedPool Mean Error StdDev Median Min Max Ratio
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 False False False 994.7 ns 6.55 ns 6.13 ns 993.4 ns 982.3 ns 1,007.0 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 False False False 984.5 ns 23.23 ns 26.75 ns 973.9 ns 943.6 ns 1,027.3 ns 1.00
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 False False True 900.9 ns 72.32 ns 83.29 ns 915.1 ns 751.2 ns 993.2 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 False False True 973.5 ns 32.02 ns 35.59 ns 975.9 ns 897.1 ns 1,033.0 ns 1.08
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 False True False 787.2 ns 14.46 ns 12.82 ns 784.0 ns 773.0 ns 818.0 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 False True False 805.4 ns 24.08 ns 27.73 ns 790.8 ns 774.4 ns 868.0 ns 1.01
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 False True True 212.8 ns 33.19 ns 38.22 ns 211.7 ns 157.5 ns 276.1 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 False True True 329.0 ns 39.39 ns 43.78 ns 328.4 ns 265.7 ns 414.3 ns 1.58
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 True False False 2,025.1 ns 37.84 ns 37.16 ns 2,011.9 ns 1,981.1 ns 2,098.5 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 True False False 2,104.4 ns 50.89 ns 54.45 ns 2,111.0 ns 1,999.8 ns 2,219.6 ns 1.04
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 True False True 2,120.9 ns 115.41 ns 132.90 ns 2,096.0 ns 1,939.9 ns 2,355.9 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 True False True 2,112.0 ns 94.69 ns 109.04 ns 2,098.3 ns 1,948.9 ns 2,338.6 ns 1.00
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 True True False 2,038.7 ns 9.23 ns 8.18 ns 2,036.6 ns 2,022.9 ns 2,055.0 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 True True False 2,004.0 ns 13.64 ns 12.09 ns 2,003.6 ns 1,978.2 ns 2,022.3 ns 0.98
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 True True True 2,188.2 ns 36.04 ns 33.71 ns 2,189.6 ns 2,132.2 ns 2,253.4 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 True True True 2,178.3 ns 43.93 ns 50.59 ns 2,168.9 ns 2,068.6 ns 2,259.2 ns 1.00

I'm assuming the one that stands out is the same as the one called out in this issue, but I can't see which inputs to the theory are being cited in the issue (other than "4096"... everything else is cut off in the image):

Method Job Runtime Toolchain RentalSize ManipulateArray Async UseSharedPool Mean Error StdDev Median Min Max Ratio
ProducerConsumer Job-VPWHSS .NET Core 3.1 netcoreapp3.1 4096 False True True 212.8 ns 33.19 ns 38.22 ns 211.7 ns 157.5 ns 276.1 ns 1.00
ProducerConsumer Job-HIDTAI .NET Core 5.0 netcoreapp5.0 4096 False True True 329.0 ns 39.39 ns 43.78 ns 328.4 ns 265.7 ns 414.3 ns 1.58

@DrewScoggins
Copy link
Member Author

DrewScoggins commented Jul 10, 2020 •
edited
Loading

This is the one where we are seeing a regression and it does match up with what you are seeing in your local testing. I found this by clicking on the Historical Data in Reporting System link in the issue, but I will also file a bug for fixing title font size on long tests names.

System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

@DrewScoggins
Copy link
Member Author

DrewScoggins commented Jul 10, 2020 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Changes diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests

Benchmark Baseline Test Test/Base Modality Baseline Outlier
ProducerConsumer 443.11 ns 634.82 ns 1.43 Bimodal False
ProducerConsumer 615.21 ns 751.29 ns 1.22 True

Related Issue on x64 Windows

[Perf -138%] System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer

Related Issue on x86 Windows

[Perf -359%] System.Buffers.Tests.RentReturnArrayPoolTests (2)

graph
graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests<Byte>*';

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

[362.360 ; 402.651) | @@
[402.651 ; 444.857) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[444.857 ; 461.748) | @@@
[461.748 ; 513.730) | @@@@@@@@@@@@@
[513.730 ; 563.598) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[563.598 ; 611.912) | @@@@@@@@@@@@@@@@@@@@
[611.912 ; 654.119) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[654.119 ; 693.997) | @@@@@@@@@@@@@
[693.997 ; 714.683) | @
[714.683 ; 756.890) | @@@@@@@@@@@@
[756.890 ; 786.963) | @

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)

[464.052 ; 492.206) | @
[492.206 ; 528.709) | @@@@@@
[528.709 ; 562.585) | @@@@@@@@@@@@@@@@@@
[562.585 ; 585.725) | @@@@@
[585.725 ; 619.601) | @@@@@@@@@@@@@@@@@@@
[619.601 ; 657.384) | @@@@@@@@@
[657.384 ; 671.155) | @
[671.155 ; 705.031) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[705.031 ; 739.166) | @@@@@@@@@@@@@@@@@@@@@@@@@@@
[739.166 ; 773.042) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[773.042 ; 809.388) | @@@@@@@@@@@@@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@DrewScoggins
Copy link
Member Author

We are also seeing this behavior on Ubuntu x64 and Windows x86

@DrewScoggins DrewScoggins added arch-x86 os-linux Linux OS (any supported distro) labels Jul 10, 2020
@adamsitnik
Copy link
Member

The data I have right now seems to be showing a regression for ManipulateArray: False, Async: True for x86 and Unix. But I am not 100% convinced that this is an actual regression, I am going to update the results as soon as I get more results from the Team.

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)

Conclusion Base Diff Base/Diff Modality Operating System Bit Processor Name Base Runtime Diff Runtime
Same 2228.67 2173.95 1.03 bimodal Windows 10.0.18363.1016 Arm Microsoft SQ1 3.0 GHz .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 940.89 963.98 0.98 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20404.3
Same 949.94 977.49 0.97 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20418.3
Slower 364.64 659.28 0.55 several? Windows 10.0.19041.450 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 926.69 926.18 1.00 Windows 10.0.19041.450 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake) .NET Core 3.1.6 5.0.100-rc.1.20419.9
Same 765.88 763.45 1.00 Windows 10.0.19042 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3
Slower 202.26 965.26 0.21 Windows 10.0.18363.959 X86 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20420.2
Slower 158.22 687.73 0.23 bimodal Windows 10.0.19041.450 X86 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20419.5
Slower 493.37 829.38 0.59 bimodal ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20403.23
Slower 342.21 617.43 0.55 bimodal macOS Mojave 10.14.5 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20404.2
Slower 593.00 741.68 0.80 several? ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

Conclusion Base Diff Base/Diff Modality Operating System Bit Processor Name Base Runtime Diff Runtime
Faster 418.83 379.40 1.10 bimodal Windows 10.0.18363.1016 Arm Microsoft SQ1 3.0 GHz .NET Core 3.1.6 5.0.100-rc.1.20413.9
Slower 308.91 486.95 0.63 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20404.3
Slower 407.75 553.97 0.74 bimodal Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20418.3
Slower 211.70 316.03 0.67 Windows 10.0.19041.450 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20413.9
Slower 473.94 675.91 0.70 bimodal Windows 10.0.19041.450 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake) .NET Core 3.1.6 5.0.100-rc.1.20419.9
Slower 218.47 343.43 0.64 Windows 10.0.19042 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3
Slower 229.96 396.16 0.58 bimodal Windows 10.0.19041.450 X86 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20419.5
Slower 789.21 1021.56 0.77 bimodal ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20403.23
Same 344.58 327.46 1.05 macOS Mojave 10.14.5 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20404.2
Slower 478.20 592.82 0.81 several? ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True)

Conclusion Base Diff Base/Diff Modality Operating System Bit Processor Name Base Runtime Diff Runtime
Slower 8257.35 11626.06 0.71 bimodal Windows 10.0.18363.1016 Arm Microsoft SQ1 3.0 GHz .NET Core 3.1.6 5.0.100-rc.1.20413.9
Slower 925.19 1321.75 0.70 bimodal Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20404.3
Same 1264.09 1298.14 0.97 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20418.3
Slower 539.34 1072.44 0.50 several? Windows 10.0.19041.450 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 2150.26 2199.10 0.98 bimodal Windows 10.0.19041.450 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake) .NET Core 3.1.6 5.0.100-rc.1.20419.9
Slower 1061.84 1149.02 0.92 bimodal Windows 10.0.19042 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3
Slower 636.82 1142.47 0.56 bimodal Windows 10.0.19041.450 X86 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20419.5
Same 1507.05 1508.84 1.00 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20403.23
Slower 437.72 939.25 0.47 several? macOS Mojave 10.14.5 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20404.2
Same 26582.74 26372.16 1.01 ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

Conclusion Base Diff Base/Diff Modality Operating System Bit Processor Name Base Runtime Diff Runtime
Slower 11132.73 12997.98 0.86 bimodal Windows 10.0.18363.1016 Arm Microsoft SQ1 3.0 GHz .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 1241.86 1277.65 0.97 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20404.3
Same 1246.25 1239.08 1.01 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20418.3
Same 1144.25 1080.37 1.06 Windows 10.0.19041.450 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 1903.41 2006.26 0.95 bimodal Windows 10.0.19041.450 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake) .NET Core 3.1.6 5.0.100-rc.1.20419.9
Same 1065.02 1044.87 1.02 Windows 10.0.19042 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3
Same 1341.31 1260.60 1.06 Windows 10.0.18363.959 X86 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20420.2
Slower 798.30 1140.56 0.70 several? Windows 10.0.19041.450 X86 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20419.5
Same 1451.78 1523.89 0.95 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20403.23
Same 1097.94 1103.37 1.00 several? macOS Mojave 10.14.5 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20404.2
Same 26549.57 26185.19 1.01 ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: True, Async: True, UseSharedPool: True)

Conclusion Base Diff Base/Diff Modality Operating System Bit Processor Name Base Runtime Diff Runtime
Slower 2252.46 3235.73 0.70 Windows 10.0.18363.1016 Arm Microsoft SQ1 3.0 GHz .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 2046.57 2064.27 0.99 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20404.3
Same 2070.63 2084.10 0.99 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20418.3
Same 2584.14 2538.22 1.02 several? Windows 10.0.19041.450 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20413.9
Slower 3105.05 3531.59 0.88 Windows 10.0.19041.450 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake) .NET Core 3.1.6 5.0.100-rc.1.20419.9
Same 2353.13 2385.56 0.99 Windows 10.0.19042 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3
Same 2480.44 2424.03 1.02 bimodal Windows 10.0.19041.450 X86 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20419.5
Same 3116.39 3221.75 0.97 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20403.23
Same 2662.47 2478.84 1.07 several? macOS Mojave 10.14.5 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20404.2
Faster 4633.61 2659.22 1.74 ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: True, Async: True, UseSharedPool: False)

Conclusion Base Diff Base/Diff Modality Operating System Bit Processor Name Base Runtime Diff Runtime
Same 2134.10 2115.18 1.01 Windows 10.0.18363.1016 Arm Microsoft SQ1 3.0 GHz .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 1895.13 1897.03 1.00 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20404.3
Same 1925.69 1900.68 1.01 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20418.3
Same 2562.60 2493.37 1.03 Windows 10.0.19041.450 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20413.9
Faster 2602.41 2410.16 1.08 Windows 10.0.19041.450 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake) .NET Core 3.1.6 5.0.100-rc.1.20419.9
Same 2232.11 2116.15 1.05 Windows 10.0.19042 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3
Same 1693.92 1703.66 0.99 Windows 10.0.18363.959 X86 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20420.2
Same 2281.45 2410.88 0.95 bimodal Windows 10.0.19041.450 X86 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20419.5
Same 2842.03 3060.01 0.93 bimodal ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20403.23
Same 2476.54 2413.86 1.03 bimodal macOS Mojave 10.14.5 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20404.2
Faster 4142.03 2190.20 1.89 bimodal ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: True, Async: False, UseSharedPool: False)

Conclusion Base Diff Base/Diff Modality Operating System Bit Processor Name Base Runtime Diff Runtime
Faster 7247.91 2478.30 2.92 several? Windows 10.0.18363.1016 Arm Microsoft SQ1 3.0 GHz .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 1926.56 1893.37 1.02 bimodal Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20404.3
Slower 1894.28 2113.88 0.90 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20418.3
Same 2354.47 2379.02 0.99 Windows 10.0.19041.450 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 3715.02 3783.27 0.98 bimodal Windows 10.0.19041.450 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake) .NET Core 3.1.6 5.0.100-rc.1.20419.9
Same 2414.21 2561.07 0.94 Windows 10.0.19042 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3
Same 1479.98 1577.31 0.94 several? Windows 10.0.18363.959 X86 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20420.2
Same 1828.98 1748.78 1.05 several? Windows 10.0.19041.450 X86 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20419.5
Same 2134.12 2277.96 0.94 bimodal ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20403.23
Same 2486.87 2423.00 1.03 macOS Mojave 10.14.5 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20404.2
Same 11013.15 12048.09 0.91 several? ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: True, Async: False, UseSharedPool: True)

Conclusion Base Diff Base/Diff Modality Operating System Bit Processor Name Base Runtime Diff Runtime
Faster 13894.59 4954.74 2.80 several? Windows 10.0.18363.1016 Arm Microsoft SQ1 3.0 GHz .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 2278.41 2327.73 0.98 Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20404.3
Same 2540.99 2474.26 1.03 several? Windows 10.0.18363.959 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20418.3
Faster 3062.12 2608.76 1.17 bimodal Windows 10.0.19041.450 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20413.9
Same 3079.30 3286.40 0.94 bimodal Windows 10.0.19041.450 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake) .NET Core 3.1.6 5.0.100-rc.1.20419.9
Same 2632.63 2505.72 1.05 several? Windows 10.0.19042 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3
Slower 1897.96 2143.81 0.89 several? Windows 10.0.19041.450 X86 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20419.5
Same 2281.83 2292.53 1.00 bimodal ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz .NET Core 3.1.6 5.0.100-rc.1.20403.23
Same 2724.23 2608.74 1.04 bimodal macOS Mojave 10.14.5 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) .NET Core 3.1.6 5.0.100-rc.1.20404.2
Same 8893.52 8823.53 1.01 bimodal ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) .NET Core 3.1.6 5.0.100-rc.1.20418.3

@adamsitnik adamsitnik added this to the Future milestone Aug 20, 2020
@adamsitnik
Copy link
Member

@DrewScoggins could you please provide a link to full historical data (I've tried to guess it but I've failed)?

@DrewScoggins
Copy link
Member Author

I have updated the table in the issue at the top with links to the full test history in the runtime repo.

@adamsitnik
Copy link
Member

@DrewScoggins thanks a lot!

It looks like there is no regression and the benchmark is:

unstable for x64

obraz

and bimodal for x86:

obraz

@adamsitnik adamsitnik mentioned this issue Sep 4, 2020
Closed
21 tasks
@jeffhandley jeffhandley removed the untriaged New issue has not been triaged by the area owner label Sep 17, 2020
@jkotas jkotas mentioned this issue Oct 5, 2020
Open
@ghost ghost locked as resolved and limited conversation to collaborators Dec 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 arch-x86 area-System.Buffers os-linux Linux OS (any supported distro) os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
Future
Development

No branches or pull requests
5 participants
@jeffhandley @stephentoub @adamsitnik @DrewScoggins @Dotnet-GitSync-Bot