Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port CoreClr benchmarks to BenchmarkDotNet #36

Merged
merged 107 commits into from
Jun 21, 2018
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
107 commits
Select commit Hold shift + click to select a range
9476fed
copy some coreclr benchmarks
adamsitnik May 23, 2018
5071775
port some some benchmarks
adamsitnik May 23, 2018
f4b81c3
copy some coreclr benchmarks 2
adamsitnik May 23, 2018
cf5951f
port some some benchmarks 2
adamsitnik May 23, 2018
df07e67
rename the folder with CoreClr benchmarks to BenchmarksGame
adamsitnik May 25, 2018
46070ae
fix SpectralNorm_1 benchmark
adamsitnik May 25, 2018
702be05
copy knucleotide benchmarks
adamsitnik May 25, 2018
1761381
get knucleotide working
adamsitnik May 25, 2018
54ff968
copy RegexRedux
adamsitnik May 25, 2018
7b1d96e
make RegexRedux work
adamsitnik May 25, 2018
2e40808
copy reverse-complement
adamsitnik May 25, 2018
2e4b9b6
make reverse-complement work
adamsitnik May 25, 2018
052fe9f
copy all .txt files to bin
adamsitnik May 25, 2018
ea468a3
update to the latest version
adamsitnik May 28, 2018
201ba6b
read the file length, dont hardcode it (it was buggy)
adamsitnik May 28, 2018
141f2f8
use Description to give benchmarks some nice display name, but preser…
adamsitnik May 29, 2018
79a21f8
remove the Rider files from repo, ignore them
adamsitnik May 29, 2018
ffc9fb2
dont run more than 20 iterations
adamsitnik May 29, 2018
2c5b539
add possibility to run benchmarks for Legacy Jits
adamsitnik May 29, 2018
0ec180c
copy all benchstones
adamsitnik May 29, 2018
5eff7f1
port Benchstones to BenchmarkDotNet
adamsitnik May 29, 2018
475793f
change the default iteration time from 0,5s to 0,25s to run benchmark…
adamsitnik May 29, 2018
78bd131
update to the version which supports jagged arrays
adamsitnik May 30, 2018
be2104e
copy Burgers
adamsitnik May 30, 2018
8581484
port Burgers
adamsitnik May 30, 2018
79675a2
copy DefaultEqualityComparerPerf
adamsitnik May 30, 2018
5e00870
port DefaultEqualityComparerPerf
adamsitnik May 30, 2018
cf2a12d
copy FractalPerf
adamsitnik May 30, 2018
eafa45c
port FractalPerf
adamsitnik May 30, 2018
3cf971a
copy Inlining benchmarks
adamsitnik May 30, 2018
5ecfcfc
port Inlining benchmarks
adamsitnik May 30, 2018
2e64eab
rename Jit folder to Devirtualization to match the coreclr folder str…
adamsitnik May 30, 2018
121d8b9
copy SearchLoops
adamsitnik May 30, 2018
945a91f
port SearchLoops
adamsitnik May 30, 2018
e2614df
copy Linq
adamsitnik May 30, 2018
5a9754e
port Linq
adamsitnik May 30, 2018
b6190e0
copy perflab
adamsitnik May 30, 2018
a79729c
port BlockCopyPerf
adamsitnik May 30, 2018
7a09b0f
port CastingPerf
adamsitnik May 30, 2018
d4534e5
port CastingPerf2
adamsitnik May 30, 2018
902832a
port DelegatePerf
adamsitnik May 30, 2018
0640e17
remove Program.cs
adamsitnik May 30, 2018
518fe86
port EnumPerf
adamsitnik May 30, 2018
1959317
fix the benchmarks
adamsitnik May 30, 2018
f7f0ae9
port StackWalk
adamsitnik May 30, 2018
0ad84ad
port ThreadingPerf
adamsitnik May 30, 2018
0324338
port LowLevelPerf
adamsitnik May 30, 2018
fff3e4e
port ReflectionPerf
adamsitnik Jun 3, 2018
bbd6468
copy V8.Crypto and V8.Richards
adamsitnik Jun 3, 2018
e3e4bbf
port V8.Crypto
adamsitnik Jun 3, 2018
9cd02f8
port V8.Richards
adamsitnik Jun 3, 2018
e6a79b5
copy SpanBench
adamsitnik Jun 3, 2018
a838265
port Span/Indexer
adamsitnik Jun 4, 2018
6e6bb8f
port SpanBench
adamsitnik Jun 4, 2018
7a19d6f
update dependencies (which contain required fixes)
adamsitnik Jun 5, 2018
26ff77e
copy SIMD benchmarks
adamsitnik Jun 5, 2018
f0cc07a
bump .NET to 4.6.1 so we can use Vector.LessThanOrEqualAny
adamsitnik Jun 5, 2018
57f3f84
port ConsoleMandel
adamsitnik Jun 5, 2018
1a5384f
port RayTracerBench
adamsitnik Jun 5, 2018
2eb2ea1
port SeekUnroll
adamsitnik Jun 5, 2018
ef09660
move all internal types of RayTracer to a separate namespace to avoid…
adamsitnik Jun 5, 2018
94ea670
move all the CoreCLR benchmarks to coreclr subfolder
adamsitnik Jun 5, 2018
fa0083b
copy CscBench
adamsitnik Jun 5, 2018
adee6ce
port CscBench
adamsitnik Jun 5, 2018
c0eb4a5
copy SciMark
adamsitnik Jun 5, 2018
128d584
port SciMark
adamsitnik Jun 5, 2018
a3a5e33
copy ByteMark
adamsitnik Jun 5, 2018
c79f955
port ByteMark
adamsitnik Jun 5, 2018
4aafc15
copy Math
adamsitnik Jun 5, 2018
4658e1d
port Math
adamsitnik Jun 5, 2018
5404062
dont try to compile Single type benchmarks which use MathF which does…
adamsitnik Jun 5, 2018
225d09d
fix line endings ;)
adamsitnik Jun 7, 2018
655fc1c
group the benchmarks into categories, allow filtering and joining int…
adamsitnik Jun 7, 2018
572c2ab
export to json (for BenchView intergration purpose), allow users to s…
adamsitnik Jun 12, 2018
cff83e7
always show Min and Max column to mimic xunit-performance behaviour
adamsitnik Jun 12, 2018
09f88e7
assign the right type to a category (the one which actually contains …
adamsitnik Jun 13, 2018
be7d67d
the benchmarks should not be static
adamsitnik Jun 13, 2018
f2d14ce
read the array from field to preserve old benchmark id
adamsitnik Jun 13, 2018
fad4695
don't use argument, keep old benchmark full name
adamsitnik Jun 13, 2018
e029abb
Merge remote-tracking branch 'upstream/master' into coreClrBdn
adamsitnik Jun 13, 2018
f582530
get correct allocated bytes after a bug fix in BenchmarkDotNet
adamsitnik Jun 13, 2018
d62beb0
don't use argument, keep old benchmark full name + the value was 16, …
adamsitnik Jun 13, 2018
2387962
missing * 2!!
adamsitnik Jun 13, 2018
e891e79
update to BDN which uses full type name for exported results (we have…
adamsitnik Jun 14, 2018
fdf84dc
move the arguments to fields to match existing benchmark id
adamsitnik Jun 14, 2018
3d5254c
don't use argument, keep old benchmark full name
adamsitnik Jun 14, 2018
8df41a9
expectedSum argument needs to remain to keep old benchmark id in Benc…
adamsitnik Jun 14, 2018
52e6d8e
don't use argument, keep old benchmark full name
adamsitnik Jun 14, 2018
c3a4ed0
move the arguments to fields to match existing benchmark id
adamsitnik Jun 14, 2018
5be5f96
add missing category
adamsitnik Jun 14, 2018
ed48860
use Params from BDN to express things that were arguments used to set…
adamsitnik Jun 14, 2018
66cc17f
should be a part of previous commit
adamsitnik Jun 15, 2018
ccba0e1
don't make PerfLabTests nano-benchmarks, it would blow up the scaling…
adamsitnik Jun 15, 2018
e175d9e
make sure ALL ids are the same
adamsitnik Jun 15, 2018
503e8d8
ResultsValidator - small helper utility to print a diff, I will remov…
adamsitnik Jun 15, 2018
a18473e
remove the workaround for handling whitespaced in benchmarks Ids, pro…
adamsitnik Jun 18, 2018
0a0cd71
allow the users to specify outlier removal mode from console args
adamsitnik Jun 18, 2018
afadbd8
add median to the default columns
adamsitnik Jun 19, 2018
56bec57
allow the users to test COMPlus_JitAlignLoop 0 vs 1 and affinity set …
adamsitnik Jun 19, 2018
d5c3bf0
make sure JIT does not optimize the benchmarks to empty loops!
adamsitnik Jun 19, 2018
84e0822
update to latest BDN which gives us some nice attributes
adamsitnik Jun 20, 2018
e38e951
apply some non-default settings to make the results from multimodal b…
adamsitnik Jun 20, 2018
cd0fa8f
add comments to benchmarks whic hare very dependent on loop alignment
adamsitnik Jun 20, 2018
101bc24
fix ObjectGetTypeNoBoxing, disable ObjectGetType which is getting opt…
adamsitnik Jun 20, 2018
d0e0250
change the order of exported columns in CSV to make it more easy to c…
adamsitnik Jun 20, 2018
21da2d6
rename console attributes, don't introduce new standards ;) + don't w…
adamsitnik Jun 20, 2018
14013d3
add docs for test alignment and test affinity + fix a bug in argument…
adamsitnik Jun 20, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,10 @@


**/.vs/**

# BenchmarkDotNet results
**/BenchmarkDotNet.Artifacts/**

# Resharper files
**/*.DotSettings.user
**/.idea/**
17 changes: 11 additions & 6 deletions src/benchmarks/Benchmarks.csproj
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">

<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFrameworks>net46;netcoreapp2.0;netcoreapp2.1</TargetFrameworks>
Expand All @@ -8,26 +7,32 @@
<DebugSymbols>true</DebugSymbols>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>

<ItemGroup>
<Compile Remove="img\**" />
<EmbeddedResource Remove="img\**" />
<None Remove="img\**" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.10.14.516" />
<PackageReference Include="BenchmarkDotNet" Version="0.10.14.560" />
<PackageReference Include="CommandLineParser" Version="2.2.1" />
<PackageReference Include="Jil" Version="2.15.4" />
<PackageReference Include="MessagePack" Version="1.7.3.4" />
<PackageReference Include="MessagePackAnalyzer" Version="1.6.0" />
<PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
<PackageReference Include="protobuf-net" Version="2.3.7" />
<PackageReference Include="System.Memory" Version="4.5.0-rc1" />
<PackageReference Include="System.Numerics.Vectors" Version="4.5.0-rc1" />
<PackageReference Include="System.Runtime.CompilerServices.Unsafe" Version="4.5.0-rc1" />
<PackageReference Include="System.Runtime.Serialization.Formatters" Version="4.3.0" />
<PackageReference Include="System.Runtime.Serialization.Json" Version="4.3.0" />
<PackageReference Include="System.Xml.XmlSerializer" Version="4.3.0" />
<PackageReference Include="Utf8Json" Version="1.3.7" />
<PackageReference Include="ZeroFormatter" Version="1.6.4" />
<PackageReference Include="ZeroFormatter.Analyzer" Version="1.1.1" />
</ItemGroup>
</Project>
<ItemGroup>
<None Update="BenchmarksGame\Inputs\*.txt">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>
25 changes: 25 additions & 0 deletions src/benchmarks/BenchmarksGame/Inputs/InputFileHelper.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
using System;
using System.IO;

namespace BenchmarksGame
{
internal static class InputFileHelper
{
internal static string FindInputFile(string inputFile)
{
if (string.IsNullOrEmpty(inputFile))
throw new ArgumentNullException(nameof(inputFile));

var fullPath = GetFullPath(inputFile);
if (!File.Exists(fullPath))
throw new FileNotFoundException("Unable to find input file.", inputFile);

return fullPath;
}

internal static int GetFileLength(string filePath) => (int) new FileInfo(filePath).Length;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(int) [](start = 62, length = 5)

Does it need to be int? Can we keep long instead? The API user can cast the result when needed. Thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure thing, I added the cast here because it was hardcoded to an int previously


private static string GetFullPath(string fileName)
=> Path.Combine(Path.GetDirectoryName(typeof(InputFileHelper).Assembly.Location), "BenchmarksGame", "Inputs", fileName);
}
}
43 changes: 43 additions & 0 deletions src/benchmarks/BenchmarksGame/Inputs/NucleotideHelpers.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

// Helper functionality to locate inputs and find outputs for
// k-nucleotide benchmark in CoreCLR test harness

using System;
using System.IO;
using System.Text;

namespace BenchmarksGame
{
class NucleotideHelpers
{
public string InputFile;
public int[] expectedCountLetter;
public int[] expectedCountPairs;
public int[] expectedCountFragments;
public int[][] expectedFrequencies;

public NucleotideHelpers(bool bigInput)
{
if (bigInput)
{
InputFile = InputFileHelper.FindInputFile("knucleotide-input-big.txt");
expectedCountLetter = new int[] { 302923, 301375, 198136, 197566 };
expectedCountPairs = new int[] { 91779, 91253, 91225, 90837, 60096, 60030, 59889, 59795, 59756, 59713, 59572, 59557, 39203, 39190, 39081, 39023 };
expectedCountFragments = new int[] { 11765, 3572, 380, 7, 7 };
}
else
{
InputFile = InputFileHelper.FindInputFile("knucleotide-input.txt");
expectedCountLetter = new int[] { 1576, 1480, 974, 970 };
expectedCountPairs = new int[] { 496, 480, 470, 420, 316, 315, 310, 302, 298, 292, 273, 272, 202, 201, 185, 167 };
expectedCountFragments = new int[] { 54, 24, 4, 0, 0 };
}
expectedFrequencies = new int[][] { expectedCountLetter, expectedCountPairs };
}


}
}
33 changes: 33 additions & 0 deletions src/benchmarks/BenchmarksGame/Inputs/RegexReduxHelpers.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

// Helper functionality to locate inputs and find outputs for
// regex-redux benchmark in CoreCLR test harness

using System;
using System.IO;
using System.Text;

namespace BenchmarksGame
{
class RegexReduxHelpers
{
public string InputFile;
public int ExpectedLength;

public RegexReduxHelpers(bool bigInput)
{
if (bigInput)
{
InputFile = InputFileHelper.FindInputFile("regexdna-input25000.txt");
ExpectedLength = 136381;
}
else
{
InputFile = InputFileHelper.FindInputFile("regexdna-input25.txt");
ExpectedLength = 152;
}
}
}
}
36 changes: 36 additions & 0 deletions src/benchmarks/BenchmarksGame/Inputs/ReverseComplementHelpers.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

// Helper functionality to locate inputs and find outputs for
// reverse-complement benchmark in CoreCLR test harness

using System;
using System.IO;
using System.Text;

namespace BenchmarksGame
{
class ReverseComplementHelpers
{
public string InputFile;
public int FileLength;
public string CheckSum;

public ReverseComplementHelpers(bool bigInput)
{
if (bigInput)
{
InputFile = InputFileHelper.FindInputFile("revcomp-input25000.txt");
CheckSum = "61-A4-CC-6D-15-8D-26-77-88-93-4F-E2-29-A2-8D-FB";
}
else
{
InputFile = InputFileHelper.FindInputFile("revcomp-input25.txt");
CheckSum = "62-45-8E-09-2E-89-A0-69-8C-17-F5-D8-C7-63-5B-50";
}

FileLength = InputFileHelper.GetFileLength(InputFile);
}
}
}
Loading