Port CoreClr benchmarks to BenchmarkDotNet (#36)

* copy some coreclr benchmarks

* port some some benchmarks

* copy some coreclr benchmarks 2

* port some some benchmarks 2

* rename the folder with CoreClr benchmarks to BenchmarksGame

* fix SpectralNorm_1 benchmark

* copy knucleotide benchmarks

* get knucleotide working

* copy RegexRedux

* make RegexRedux work

* copy reverse-complement

* make reverse-complement work

* copy all .txt files to bin

* update to the latest version

* read the file length, dont hardcode it (it was buggy)

* use Description to give benchmarks some nice display name, but preserve the Id which is exported for BenchView purpose

* remove the Rider files from repo, ignore them

* dont run more than 20 iterations

* add possibility to run benchmarks for Legacy Jits

* copy all benchstones

* port Benchstones to BenchmarkDotNet

* change the default iteration time from 0,5s to 0,25s to run benchmarks faster

* update to the version which supports jagged arrays

* copy Burgers

* port Burgers

* copy DefaultEqualityComparerPerf

* port DefaultEqualityComparerPerf

* copy FractalPerf

* port FractalPerf

* copy Inlining benchmarks

* port Inlining benchmarks

* rename Jit folder to Devirtualization to match the coreclr folder structure

* copy SearchLoops

* port SearchLoops

* copy Linq

* port Linq

* copy perflab

* port BlockCopyPerf

* port CastingPerf

* port CastingPerf2

* port DelegatePerf

* remove Program.cs

* port EnumPerf

* fix the benchmarks

* port StackWalk

* port ThreadingPerf

* port LowLevelPerf

* port ReflectionPerf

* copy V8.Crypto and V8.Richards

* port V8.Crypto

* port V8.Richards

* copy SpanBench

* port Span/Indexer

* port SpanBench

* update dependencies (which contain required fixes)

* copy SIMD benchmarks

* bump .NET to 4.6.1 so we can use Vector.LessThanOrEqualAny

* port ConsoleMandel

* port RayTracerBench

* port SeekUnroll

* move all internal types of RayTracer to a separate namespace to avoid conflicts with other existign types (like Vector)

* move all the CoreCLR benchmarks to coreclr subfolder

* copy CscBench

* port CscBench

* copy SciMark

* port SciMark

* copy ByteMark

* port ByteMark

* copy Math

* port Math

* dont try to compile Single type benchmarks which use MathF which does not support .NET Framework (only Core)

* fix line endings ;)

* group the benchmarks into categories, allow filtering and joining into single summary

* export to json (for BenchView intergration purpose), allow users to specify "--baseJob Dry" to run every benchmark just once (good for testing)

* always show Min and Max column to mimic xunit-performance behaviour

* assign the right type to a category (the one which actually contains benchmarks, not helpers)

* the benchmarks should not be static

* read the array from field to preserve old benchmark id

* don't use argument, keep old benchmark full name

* get correct allocated bytes after a bug fix in BenchmarkDotNet

* don't use argument, keep old benchmark full name + the value was 16, not 10

* missing * 2!!

* update to BDN which uses full type name for exported results (we have two classes called "Support")

* move the arguments to fields to match existing benchmark id

* don't use argument, keep old benchmark full name

* expectedSum argument needs to remain to keep old benchmark id in BenchView, do NOT remove it

* don't use argument, keep old benchmark full name

* move the arguments to fields to match existing benchmark id

* add missing category

* use Params from BDN to express things that were arguments used to setup benchmark, it allows to preserve ID

* should be a part of previous commit

* don't make PerfLabTests nano-benchmarks, it would blow up the scaling in BenchView

* make sure ALL ids are the same

* ResultsValidator - small helper utility to print a diff, I will remove it after we port all the benchmarks

* remove the workaround for handling whitespaced in benchmarks Ids, proper fix has been applied to BDN

* allow the users to specify outlier removal mode from console args

* add median to the default columns

* allow the users to test COMPlus_JitAlignLoop 0 vs 1 and affinity set vs no affinity and the permutation of it

* make sure JIT does not optimize the benchmarks to empty loops!

* update to latest BDN which gives us some nice attributes

* apply some non-default settings to make the results from multimodal benchmarks more stable

* add comments to benchmarks whic hare very dependent on loop alignment

* fix ObjectGetTypeNoBoxing, disable ObjectGetType which is getting optimized to an empty loop

* change the order of exported columns in CSV to make it more easy to copy paste it to an excel with some formulas

* rename console attributes, don't introduce new standards ;) + don't write about default value, command line parser does it out of the box

* add docs for test alignment and test affinity + fix a bug in arguments handling

.gitattributes

@@ -7,3 +7,10 @@ src/coreclr/BenchmarksGame/regex-redux/regexdna-input25.txt text eol=lf
 src/coreclr/BenchmarksGame/regex-redux/regexdna-input25000.txt text eol=lf
 src/coreclr/BenchmarksGame/reverse-complement/revcomp-input25.txt text eol=lf
 src/coreclr/BenchmarksGame/reverse-complement/revcomp-input25000.txt text eol=lf
+
+src/benchmarks/coreclr/BenchmarksGame/Inputs/knucleotide-input-big.txt text eol=lf
+src/benchmarks/coreclr/BenchmarksGame/Inputs/knucleotide-input.txt text eol=lf
+src/benchmarks/coreclr/BenchmarksGame/Inputs/regexdna-input25.txt text eol=lf
+src/benchmarks/coreclr/BenchmarksGame/Inputs/regexdna-input25000.txt text eol=lf
+src/benchmarks/coreclr/BenchmarksGame/Inputs/revcomp-input25.txt text eol=lf
+src/benchmarks/coreclr/BenchmarksGame/Inputs/revcomp-input25000.txt text eol=lf

.gitignore

@@ -14,4 +14,10 @@
 
 
 **/.vs/**
+
+# BenchmarkDotNet results
 **/BenchmarkDotNet.Artifacts/**
+
+# Resharper files
+**/*.DotSettings.user
+**/.idea/**

src/benchmarks/Benchmarks.csproj

@@ -1,30 +1,42 @@
 <Project Sdk="Microsoft.NET.Sdk">
   <PropertyGroup>
     <OutputType>Exe</OutputType>
-    <TargetFrameworks>net46;netcoreapp2.0;netcoreapp2.1</TargetFrameworks>
+    <TargetFrameworks>net461;netcoreapp2.0;netcoreapp2.1</TargetFrameworks>
     <PlatformTarget>AnyCPU</PlatformTarget>
     <DebugType>pdbonly</DebugType>
     <DebugSymbols>true</DebugSymbols>
     <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
+    <LangVersion>7.3</LangVersion>
   </PropertyGroup>
   <ItemGroup>
     <Compile Remove="img\**" />
     <EmbeddedResource Remove="img\**" />
     <None Remove="img\**" />
   </ItemGroup>
   <ItemGroup>
-    <PackageReference Include="BenchmarkDotNet" Version="0.10.14.589" />
+    <PackageReference Include="BenchmarkDotNet" Version="0.10.14.657" />
     <PackageReference Include="CommandLineParser" Version="2.2.1" />
     <PackageReference Include="Jil" Version="2.15.4" />
     <PackageReference Include="MessagePack" Version="1.7.3.4" />
     <PackageReference Include="MessagePackAnalyzer" Version="1.6.0" />
     <PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
     <PackageReference Include="protobuf-net" Version="2.3.7" />
+    <PackageReference Include="System.Memory" Version="4.5.0" />
+    <PackageReference Include="System.Numerics.Vectors" Version="4.5.0" />
+    <PackageReference Include="System.Runtime.CompilerServices.Unsafe" Version="4.5.0" />
     <PackageReference Include="System.Runtime.Serialization.Formatters" Version="4.3.0" />
     <PackageReference Include="System.Runtime.Serialization.Json" Version="4.3.0" />
     <PackageReference Include="System.Xml.XmlSerializer" Version="4.3.0" />
     <PackageReference Include="Utf8Json" Version="1.3.7" />
     <PackageReference Include="ZeroFormatter" Version="1.6.4" />
     <PackageReference Include="ZeroFormatter.Analyzer" Version="1.1.1" />
   </ItemGroup>
+  <ItemGroup>
+    <None Update="coreclr\BenchmarksGame\Inputs\*.txt">
+      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
+    </None>
+  </ItemGroup>
+  <ItemGroup Condition=" '$(TargetFrameworkIdentifier)' == '.NETFramework' ">
+    <Compile Remove="coreclr\Math\Functions\Single\**" />
+  </ItemGroup>
 </Project>

src/benchmarks/Categories.cs

@@ -0,0 +1,16 @@
+namespace Benchmarks
+{
+    public static class Categories
+    {
+        public const string CoreCLR = "CoreCLR";
+            public const string BenchmarksGame = "BenchmarksGame";
+            public const string Benchstones = "Benchstones";
+                public const string BenchF = "BenchF";
+                public const string BenchI = "BenchI";
+            public const string Inlining = "Inlining";
+            public const string SIMD = "SIMD";
+            public const string Span = "Span";
+            public const string V8 = "V8";
+            public const string Perflab = "Perflab";
+    }
+}

src/benchmarks/Program.cs

@@ -1,4 +1,5 @@
-using System.Collections.Generic;
+using System;
+using System.Collections.Generic;
 using System.IO;
 using System.Linq;
 using BenchmarkDotNet.Columns;
@@ -7,8 +8,11 @@
 using BenchmarkDotNet.Environments;
 using BenchmarkDotNet.Exporters;
 using BenchmarkDotNet.Exporters.Csv;
+using BenchmarkDotNet.Exporters.Json;
+using BenchmarkDotNet.Filters;
 using BenchmarkDotNet.Horology;
 using BenchmarkDotNet.Jobs;
+using BenchmarkDotNet.Mathematics;
 using BenchmarkDotNet.Running;
 using BenchmarkDotNet.Toolchains.CoreRt;
 using BenchmarkDotNet.Toolchains.CsProj;
@@ -31,16 +35,20 @@ static void Main(string[] args)
         private static void RunBenchmarks(Options options)
             => BenchmarkSwitcher
                 .FromAssemblyAndTypes(typeof(Program).Assembly, SerializerBenchmarks.GetTypes())
-                .Run(config: GetConfig(options));
+                .Run(GetArgs(options), GetConfig(options));
+
+        private static string[] GetArgs(Options options)
+            => options.Join ? new[] {"--join"} : Array.Empty<string>(); 
 
         private static IConfig GetConfig(Options options)
         {
-            var baseJob = Job.ShortRun; // let's use the Short Run for better first user experience ;)
-            var jobs = GetJobs(options, baseJob).ToArray();
+            var baseJob = GetBaseJob(options);
 
-            var config = DefaultConfig.Instance
-                .With(jobs.Any() ? jobs : new[] { baseJob });
+            var baseJobPermutations = GetBaseJobPermutations(baseJob, options).ToArray(); 
+            var jobs = baseJobPermutations.SelectMany(job => GetJobs(options, job)).ToArray();
 
+            var config = DefaultConfig.Instance.With(jobs.Any() ? jobs : baseJobPermutations);
+
             if (options.UseMemoryDiagnoser)
                 config = config.With(MemoryDiagnoser.Default);
             if (options.UseDisassemblyDiagnoser)
@@ -49,16 +57,93 @@ private static IConfig GetConfig(Options options)
             if (options.DisplayAllStatistics)
                 config = config.With(StatisticColumn.AllStatistics);
 
+            if (options.AllCategories.Any())
+                config = config.With(new AllCategoriesFilter(options.AllCategories.ToArray()));
+            if (options.AnyCategories.Any())
+                config = config.With(new AnyCategoriesFilter(options.AnyCategories.ToArray()));
+            if (options.Namespaces.Any())
+                config = config.With(new NamespacesFilter(options.Namespaces.ToArray()));
+            if (options.MethodNames.Any())
+                config = config.With(new MethodNamesFilter(options.MethodNames.ToArray()));
+            if (options.TypeNames.Any())
+                config = config.With(new TypeNamesFilter(options.TypeNames.ToArray()));
+
+            config = config.With(JsonExporter.Full); // make sure we export to Json (for BenchView integration purpose)
+
+            config = config.With(StatisticColumn.Median, StatisticColumn.Min, StatisticColumn.Max);
+
             return config;
         }
 
+        private static Job GetBaseJob(Options options)
+        {
+            Job baseJob = null;
+
+            switch (options.BaseJob.ToLowerInvariant())
+            {
+                case "dry":
+                    baseJob = Job.Dry;
+                    break;
+                case "short":
+                    baseJob = Job.ShortRun;
+                    break;
+                case "medium":
+                    baseJob = Job.MediumRun.WithOutlierMode(options.Outliers);
+                    break;
+                case "long":
+                    baseJob = Job.LongRun;
+                    break;
+                default: // the recommended settings
+                    baseJob = Job.Default
+                        .WithIterationTime(TimeInterval.FromSeconds(0.25)) // the default is 0.5s per iteration, which is slighlty too much for us
+                        .WithWarmupCount(1) // 1 warmup is enough for our purpose
+                        .WithOutlierMode(options.Outliers)
+                        .WithMaxIterationCount(20);  // we don't want to run more that 20 iterations
+                    break;
+            }
+
+            baseJob = baseJob.WithOutlierMode(options.Outliers);
+
+            if (options.Affinity.HasValue && !options.TestAffinity)
+                baseJob = baseJob.WithAffinity((IntPtr) options.Affinity.Value);
+
+            return baseJob;
+        }
+
+        private static IEnumerable<Job> GetBaseJobPermutations(Job baseJob, Options options)
+        {
+            IEnumerable<Job> CreateLoopAligmentPermutations(Job job)
+            {
+                yield return job.With(new[] {new EnvironmentVariable("COMPlus_JitAlignLoops", "1")});
+                yield return job.With(new[] {new EnvironmentVariable("COMPlus_JitAlignLoops", "0")});
+            }
+
+            IEnumerable<Job> CreateAffinityPermutations(Job job)
+            {
+                yield return job;
+                yield return job.WithAffinity((IntPtr)options.Affinity.Value);
+            }
+
+            Job[] jobs = { baseJob };
+            if (options.TestAlignLoops)
+                jobs = jobs.SelectMany(CreateLoopAligmentPermutations).ToArray();
+            if (options.TestAffinity && options.Affinity.HasValue)
+                jobs = jobs.SelectMany(CreateAffinityPermutations).ToArray();
+
+            return jobs;
+        }
+
         private static IEnumerable<Job> GetJobs(Options options, Job baseJob)
         {
             if (options.RunInProcess)
                 yield return baseJob.With(InProcessToolchain.Instance);
 
             if (options.RunClr)
                 yield return baseJob.With(Runtime.Clr);
+            if (options.RunLegacyJitX64)
+                yield return baseJob.With(Runtime.Clr).With(Jit.LegacyJit).With(Platform.X64);
+            if (options.RunLegacyJitX86)
+                yield return baseJob.With(Runtime.Clr).With(Jit.LegacyJit).With(Platform.X86);
             if (!string.IsNullOrEmpty(options.ClrVersion))
                 yield return baseJob.With(new ClrRuntime(options.ClrVersion));
 
@@ -136,6 +221,12 @@ public class Options
 
         [Option("clr", Required = false, Default = false, HelpText = "Run benchmarks for Clr")]
         public bool RunClr { get; set; }
+
+        [Option("legacyJitx64", Required = false, Default = false, HelpText = "Run benchmarks for Legacy JIT x64")]
+        public bool RunLegacyJitX64 { get; set; }
+
+        [Option("legacyJitx86", Required = false, Default = false, HelpText = "Run benchmarks for Legacy JIT x86")]
+        public bool RunLegacyJitX86 { get; set; }
 
         [Option("clrVersion", Required = false, HelpText = "Optional version of private CLR build used as the value of COMPLUS_Version env var.")]
         public string ClrVersion { get; set; }
@@ -184,6 +275,39 @@ public class Options
 
         [Option("coreFxBin", Required = false, HelpText = @"Optional path to folder with CoreFX NuGet packages, Example: ""C:\Projects\forks\corefx\bin\packages\Release""")]
         public string CoreFxBinPackagesPath { get; set; }
+
+        [Option("categories", Required = false, HelpText = "Categories to run. If few are provided, only the benchmarks which belong to all of them are going to be executed")]
+        public IEnumerable<string> AllCategories { get; set; }
+
+        [Option("anyCategories", Required = false, HelpText = "Any Categories to run")]
+        public IEnumerable<string> AnyCategories { get; set; }
+
+        [Option("namespace", Required = false, HelpText = "Namespace(s) to run")]
+        public IEnumerable<string> Namespaces { get; set; }
+
+        [Option("method", Required = false, HelpText = "Method(s) to run")]
+        public IEnumerable<string> MethodNames { get; set; }
+
+        [Option("class", Required = false, HelpText = "Class(es) with benchmarks to run")]
+        public IEnumerable<string> TypeNames { get; set; }
+
+        [Option("join", Required = false, Default = false, HelpText = "Prints single table with results for all benchmarks")]
+        public bool Join { get; set; }
+
+        [Option("baseJob", Required = false, Default = "Default", HelpText = "Dry/Short/Medium/Long or Default")]
+        public string BaseJob { get; set; }
+
+        [Option("outliers", Required = false, Default = OutlierMode.OnlyUpper, HelpText = "None/OnlyUpper/OnlyLower/All")]
+        public OutlierMode Outliers { get; set; }
+
+        [Option("testAlignment", Required = false, Default = false, HelpText = "Test COMPlus_JitAlignLoop 0 vs 1")]
+        public bool TestAlignLoops { get; set; }
+
+        [Option("testAffinity", Required = false, Default = false, HelpText = "Test affinity set vs no affinity")]
+        public bool TestAffinity { get; set; }
+
+        [Option("affinity", Required = false, HelpText = "Affinity mask to set for the benchmark process")]
+        public int? Affinity { get; set; }
     }
 
     /// <summary>

src/benchmarks/README.md

@@ -10,10 +10,12 @@ BenchmarkDotNet will protect you from the common pitfalls (even for experienced
 * it generates an isolated project per runtime
 * it builds the project in `Release`
 * it runs every benchmark in a stand-alone process (to achieve process isolation and avoid side effects)
-* it estimates the perfect invocation count per iteration
+* it estimates the perfect invocation count per iteration (based on `IterationTime`)
 * it warms-up the code
 * it evaluates the overhead
-* it runs multiple iterations of the method until the requested level of precision is met.
+* it runs multiple iterations of the method until the requested level of precision is met
+* it consumes the benchmark result to avoid dead code elimination
+* it prevents from inlining of the benchmark by wrapping it with a delegate.
 
 A few useful links for you:
 
@@ -49,6 +51,19 @@ BenchmarkDotNet will build the executables, run the benchmarks, print the result
 
 BenchmarkDotNet by default exports the results to GitHub markdown, so you can just find the right `.md` file in `results` folder and copy-paste the markdown to GitHub.
 
+## Filtering
+
+You can filter the benchmarks by namespace, category, type name and method name. Examples:
+
+* `dotnet run -c Release -f netcoreapp2.1 -- --categories CoreCLR Span` - will run all the benchmarks that belong to CoreCLR **AND** Span category
+* `dotnet run -c Release -f netcoreapp2.1 -- --anyCategories CoreCLR CoreFX` - will run all the benchmarks that belong to CoreCLR **OR** CoreFX category
+* `dotnet run -c Release -f netcoreapp2.1 -- --namespace=BenchmarksGame` - will run all the benchmarks from BenchmarksGame namespace
+* `dotnet run -c Release -f netcoreapp2.1 -- --method=ToStream` - will run all the benchmarks with method name ToStream
+* `dotnet run -c Release -f netcoreapp2.1 -- --class=Richards` - will run all the benchmarks with type name Richards
+
+**Note:** To print a single summary for all of the benchmarks, use `--join`. 
+Example: `dotnet run -c Release -f netcoreapp2.1 -- --join --namespace=BenchmarksGame` - will run all of the benchmarks from BenchmarksGame namespace and print a single summary.
+
 ## All Statistics
 
 By default BenchmarkDotNet displays only `Mean`, `Error` and `StdDev` in the results. If you want to see more statistics, please pass `--allStats` as an extra argument to the app: `dotnet run -c Release -f netcoreapp2.1 -- --allStats`. If you build your own config, please use `config.With(StatisticColumn.AllStatistics)`.
@@ -224,3 +239,35 @@ var config = DefaultConfig.Instance
             .ToToolchain()));
 ```
 
+## Testing how Processor Affinity and Loop Alignment affect results
+
+To run the benchmarks with specific Processor Affinity, you need to provide the processor mask as an argument called `--affinity`.
+
+Example: `dotnet run -c Release -f netcoreapp2.1 -- --affinity=8`
+
+To run same benchmarks with and without specific Processor Affinity you need to use `--testAffinity` and also provide the mask with `--affinity`
+
+Example: `dotnet run -c Release -f netcoreapp2.1 -- --class=BinaryTrees_2 --affinity=8 --testAffinity`
+
+|        Method |     Affinity |
+|-------------- |------------- |
+| BinaryTrees_2 | 000000001000 |
+| BinaryTrees_2 | 111111111111 |
+
+To test how loop alignment affects the results, you can use `--testAlignment` which is going to run the benchmarks with env var `COMPlus_JitAlignLoops` set to `0` and `1`
+
+Example: `dotnet run -c Release -f netcoreapp2.1 -- --class=BinaryTrees_2 --testAlignment`
+
+|        Method |    EnvironmentVariables |
+|-------------- |------------------------ |
+| BinaryTrees_2 | COMPlus_JitAlignLoops=0 |
+| BinaryTrees_2 | COMPlus_JitAlignLoops=1 |
+
+**Note:** You can combine `--testAlignment` with `--testAffinity` which will results in 4 different benchmark runs:
+
+|        Method |     Affinity |    EnvironmentVariables |
+|-------------- |------------- |------------------------ |
+| BinaryTrees_2 | 000000001000 | COMPlus_JitAlignLoops=0 |
+| BinaryTrees_2 | 000000001000 | COMPlus_JitAlignLoops=1 |
+| BinaryTrees_2 | 111111111111 | COMPlus_JitAlignLoops=0 |
+| BinaryTrees_2 | 111111111111 | COMPlus_JitAlignLoops=1 |