Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic snap support #1652

Merged
merged 1 commit into from
Mar 4, 2021
Merged

Basic snap support #1652

merged 1 commit into from
Mar 4, 2021

Conversation

adamsitnik
Copy link
Member

Fixes #1570

@AndreyAkinshin this is my last PR for 0.13.0

In the example below I have dotnet50 (snap) but no dotnet (typicall installation)

adam@adsitnik-ubuntu:~/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples$ dotnet --info
bash: /usr/bin/dotnet: No such file or directory
adam@adsitnik-ubuntu:~/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples$ dotnet50 run -c Release -f net5.0 --filter BenchmarkDotNet.Samples.IntroColdStart.Foo
/home/adam/projects/BenchmarkDotNet/src/BenchmarkDotNet/Attributes/Filters/OperatingSystemsArchitectureFilterAttribute.cs(15,96): warning CS1573: Parameter 'architectures' has no matching param tag in the XML comment for 'OperatingSystemsArchitectureFilterAttribute.OperatingSystemsArchitectureFilterAttribute(bool, params Architecture[])' (but other parameters do) [/home/adam/projects/BenchmarkDotNet/src/BenchmarkDotNet/BenchmarkDotNet.csproj]
/home/adam/projects/BenchmarkDotNet/src/BenchmarkDotNet/Attributes/Filters/OperatingSystemsFilterAttribute.cs(29,74): warning CS1573: Parameter 'platforms' has no matching param tag in the XML comment for 'OperatingSystemsFilterAttribute.OperatingSystemsFilterAttribute(bool, params OS[])' (but other parameters do) [/home/adam/projects/BenchmarkDotNet/src/BenchmarkDotNet/BenchmarkDotNet.csproj]
/home/adam/projects/BenchmarkDotNet/src/BenchmarkDotNet/Toolchains/DotNetCli/NetCoreAppSettings.cs(48,20): warning CS1573: Parameter 'customRuntimePack' has no matching param tag in the XML comment for 'NetCoreAppSettings.NetCoreAppSettings(string, string, string, string, string, TimeSpan?, string)' (but other parameters do) [/home/adam/projects/BenchmarkDotNet/src/BenchmarkDotNet/BenchmarkDotNet.csproj]
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 1 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start /snap/dotnet-sdk/112/dotnet restore  /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b
// command took 1.49s and exited with 0
// start /snap/dotnet-sdk/112/dotnet build -c Release  --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b
// command took 2.78s and exited with 0
// ***** Done, took 00:00:04 (4.37 sec)   *****
// Found 1 benchmarks:
//   IntroColdStart.Foo: Job-NECTOD(IterationCount=5, RunStrategy=ColdStart)

// **************************
// Benchmark: IntroColdStart.Foo: Job-NECTOD(IterationCount=5, RunStrategy=ColdStart)
// *** Execute ***
// Launch: 1 / 1
// Execute: /snap/dotnet-sdk/112/dotnet "9a018ee4-0f33-46dd-9093-01d3bf31233b.dll" --benchmarkName "BenchmarkDotNet.Samples.IntroColdStart.Foo" --job "IterationCount=5, RunStrategy=ColdStart" --benchmarkId 0 in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b/bin/Release/net5.0
Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied
// BeforeAnythingElse

// Benchmark Process Environment Information:
// Runtime=.NET 5.0.3 (5.0.321.7212), X64 RyuJIT
// GC=Concurrent Workstation
// Job: Job-VYBEBI(IterationCount=5, RunStrategy=ColdStart)

// BeforeActualRun
// First call
WorkloadActual   1: 1 op, 1000798836.00 ns, 1.0008 s/op
WorkloadActual   2: 1 op, 10096348.00 ns, 10.0963 ms/op
WorkloadActual   3: 1 op, 10115168.00 ns, 10.1152 ms/op
WorkloadActual   4: 1 op, 10110514.00 ns, 10.1105 ms/op
WorkloadActual   5: 1 op, 10081318.00 ns, 10.0813 ms/op

// AfterActualRun
WorkloadResult   1: 1 op, 1000798836.00 ns, 1.0008 s/op
WorkloadResult   2: 1 op, 10096348.00 ns, 10.0963 ms/op
WorkloadResult   3: 1 op, 10115168.00 ns, 10.1152 ms/op
WorkloadResult   4: 1 op, 10110514.00 ns, 10.1105 ms/op
WorkloadResult   5: 1 op, 10081318.00 ns, 10.0813 ms/op

// AfterAll
// Benchmark Process 28019 has exited with code 0

Mean = 208.240 ms, StdErr = 198.140 ms (95.15%), N = 5, StdDev = 443.054 ms
Min = 10.081 ms, Q1 = 10.096 ms, Median = 10.111 ms, Q3 = 10.115 ms, Max = 1,000.799 ms
IQR = 0.019 ms, LowerFence = 10.068 ms, UpperFence = 10.143 ms
ConfidenceInterval = [-1,497.801 ms; 1,914.282 ms] (CI 99.9%), Margin = 1,706.042 ms (819.27% of Mean)
Skewness = 1.07, Kurtosis = 2.08, MValue = 2

// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts/results/BenchmarkDotNet.Samples.IntroColdStart-report.csv
  BenchmarkDotNet.Artifacts/results/BenchmarkDotNet.Samples.IntroColdStart-report-github.md
  BenchmarkDotNet.Artifacts/results/BenchmarkDotNet.Samples.IntroColdStart-report.html

// * Detailed results *
IntroColdStart.Foo: Job-NECTOD(IterationCount=5, RunStrategy=ColdStart)
Runtime = .NET 5.0.3 (5.0.321.7212), X64 RyuJIT; GC = Concurrent Workstation
Mean = 208.240 ms, StdErr = 198.140 ms (95.15%), N = 5, StdDev = 443.054 ms
Min = 10.081 ms, Q1 = 10.096 ms, Median = 10.111 ms, Q3 = 10.115 ms, Max = 1,000.799 ms
IQR = 0.019 ms, LowerFence = 10.068 ms, UpperFence = 10.143 ms
ConfidenceInterval = [-1,497.801 ms; 1,914.282 ms] (CI 99.9%), Margin = 1,706.042 ms (819.27% of Mean)
Skewness = 1.07, Kurtosis = 2.08, MValue = 2
-------------------- Histogram --------------------
[-329.987 ms ;   350.166 ms) | @@@@
[ 350.166 ms ;   660.731 ms) | 
[ 660.731 ms ; 1,340.867 ms) | @
---------------------------------------------------

// * Summary *

BenchmarkDotNet=v0.12.1.20210220-develop, OS=ubuntu 18.04
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=5.0.103
  [Host]     : .NET 5.0.3 (5.0.321.7212), X64 RyuJIT
  Job-NECTOD : .NET 5.0.3 (5.0.321.7212), X64 RyuJIT

IterationCount=5  RunStrategy=ColdStart  

| Method |     Mean |      Error |   StdDev |   Median |      Min |        Max |
|------- |---------:|-----------:|---------:|---------:|---------:|-----------:|
|    Foo | 208.2 ms | 1,706.0 ms | 443.1 ms | 10.11 ms | 10.08 ms | 1,000.8 ms |

// * Warnings *
MinIterationTime
  IntroColdStart.Foo: IterationCount=5, RunStrategy=ColdStart -> The minimum observed iteration time is 10.0813 ms which is very small. It's recommended to increase it to at least 100.0000 ms using more operations.

// * Hints *
Outliers
  IntroColdStart.Foo: IterationCount=5, RunStrategy=ColdStart -> 1 outlier  was  detected (1.00 s)

// * Legends *
  Mean   : Arithmetic mean of all measurements
  Error  : Half of 99.9% confidence interval
  StdDev : Standard deviation of all measurements
  Median : Value separating the higher half of all measurements (50th percentile)
  Min    : Minimum
  Max    : Maximum
  1 ms   : 1 Millisecond (0.001 sec)

// ***** BenchmarkRunner: End *****
// ** Remained 0 benchmark(s) to run **
Run time: 00:00:01 (1.45 sec), executed benchmarks: 1

Global total time: 00:00:05 (5.83 sec), executed benchmarks: 1
// * Artifacts cleanup *

@adamsitnik adamsitnik merged commit 8c28c87 into master Mar 4, 2021
@adamsitnik adamsitnik deleted the snap branch March 4, 2021 16:02
naricc pushed a commit to naricc/BenchmarkDotNet that referenced this pull request Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Benchmark runs failing when using .NET 5 RC2 SDK installed via snap
2 participants