From 1a3e37a324070a989559a38a03a384cd46ee69fb Mon Sep 17 00:00:00 2001 From: Tobias Pfeiffer Date: Sun, 10 Dec 2023 14:40:17 +0100 Subject: [PATCH] Docs for Benchee.report/1 --- README.md | 13 ++++---- samples/save_and_report.exs | 59 +++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 7 deletions(-) create mode 100644 samples/save_and_report.exs diff --git a/README.md b/README.md index cf62679f..27198798 100644 --- a/README.md +++ b/README.md @@ -530,12 +530,13 @@ Enum."-map/2-lists^map/1-0-"/2 10001 26.38 2282 0.23 ### Saving, loading and comparing previous runs -Benchee can store the results of previous runs in a file and then load them again to compare them. For example this is useful to compare what was recorded on the main branch against a branch with performance improvements. +Benchee can store the results of previous runs in a file and then load them again to compare them. For example this is useful to compare what was recorded on the main branch against a branch with performance improvements. You may also use this to benchmark across different exlixir/erlang versions. -**Saving** is done through the `save` configuration option. You can specify a `path` where results are saved, or you can use the default option of`"benchmark.benchee"` if you don't pass a `path`. You can also pass a `tag` option which annotates these results (for instance with a branch name). The default option for the `tag` is a timestamp of when the benchmark was run. +**Saving** is done through the `:save` configuration option. You can specify a `:path` where results are saved, or you can use the default option of`"benchmark.benchee"` if you don't pass a `:path`. You can also pass a `:tag` option which annotates these results (for instance with a branch name or elixir version number). The default option for the `:tag` is a timestamp of when the benchmark was run. -**Loading** is done through the `load` option specifying a path to the file to -load (for instance `"benchmark.benchee"`). You can also specify multiple files to load through a list of paths (`["my.benchee", "main_save.benchee"]`) - each one of those can also be a glob expression to match even more files glob (`"save_number*.benchee"`). +**Loading** is done through the `:load` option specifying a path to the file to load (for instance `"benchmark.benchee"`). You can also specify multiple files to load through a list of paths (`["my.benchee", "main_save.benchee"]`) - each one of those can also be a glob expression to match even more files glob (`"save_number*.benchee"`). +If all you want to do is to use `:load` without running any benchmarks then you can use `Benchee.report/1` which will take a normal configuration with a `:load` key and just format the loaded saved results with the given formatters. +In the more verbose API loading is triggered via `Benchee.load/1`. ```elixir Benchee.run( @@ -545,11 +546,9 @@ Benchee.run( save: [path: "save.benchee", tag: "first-try"] ) -Benchee.run(%{}, load: "save.benchee") +Benchee.report(load: "save.benchee") ``` -In the more verbose API this is triggered via `Benchee.load/1`. - ### Hooks (Setup, Teardown etc.) Most of the time, it's best to keep your benchmarks as simple as possible: plain old immutable functions work best. But sometimes you need other things to happen. When you want to add before or after hooks to your benchmarks, we've got you covered! Before you dig into this section though remember one thing: **you usually don't need hooks!** diff --git a/samples/save_and_report.exs b/samples/save_and_report.exs new file mode 100644 index 00000000..38e4b4ef --- /dev/null +++ b/samples/save_and_report.exs @@ -0,0 +1,59 @@ +list = Enum.to_list(1..10_000) +map_fun = fn i -> [i, i * i] end + +Benchee.run( + %{ + "flat_map" => fn -> Enum.flat_map(list, map_fun) end, + "map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten() end + }, + # report will give us the output + formatters: [], + time: 5, + save: [path: "save.benchee", tag: "save-me"] +) + +IO.puts("----------------------------------------------") + +Benchee.report( + load: "save.benchee", + formatters: [{Benchee.Formatters.Console, extended_statistics: true}] +) + +# Operating System: Linux +# CPU Information: AMD Ryzen 9 5900X 12-Core Processor +# Number of Available Cores: 24 +# Available memory: 31.25 GB +# Elixir 1.16.0-rc.0 +# Erlang 26.1.2 +# JIT enabled: true + +# Benchmark suite executing with the following configuration: +# warmup: 2 s +# time: 5 s +# memory time: 0 ns +# reduction time: 0 ns +# parallel: 1 +# inputs: none specified +# Estimated total run time: 14 s + +# Benchmarking flat_map ... +# Benchmarking map.flatten ... +# Calculating statistics... +# Formatting results... +# Suite saved in external term format at save.benchee +# ---------------------------------------------- +# Formatting results... + +# Name ips average deviation median 99th % +# flat_map (save-me) 3.69 K 270.69 μs ±21.13% 259.71 μs 703.98 μs +# map.flatten (save-me) 1.88 K 530.50 μs ±45.74% 410.19 μs 1227.46 μs + +# Comparison: +# flat_map (save-me) 3.69 K +# map.flatten (save-me) 1.88 K - 1.96x slower +259.82 μs + +# Extended statistics: + +# Name minimum maximum sample size mode +# flat_map (save-me) 215.30 μs 1066.77 μs 18.44 K 257.25 μs +# map.flatten (save-me) 203.19 μs 1522.54 μs 9.41 K406.65 μs, 406.35 μs, 377