Benchmark groups

[jO-Osko]

For our OOPSLA submission, we used a lot of similar benchmarks: 5 different loop varieties, multiple tree explorations... to see how different micro-changes impact compilation and final performance.

For example:

let rec loop_incr n =
    if n = 0 then
        ()
    else
        (perform (Incr ()); loop_incr (n - 1))

and similar that is not tail-recursive

let rec loop_incr' n =
    if n = 0 then
        ()
    else
        (loop_incr' (n - 1); perform (Incr ()))

Would it make sense to group them in some way or maybe add some sort of labels? It seems strange to have 7 benchmarks with 5 different loop versions.

[slindley]

Yeah. We had something similar with Handlers in Action: multiple implementations of handlers, multiple control implementations not using handlers at all, and a range of different parameters for each benchmark.

What we've done so far with that is to add the HIA code as is in the artifacts directory. We haven't yet documented the artifacts directory, but that's where you should put your original code.

The benchmarks directory is intended for smaller more uniform comparisons between systems that may not exactly match the original experiments. Having said that, you can put multiple files in a single benchmark directory. I don't think we've worked out how best to do multiple runs of the same benchmark there yet.

In general, I suspect we're going to want much finer grained control over which benchmarks are run. Running everything is likely to quickly become impractical for the CI, and on local machines we will often just want to run one particular benchmark.

For CI I suspect we really only want to check that the benchmarks compile and run (probably with some small parameter), rather than actually trying to do any actual performance measurements. We may also want to check that they give the correct results. For running locally we'll want a lot more control.