Simple API for specifying each benchmark workload

[JackKelly]

In today's Benchmarking meeting, we all agreed that it'd be great to get a simple benchmarking solution implemented ASAP, so we can get on with the fun work of trying to speed things up 🙂. To start with, we just need something simple that'll allow us to run benchmark workloads on our machines, figure out how close the current code is to the theoretical IO performance of the hardware, and compare performance between git commits.

Here's an early attempt to specify a very simple framework to allow people to specify workloads. Here's a quick example of what is required to implement a simple dataset and workload.

To implement a new workload, you'd implement a class which inherits from Workload, and you'd just override init_dataset and run. To implement a new dataset, you'd implement a class which inherits from 'Dataset', and override prepare.

Does this look OK? Is there anything missing?

The (very simple) framework would then take care of discovering & running the workloads, whilst also recording relevant metrics to disk as JSON, and printing a summary.

The framework isn't ready yet! But the idea is that the MVP will expose a very simple CLI that allows you to run all benchmarks, or a specific benchmark. It'll automatically record metadata about the system and the git commit. And it'll make it easy to compare performance between different git commits.

The idea is to make it super-easy for anyone to submit a new workload via a PR. And easy for us to share "human-readable" metrics ("My new PR speeds up Zarr-Python on workload x by 10x on my machine! Hurray!") and share machine-readable metrics (JSON).

I'd imagine moving this code to zarr-developers if folks agree that the approach is OK.

(See zarr-developers/zarr-benchmark#1 for a discussion of why I'm currently leaning towards the idea of implementing our own benchmarking tool. But feel free to make the case for using an existing tool! I'm just not sure that any existing tool would allow us to measure IO performance.)

[JackKelly]

Today, I've fleshed out the simple benchmarking tool, and I've updated the comment above because the API has slightly changed. The benchmarking tool can now create an example dataset, and run an example workload. Next week I'll start capturing performance metrics, and implement some zarr-specific datasets & workloads 🙂

[JackKelly]

This is done now. But please shout if you have any comments on the API!