Benchmarking is a well known topic of Computer Science, primarily aimed at comparing software and/or hardware systems. With the rise of machine learning and AI, the overall efforts on AI benchmarking in constantly increasing. The SciMLBench is aimed at the AI for Science domain, as opposed to generic AI benchmarking.
With an ever growing number of machine learning models and algorithms, a range of scientific problems, and the proliferation of AI systems, developing cutting-edge ML/AI algorithms requires a detailed understanding of the interactions between these aspects.
The SciMLBench is aimed at providing an answer to this open question. It is an open-source initiative, and covers a range of scientific problems from various domains of science, including material, life, and earth sciences, particle physics and astronomy. The benchmarks are implemented in Python, relying on one or more machine learning frameworks, such as TensorFlow, PyTorch or SciKit-Learn.
The overarching purpose of this initiative is many-fold, including, supporting benchmarking of machine learning models, AI systems, and supporting the AI for Science community for developing better solutions.
The suite has three components, namely,
-
Benchmarks: The benchmarks are machine learning applications performing a specific scientific task, written in Python. These are included as part of this package, and can be found inside the
./sciml_bench/benchmarksdirectory. In the scale of micro-apps, mini-apps, and apps, these are full-fledged applications. -
Datasets: Each benchmark in (1) relies on one or more datasets, for example for training and/or inferencing. These datasets are open, task- or domain-specific, and FAIR compliant. Most of these datasets being large, they are hosted separately, on one of the servers (or mirrors), and are automatically or explicitly downloaded on demand. The framework (see (3)), supports manual downloading of these datasets.
-
Framework: The framework serves two purposes: first, at the user level, it facilitates an easier approach to benchmarking, logging and reporting of the results. Secondly, at the developer level, it provides a coherent API for unifying and simplifying the development of AI benchmarks. This can be found in
./sciml_bench/coredirectory.
The source tree, which captures these aspects, is organised as follows:
├── README.md <This file>
├── config <Container configuration files>
│ ├── Dockerfile
│ ├── ompi4.def
│ └── sciml-bench.def
├── doc <User documentation>
│ ├── benchmarks_datasets.md <List of benchmarks / datasets>
│ ├── contributing.md <How to contribute>
│ ├── credits.md
│ ├── demo_output__MNIST_torch
│ │ └── <Sample benchmark outputs>
│ ├── resources
│ │ └── <Various resources>
│ └── usage.md <Usage documentation>
├── requirements.txt
├── sciml_bench
├── benchmarks <Benchmark sources>
│ ├── registration.yml <Key registration file>
│ └── template
│ └── template.py <Benchmark Template>
├── core <Core scripts>
│ ├── messages <Display messages>
└── sciml_bench_config.yml <Directory configurations>
We have annotated the purpose of each folder/directory witin <>.
A typical user-base for the benchmarking framework may include a number of user communities, such as system manufacturers and integrators (for assessing system performance), scientists (for developing new algorithms), and ML enthusiasts (for understanding the basics of various machine learning models and algorithms). It is a challenging task to design for and cover all these requirements in a single framework. Here, with SciMLBench, we have attempted to cover these requirements through the following set of features:
- Very flexible, customisable and lightweight framework,
- Powerful logging and monitoring capabilities,
- Support for multiple machine learning frameworks (Tensorflow, PyTorch, and SciKit-Learn),
- Simplified application programming interface (API), to support easier development of benchmarks,
- Fully customisable installation,
- Simplified use of framework encouraging a wide range of users, and
- Fully decoupled, on-demand, and user-initiated data downloads.
The number of datasets and benchmarks may vary with every release. Please consult the Benchmarks document for this. A number authors have contributed towards the development of the benchmarks, and these can be see in the Credits. If you are thinking of contributing towards the benchmarks or datasets, please see the Contributing Datasets & Benchmarks.
Please consult the Installation & Usage file, placed inside the doc folder, for getting started.
Cite this benchmark suite as follows:
```
@misc{scimlbench:2021,
title = {SciMLBench: A Benchmarking Suite for AI for Science},
author = {Jeyan Thiyagalingam, Kuangdai Leng, Samuel Jackson, Juri Papay, Mallikarjun Shankar, Geoffrey Fox, Tony Hey},
url = {https://github.com/stfc-sciml/sciml-bench},
year = {2021}
}
```
This benchmarking programme is supported by [1] Wave I of the UKRI Strategic Priorities Fund under the EPSRC grant (EP/T001569/1), particularly the AI for Science theme in that grant and the Alan Turing Institute, and [2] the Benchmarking for AI for Science at Exascale (BASE), EPSRC ExCALIBUR Phase I grant (EP/V001310/1).