MUBench

MUBench (pronounced "Moo Bench") is an automated benchmark for API-misuse detectors, based on the MUBench benchmarking dataset. If you encounter any problems using MUBench, please report them to us. If you have any questions, please contact Sven Amann.

CI Status:

Contributors

The people (in alphabetical order by their last name) who actively contributed to the concept, the dataset, or the codebase of MUBench since the project's inception in October 2015:

• Sven Amann (Project Lead, Since 10/2015)
• Mattis Kämmerer (Developer, 03/2016 - 03/2018)
• Sarah Nadi (Dataset Reviewer, 10/2015 - 05/2016)
• Hoan Anh Nguyen (Dataset Reviewer, 10/2015 - 05/2016)
• Tien N. Nguyen (Dataset Reviewer, 10/2015 - 05/2016)
• Jonas Schlitzer (Developer, 10/2016 - 07/2018)

The MUBench project was developed at the Software Technology Group of Technische Universität Darmstadt from October 2015 to September 2018. From October 2018, it is being maintained by Sven Amann as an independent Open Source project.

Publications

• 'Exposing Library API Misuses via Mutation Analysis' (ICSE '19)
• 'Investigating Next Steps in Static API-Misuse Detection' (MSR '19)
• 'A Systematic Evaluation of Static API-Misuse Detectors' (TSE '18)
• 'MUBench: A Benchmark for API-Misuse Detectors' (MSR '16 Data Showcase)

We provide instructions to reproduce the MUBench experiments presented in the above publications.

Getting Started

With MUBench, you may run different API-misuse detectors in a number of experiments to determine their precision and recall. To this end, MUBench provides a curated dataset of real-world projects and known misuses. In each experiment run, the respective detector emits findings which you need to review manually. To this end, MUBench publishes (a subset of) the findings to a review website. After you completed your reviews, the site automatically computes experiment statistics.

Setup

1. MUBench comes with a Docker image to allow platform-independent execution of experiments. Therefore, you first need to install Docker.

2. Afterwards, you can start the MUBench Interactive Shell using

   $> docker run -it -v mubench-checkouts:/mubench/checkouts -v mubench-findings:/mubench/findings --rm -p 8080:80 svamann/mubench:stable
   

   This docker command starts in interactive shell (-it) based on our Docker image svamann/mubench in the latest stable version. It persists experiment data on Docker volumes named mubench-checkouts and mubench-findings (-v), while disposing of all other state on exit (--rm). And it forwards port 80 from within the shell to port 8080 of your host system, to allow running a standalone review site.

3. (Optional) You may want to create a script with the above command, which allows you to conveniently open a MUBench Interactive Shell by typing ./mubench or running individual commands by typing ./mubench <command>.

4. (Optional) You may want to setup a review site to collaboratively review detector findings and publish your results.

Hint: We recommend to use the latest stable version svamann/mubench:stable of our Docker image. However, you may also chose to use the latest development version svamann/mubench:latest, which is continuously deployed from the master branch of this repository.

Use

For all usage examples in this documentation, we assume that you opened a MUBench Interactive Shell. Alternatively, you may execute individual commands by passing them as arguments to the docker command for running MUBench.

1. Run experiments.
2. Publish misuse metadata to a review site.
3. Publish detector findings to a review site.
4. Debug a detector (runner).

Contribute

We want MUBench to grow, so please be welcome to

• Add your own projects or misuses to the benchmarking dataset.
• Add your own detector to the benchmark.
• Contribute to the benchmarking platform

License

All software provided in this repository is subject to the CRAPL license.

The detectors included in MuBench are subject to the licensing of their respective creators. See the information in the detectors' folders.

The projects referenced in the MuBench dataset are subject to their respective licenses.

The project artwork is subject to the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0).