ipflakies

Experimental scripts for running iPFlakies to detect polluters of victims as well as state-setters for brittles in the dataset from Gruber et al..

What's inside

./victims_brittles.csv The dataset provided by Gruber et al, including 4,312 claimed order-dependent flaky tests.

./dataset_amended.csv The original dataset is fixed in column Test_classname for our experimental use.

./ipflakies_result/ The data generated by the experiment on 456 Python Projects.

./summary/ The summary of the experiment. To be soecific, run the command python3 parse_result.py ipflakies_result summary.

How to run

To start up, download and export the docker image: https://zenodo.org/record/6178311/files/ipflakies_docker_image.tar

$ docker import ipflakies_docker_container.tar ipflakies

$ docker run -it -v $(pwd)/docker_output:/root/external --name ipflakies_test ipflakies /bin/bash

Inside the docker container, run the commands

cd /root
grep IOCynergy pytest_suites.csv > test.in
# Select the projects to run in this docker container, you may also want to select tuples in pytest_suites.csv:
# sed -n '436,440p' pytest_suites.csv > test.in
bash exec_ipflakies.sh test.in dataset.csv
exit

Then leave the docker container and get the result:

$ cp -r docker_output/ipflakies_result ./ipflakies_result_test
$ bash unzip_output.sh ipflakies_result_test
$ mkdir summary_test
$ python3 parse_result.py ipflakies_result_test summary_test

The expected summary is saved in directory ./summary_test_expected/.

The expected screen output of this case is saved in file ./example_output.