Use mutation testing

Use mutation testing

Unit testing is great for testing your production code. But have you thought about testing your unit tests? What that means is, how are you sure your tests really check what you meant them to? Fortunately, there is an automated way to do just that, no code from you required, only some build configuration.

Mutation testing is a simple concept: Go "break" some production code, and see if any unit tests fail. Production bytecode is changed during the build— for example, an if (x) is changed to if (!x)—, and the unit tests run. With good code coverage, there should now be a failing unit test.

The best option for Modern Java/JVM mutation testing is PITest. It is under active development, does rather clever things with compiled bytecode, and has Gradle and Maven plugins. The main drawback for your local build is that PITest is noisy, so there might be more build output than you might expect.

Mutation testing is one of the slowest parts of a local build. You might consider moving mutation testing to CI-only to speed up local red-green-refactor cycle (_Red, Green, Refactor!, The Cycles of TDD). Use your judgment on the value of the CI build never or rarely failing (modulo external resources) when local build passes vs the speed of pushing good code.

After running a build using PITest, to see the mutation report (on passed or failed mutation coverage), open:

• For Gradle, open build/reports/pitest/index.html
• For Maven, open target/pit-reports/index.html

This project provides the PIT report as part of Maven's project report.

Tips

• Without further configuration, PITest defaults to mutating classes using your project group as the package base. Example: Set the project group to "demo" for either Gradle or Maven if your classes are underneath the "demo.*" package namespace, otherwise PITest may complain that there are no classes to mutate, or no unit tests to run
• If you need to open modules (eg, --add-opens flags), you need to include these flags in "jvm args" configuration for the plugin
• Read more about Mutation Testing from Google
• To open the report for PITest, build locally and use the <project root>/build/reports/pitest/ (Gradle) or <project root>/target/pit-reports/ (Maven) path. The path shown in a Docker build is relative to the interior of the container
• You can temporarily disable mutation testing via -Dpitest.skip=true for Maven (the Gradle plugin does not support this), for example while debugging your build script.

Going further

TODO: Placeholder section