test-commit-range -- make sure you do not break git bisect

The advent of Git also brought tighter rules regarding the quality of commits.

One rule is that - upon hitting save on the commit message - you must be confident that you can relate everyone who is asking questions (why?, not what? because that's in the diff) to the commit message (RTFCM). Chances are high that the one having questions is your future self. There is no way to automatically verify this rule but questions from your reviewers tend to be a very good indicator for the quality of commit messages.

A rule that can be be verified automatically is "no commit breaks git bisect?". test-commit-range.sh is a shell script that provides nothing more but a simple structure to define your quality criteria and run it over a range of commits prior to publishing. The ability to automatically ensure commits don't break git bisect is especially important if you like to make small commits that you later reorder and combine using git rebase --interactive --autosquash.

Example

You could define the following stages. A commit is fine if all stages are passed. Bear in mind that execution time easily accumulates.

• does it configure? (e.g. ./configure generated by autotools, CMake, ...)
• does it build?
• does it pass unit tests?
• does it pass integration tests?

The script generates YAML output indicating for each commit if the stages have been passed:

- 23ef98:
    * cmake: true
    * make: true
    * test: true
    * at: true
- 458de99:
    * cmake: true
    * make: true
    * test: false
    * at: true

Alternatives

Depending on your particular workflow you may want a solution based on Git hooks so that verification is triggered after each new commit.

TODO

• since stages generally depend on each other, script execution should advance to the next commit as soon as one stage fails
• report overall success/failure via exit code