Bisection

KernelCI periodically monitors a series of kernel trees (mainline, stable, next...), and builds them when it detects some changes in them. This typically happens after a merge. It then runs tests (boot at least) with the resulting kernels on a variety of platforms. When a test fails, it compares the results with previous kernels from that same branch on the same platform. If it was working previously, then KernelCI has detected a new failure and stores it as a regression.

As there may have been a topic branch merge with many commits between the last working revision and the now failing one, a bisection is needed in order to isolate the individual commit that introduced the failure. At least this is the idea, it can get more complicated if several different failures were introduced or if the branch got rebased.

The current status of automated bisection is as follows:

• triggered for each regression found
• only run with plain ramdisk boot tests for now
• only run on mainline, stable and a few maintainer trees
• several checks are in place to avoid false positives:
  • check the initial good and bad revisions coming from the found regression
  • when the bisection finds a commit, check that it does fail 3 times to avoid intermittent board issues
  • revert the found commit in-place and check that it does boot 3 times
• send an email to a fixed list of recipients who will manually report the result if it appears valid

We've been monitoring the results for a couple of months and haven't had any false positives so far. There are a few minor things that can be improved, but we're now considering automatically generating the list of recipients including:

• kernelci-build-reports mailing list
• author, committer, signed-off-by, reviewed-by individuals
• mailing list(s) associated with the tree being bisected
• maintainer and/or mailing list associated with the areas of the code touched by the commit found

Future improvements to the automated bisection are:

• extend to linux-next (which gets rebased on a daily basis and has typically many more bugs)
• extend to stable-rc (with branches that get rebased when pushed to the tree)
• extend to test plans other than boot (with fixed version of the test to run while iterating kernel revisions)
• possibility to manually start semi-automatic bisections for special cases