Try out mutation testing

[mstoykov]

The idea behind mutation testing (as I understand it) is that in a lot of cases just because you have 100% coverage and tests for everything you have imagined, doesn't mean that part of your code are actually tested. Mutation testing runs the tests without any changes and then starts introducing random changes to the codebase and rerunning the tests. It is expected that changing the code will change the output of the test cases. If it doesn't then the made mutation is considered to be undetected bug.

The unfortunate reality is that there are a LOT of false positives especially if you do any kind of optimization. For example a simple min function

func min(a []int) int {
        if a == nil {
                return 0
        }
        var min = a[0]
        for i := 0; i < len(a); i++ {
                if a[i] < min {
                        min = a[i]
                }
        }
        return min
}

A possible mutaton in this code is for the if a[i] < min to become if a[i] <= min but the output wont change ... which I would not consider a bug. This is the same for any such optimization.

Looking at awesome-mutation-testing there is currently only one not abandoned mutation testing tool for golang go-mutesting which is not all that active as well.
The awesome-mutation-testing has links to some articles and thesis that are probably good idea to be read while/when/if we try mutation testing.
My original try of go-mutesting shows that it somewhat not fast enough for automatic testing but probably is good enough for some prerelease testing and maybe running it locally every month/week or something like that.
I did try it on the stats package by running

go-mutesting ./...

inside of the stats package
the end result is

The mutation score is 0.278020 (191 passed, 313 failed, 125 duplicated, 183 skipped, total is 687)

which took 16 minutes 39 seconds. I intend on running it on the whole codebase during the weekend (as it will probably take hours if not days). The end score of 0.27 is ... bad but also a lot of the failed mutations are because of optimizations

--- cloud/config.go     2019-03-20 14:31:18.816996005 +0200
+++ /tmp/go-mutesting-081972372/cloud/config.go.18      2019-05-17 11:43:58.346697075 +0300
@@ -185,7 +185,7 @@
        if cfg.NoCompress.Valid {
                c.NoCompress = cfg.NoCompress
        }
-       if cfg.ProjectID.Valid && cfg.ProjectID.Int64 > 0 {
+       if cfg.ProjectID.Valid && cfg.ProjectID.Int64 >= 0 {
                c.ProjectID = cfg.ProjectID
        }
        if cfg.MetricPushInterval.Valid {

FAIL "/tmp/go-mutesting-081972372/cloud/config.go.18" with checksum fe8876f2a442c1ae19d82b4d5cd5c0ef

or log statements

--- kafka/collector.go  2019-03-20 14:31:18.823996042 +0200
+++ /tmp/go-mutesting-081972372/kafka/collector.go.4    2019-05-17 11:50:32.564396058 +0300
@@ -72,7 +72,7 @@

                        err := c.Producer.Close()
                        if err != nil {
-                               log.WithError(err).Error("Kafka: Failed to close producer.")
+
                        }
                        return
                }

FAIL "/tmp/go-mutesting-081972372/kafka/collector.go.4" with checksum 40483ae872146eff68a246de2e146533

also the package in question has 71% coverage which probably means that a lot of the reported mutations are for code that is not covered where whatever change is made as long as it compiles it will give the same result as previously.

Also unfortunately the current blacklsting of changes is through a file with the md5sum of the change that should not be tested ... which as is mentioned in the documentation will mean that if the source code changes some of those md5sums will no longer be true.

Another problem with go-mutesting is the fact that you will need to read it output to see what changes failed and copy-paste the patch and apply it yourself which can be somewhat time consuming :(.

With everything here said I think it is still good idea to do some more testing and maybe search for better ways to blacklist changes we don't care about. I doubt that we will ever put this in the CI/CD pipeline but maybe we can run it every month or when making huge changes such as #1007 to see if the new tests actually cover a lot of the corner cases. Unfortunately it also needs high test coverage as all not covered lines will produce errors. There is an issue to make go-mutesting coverage aware but there is no movement as far as I can see.
I am going to look through the reported issues and see if I can find some that are covered but mutation the code still produces errors.

[mstoykov]

I am closing this as it hasn't been picked up in 4 years and there is not really any specific information here