bazel: low cache utilization between builds and tests

[irfansharif]

Describe the problem

I'm observing that the build artifacts from a {dev,bazel} build are not being used by {dev,bazel} test and vice-versa. When going from building the full cockroach-short binary to running a specific test, the build cache is trampled upon by each command with the results no longer usable by the other or by itself afterwards. This is surprising, is it because of the differing .bazelrc configs between the two? Something else?

To Reproduce

Some numbers from my machine, I'm seeing each step start the build from scratch(-ish) without achieving much cache utilization.

$ dev test pkg/spanconfig/spanconfigstore # 116.655s
$ dev build cockroach-short               # 210.004s
$ dev test pkg/spanconfig/spanconfigstore # 97.852s
$ dev build cockroach-short               # 176.392s

Here are two profiles (open using chrome://tracing), they show that we're pretty much starting from scratch each time.

• dev.gz (for an unexpected long build step)
• test.gz (ditto, for test)

Expected behavior

Build artifacts from a dev build cockroach-short should be re-usable for tests and vice-versa.

Epic CRDB-8036

[rickystewart]

Unfortunately the build and tests are compiled with disjoint gotags (crdb_test for tests, crdb_test_off for non-tests) which erases much of Bazel's ability to cache between the two. I'd be very interested in any changes that allow us to get rid of crdb_test{,_off} to solve this.

[rickystewart]

We could add some select()s in the BUILD files to enable conditional compilation, and Bazel would have an easier time caching that stuff, but without getting rid of the gotags you still wouldn't see any improvement.

[stevendanna]

I've been playing with this locally recently and I think explicitly setting --disk_cache in .bazelrc.user helps with with the runtime substantially for me at least. You still don't get sharing between test and non-test artifacts, but you do get to keep around build artifacts from more than just the previous run. One issue is that the disk cache grows without bound, so it is occasionally useful to delete it and do a full rebuild.

[jordanlewis]

crdb_test is an important piece of the way that we test CockroachDB - it's how we enable metamorphic and other expensive checks in our tests, without affecting the performance of our production builds. I don't think it's likely that we'd prioritize removing it anytime soon.

I guess I would expect that Bazel would keep separate caches for each configuration of the gotags, is that not how things work?

[irfansharif]

A slack thread about the bazel in-use space here.

[irfansharif]

Using an explicit --disk_cache seems to have helped in that build artifacts for dev build and dev test are both kept around for re-use. Something that isn't shared however is the analysis cache -- it seems that bazel keeps only one analysis in the cache and readily discards it whenever the relevant options change (for us it's the buildtags). Filed an issue upstream: bazelbuild/bazel#14179.

[irfansharif]

crdb_test is an important piece of the way that we test CockroachDB

Instead of using a build tag (which necessarily generates non-shareable artifacts), I wonder if we can get away with using an init time envvar check instead. Downside: we'd have to compile test-only code for regular builds (will it be much slower?). Upside: we'd be able to re-use artifacts across dev build and dev test.