build: add microbenchmark script for roachprod-microbench

[herkolategan]

Add a new nightlies teamcity build script that utilises the dev test-binaries and the roachprod-microbench command. The script can be used to store a microbenchmarks checkpoint for a given git revision to a bucket in gcs, by specifying a publish path. A roachprod cluster is created by the script, as specified by environment variables. The clean-up is managed by TeamCity.

Alternatively two binaries can be compared by supplying the related comparison binaries environment variables. This compare directory should be omitted in this case. The script requires a gcs path to download the comparison binaries from. The running time will be considerably longer, but there will be less variance in the results.

Subsequent runs can then be made with the same script to compare against a previous checkpoint, by supplying a compare path. The script takes numerous environment variables to configure the microbenchmarks cluster and arguments. These can be tweaked to configure different run variations and packages to target.

Resolves: #93893
See also: #90837, #91184

Epic: CRDB-20903

Release note: None

[cockroach-teamcity]

This change is 

[srosenberg]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @herkolategan)

---

-- commits line 4 at r1:
Should it be weekly?

---

build/teamcity/cockroach/nightlies/microbenchmark.sh line 21 at r1 (raw file):

"test --test_tmpdir=/tmp/cockroach"
)
printf "%s\n" "${bazelOpts[@]}" > ~/.bazelrc

I forget how/where $HOME is mapped inside the docker container. Could this have a side-effect on reusing the same TC agent with a different CI job?

---

build/teamcity/cockroach/nightlies/microbenchmark.sh line 51 at r1 (raw file):

  -libdir=./lib \
  -shell="$BENCH_SHELL" \
  ${BENCH_COMPARE_BINARIES:+-compare-binaries="$BENCH_COMPARE_BINARIES"} \

All of these env. vars can be overridden via the CI job? We should document their meaning and default value.

[herkolategan]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @srosenberg)

---

-- commits line 4 at r1:

Previously, srosenberg (Stan Rosenberg) wrote…

Should it be weekly?

True, the parent directory is sometimes confusing as this is where the weekly scripts live as well. I'll add the weekly postfix to the script as well.

---

build/teamcity/cockroach/nightlies/microbenchmark.sh line 21 at r1 (raw file):

Previously, srosenberg (Stan Rosenberg) wrote…

I forget how/where $HOME is mapped inside the docker container. Could this have a side-effect on reusing the same TC agent with a different CI job?

Good point, for assurance I'll change this to rather use .bazelrc.user in the pwd which should have the same effect.

---

build/teamcity/cockroach/nightlies/microbenchmark.sh line 51 at r1 (raw file):

Previously, srosenberg (Stan Rosenberg) wrote…

All of these env. vars can be overridden via the CI job? We should document their meaning and default value.

Good idea, I'll add some comments to the script.

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[srosenberg]

Reviewable status: complete! 1 of 0 LGTMs obtained

[herkolategan]

bors r=srosenberg

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[herkolategan]

Don't merge this yet, still need to switch over to the command line docker args vs. env docker args.

[herkolategan]

bors r=srosenberg

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)