workflow: add cockroach-microbench-ci step in github actions

Fixes: cockroachdb#106661
Release note: None
Epic: https://cockroachlabs.atlassian.net/browse/CRDB-29662

This change intends to add a new CI step which will run `BenchmarkKV` microbenchmark for the current commit and compares the result with that of the base commit's results which is stored in a cloud bucket.

This is needed to avoid minor perf regressions so that any regression is caught in the PR itself. The current threshold is set to 5%.

There are highly 3 steps
1. Run benchmarks
2. If this is PR branch, download the base commit's bench results from the cloud storage and compare the results. If there's a regression fail here itself
3. If it is a release/master branch, push the result to the cloud storage with the current commit sha.

In cases that a regression is expected, the developers can add [PERF_REGRESSION_EXPECTED] in the PR title which skips the step. (only for PR branches)

<what was there before: Previously, ...>
<why it needed to change: This was inadequate because ...>
<what you did about it: To address this, this patch ...>

.github/workflows/github-actions-essential-ci.yml

@@ -344,3 +344,29 @@ jobs:
       - name: clean up
         run: ./build/github/cleanup-engflow-keys.sh
         if: always()
+  cockroach-microbench-ci:
+    runs-on: [ self-hosted, basic_runner_group ]
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha || github.ref }}
+      - name: check 'regression-expected' label in title
+        id: check-title
+        uses: actions/github-script@v5
+        with:
+          script: |
+            const title = context.payload.pull_request.title;
+            const skipComparison = title.includes("[PERF_REGRESSION_EXPECTED]");
+            console.log(`Does PR title contain [PERF_REGRESSION_EXPECTED]? ${skipComparison}`);
+            core.exportVariable('SKIP_COMPARISON', skipComparison);
+      - name: run scripts
+        run: ./build/github/cockroach-microbench-ci.sh
+        env:
+          COMPARE_THRESHOLD: "0.05"
+          PUSH_STEP: ${{ github.event_name == 'push' }}
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+      - name: clean up
+        run: ./build/github/cleanup-engflow-keys.sh
+        if: always()
+

build/github/cockroach-microbench-ci.sh

@@ -0,0 +1,62 @@
+#!/bin/bash
+
+set -euxo pipefail
+
+# Directories and file names
+output_dir="./artifacts/microbench"
+cleaned_current_dir="$output_dir/current"
+cleaned_base_dir="$output_dir/base"
+
+# hardcoded the file name as of now since `compare` requires a particular format for the file name.
+benchmark_file_name="pkg_sql-report.log"
+log_output_file_path="$output_dir/microbench.log"
+
+storage_bucket_url="gs://cockroach-microbench-ci"
+
+# Threshold for comparison
+threshold=$COMPARE_THRESHOLD
+exit_status=0
+
+# Configure Bazel and dev tooling
+cat <<EOF > ./.bazelrc.user
+build --config nolintonbuild
+build --remote_cache=http://127.0.0.1:9867
+test --test_tmpdir=/tmp/cockroach
+test --config crosslinux
+EOF
+
+# Exit early if SKIP_COMPARISON is true and not a push step
+if $SKIP_COMPARISON && ! $PUSH_STEP; then
+  echo "Exiting since skip comparison is enabled and this is not a push step."
+  exit $exit_status
+fi
+
+./dev doctor
+./dev build roachprod-microbench
+
+# Run benchmarks and clean output
+mkdir -p "$output_dir"
+
+# running 4 here because that's the minumum required for a comparison
+./dev bench pkg/sql/tests '--filter=^BenchmarkKV$' --count 4 --verbose --bench-mem > "$log_output_file_path"
+./bin/roachprod-microbench clean "$log_output_file_path" "$cleaned_current_dir/$benchmark_file_name"
+
+# Push artifact if this is a base merge and skip comparison
+if $PUSH_STEP; then
+  gcloud storage cp "$cleaned_current_dir/$benchmark_file_name" "$storage_bucket_url/$GITHUB_HEAD_REF/$GITHUB_SHA.log"
+  echo "Skipping comparison since this is a base merge."
+  exit $exit_status
+fi
+
+# Compare benchmarks
+if ! gcloud storage cp "$storage_bucket_url/$GITHUB_BASE_REF/$BASE_SHA.log" "$cleaned_base_dir/$benchmark_file_name"; then
+  echo "Couldn't download base bench file, exiting."
+  exit $exit_status
+fi
+
+if ! ./bin/roachprod-microbench compare "$cleaned_base_dir" "$cleaned_current_dir" --threshold "$threshold"; then
+  echo "There is an error during comparison. If it's a perf regression, please fix it. If a regression is expected, add [PERF_REGRESSION_EXPECTED] in the PR head title."
+  exit 1
+fi
+
+exit $exit_status