add nightly benchmark

This should ensure we don't creep little percentages over the course of multiple commits into main, compared to the last release branch. Signed-off-by: Thomas Jungblut <[email protected]>
etcd-io · Jul 9, 2024 · ab8520b · ab8520b
1 parent 46d9b10
commit ab8520b
Showing 1 changed file with 47 additions and 0 deletions.
diff --git a/.github/workflows/benchmark-releases.yaml b/.github/workflows/benchmark-releases.yaml
@@ -0,0 +1,47 @@
+---
+name: Benchmarks nightly between last release and current main on AMD64
+permissions: read-all
+on:
+  schedule:
+    - cron: '25 5 * * *' # runs every day at 09:25 UTC
+  # workflow_dispatch enables manual testing of this job by maintainers
+  workflow_dispatch:
+jobs:
+  benchmark-pull-request:
+    runs-on: ubuntu-latest-8-cores
+    steps:
+    - uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4.1.4
+      with:
+        fetch-depth: 0
+    - id: goversion
+      run: echo "goversion=$(cat .go-version)" >> "$GITHUB_OUTPUT"
+    - uses: actions/setup-go@0c52d547c9bc32b1aa3301fd7a9cb496313a4491 # v5.0.0
+      with:
+        go-version: ${{ steps.goversion.outputs.goversion }}
+    - name: Run Benchmarks
+      run: |
+        BENCHSTAT_OUTPUT_FILE=result.txt make test-benchmark-compare REF=release-1.3
+    - run: |
+        echo "\`\`\`" >> "$GITHUB_STEP_SUMMARY"
+        cat result.txt >> "$GITHUB_STEP_SUMMARY"
+        echo "\`\`\`" >> "$GITHUB_STEP_SUMMARY"
+        cat <<EOL >> "$GITHUB_STEP_SUMMARY"
+        <hr />
+        The table shows the median and 90% confidence interval (CI) summaries for each benchmark comparing the HEAD and the BASE of the pull request, and an A/B comparison under "vs base". The last column shows the statistical p-value with ten runs (n=10).
+        The last row has the Geometric Mean (geomean) for the given rows in the table.
+        Refer to [benchstat's documentation](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat) for more help.
+        EOL
+    - name: Validate results under acceptable limit
+      run: |
+        export MAX_ACCEPTABLE_DIFFERENCE=5
+        while IFS= read -r line; do
+          # Get fourth value, which is the comparison with the base.
+          value="$(echo "$line" | awk '{print $4}')"
+          if [[ "$value" = +* ]] || [[ "$value" = -* ]]; then
+            if (( $(echo "${value//[^0-9.]/}"'>'"$MAX_ACCEPTABLE_DIFFERENCE" | bc -l) )); then
+              echo "::error::$value is above the maximum acceptable difference ($MAX_ACCEPTABLE_DIFFERENCE)"
+              exit 1
+            fi
+          fi
+        done < <(grep geomean result.txt)
+