workflow: add a stale issue bot for KV test failures

KV receives the bulk of roachtest and unit test falures. To deal
with this issue we have institued a new triage process for all
test failures. As part of the process we want to aggresively remove
flakes and old failures that are no longer relevant. This bot will
auto-close all KV test-failures (except skipped tests) after 35 days of
inactivity.

Release note: None

.github/workflows/kv-test-failure-stale.yml

@@ -0,0 +1,34 @@
+name: Mark stale test failure issues for KV
+
+on:
+  schedule:
+  - cron: "0 10 * * 1-4"
+  workflow_dispatch:
+
+jobs:
+  stale:
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+    steps:
+    - uses: actions/stale@v3
+      with:
+        operations-per-run: 1000
+        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        stale-issue-message: |
+           We have marked this test failure issue as stale because it has been 
+           inactive for 1 month. If this failure is still relevant, removing the 
+           stale label or adding a comment will keep it active. Otherwise, 
+           we'll close it in 5 days to keep the test failure queue tidy. 
+        stale-pr-message: 'Stale pull request message'
+        stale-issue-label: 'no-test-failure-activity'
+        stale-pr-label: 'no-pr-activity'
+        close-issue-label: 'X-stale'
+        close-pr-label: 'X-stale'
+        # Disable this for PR's, by setting a very high bar
+        days-before-pr-stale: 99999
+        days-before-issue-stale: 30
+        days-before-close: 5
+        only-labels: 'T-kv,C-test-failure'
+        exempt-issue-labels: 'skipped-test'