kv: hand off a number of roachtests

[tbg]

At the time of writing, KV owns many roachtests:

bin/roachtest list tag:owner-kv | grep -cF '[kv]'
220

We expect that for a lot of them, we can transfer ownership at some point in the near future.

I prepared a spreadsheet with a v0 of proposed movement: https://docs.google.com/spreadsheets/d/1w_zx8W1ObHVL1oMXNjh4LTcEMkTHyCycbGSnji1BvCk/edit#gid=2080879335

cc @lidorcarmel (for visibility) @srosenberg (since TE is the would-be owner for a lot of the tests and we should discuss the specifics of such a handover). Note that PERF is just TE today, but I thought it would make sense to make that distinction early.

Jira issue: CRDB-16193

[nvanbenschoten]

@tbg what are the next steps here? To determine which tests can be handed off? To relabel those tests?

[tbg]

Test-Eng will take the perf tests mid-July and we'll hand of another batch after that (details TBD), possibly the version-upgrade tests.

[mwang1026]

Removing KV then since seems like Test Eng will take the lead on actually executing the change. Let me know if this is wrong @srosenberg

[tbg]

I was going to make the PR.

[blathers-crl]

cc @cockroachdb/replication

[tbg]

Still need to talk the schedule and selection through with @srosenberg but this is the rest that I believe should be owned by test-eng:

• Jepsen - if they fail then mostly due to flakes (apt-get etc). For actual test failure tutorial can reach out to KV if it fails (the average KV person doesn’t know how to investigate these either, but Tobi or Nathan do)
• tpcc/headroom/n4cpu16
• tpcc/mixed-headroom/n5cpu16
• tpcc/w=100/nodes=3/chaos=true
• version/mixed/nodes=3
• version/mixed/nodes=5
• weekly/tpcc/headroom

[tbg]

Here's what's left after #86300

$ roachtest list | grep -F '[kv]' | awk '{print $1}' | grep -Eo '^[^/]+' | uniq -c | sort
   1 autoupgrade
   1 drain-and-decommission
   1 drop
   1 gossip
   1 hotspotsplits
   1 inconsistency
   1 indexes
   1 ledger
   1 network
   1 overload
   1 queue
   1 restart
   1 slow-drain
   2 copy
   2 copyfrom
   2 kv50
   2 replicagc-changed-peers
   2 roachmart
   3 replicate
   3 transfer-leases
   4 loqrecovery
   4 splits
   5 decommission
   5 rebalance
   6 clock
   6 kv
   8 kv0bench
  10 decommissionBench
  11 acceptance
  13 follower-reads