roachtests: introduce admission-control/elastic-backup

[irfansharif]

Informs #89208. This test sets up a 3-node CRDB cluster on 8vCPU machines running 1000-warehouse TPC-C with an aggressive (every 20m) full backup schedule. We've observed latency spikes during backups because of its CPU-heavy nature -- it can elevate CPU scheduling latencies which in turn translates to an increase in foreground latency. In #86638 we introduced admission control mechanisms to dynamically pace such work while maintaining acceptable CPU scheduling latencies (sub millisecond p99s). This roachtest exercises that machinery. In future commits we'll add libraries to the roachtest package to automatically spit out the degree to which {CPU-scheduler,foreground} latencies are protected.

Release note: None

[cockroach-teamcity]

This change is 

[irfansharif]

12:42:25 util.go:40: waiting for initial up-replication... (<2m0s)
12:42:25 util.go:59: still waiting for full replication (52 ranges left)
12:42:26 util.go:59: still waiting for full replication (42 ranges left)
...
12:43:08 util.go:59: still waiting for full replication (2 ranges left)
12:43:09 util.go:59: still waiting for full replication (1 ranges left)
12:43:10 util.go:55: up-replication complete
12:43:10 tpcc.go:175: test status: loading fixture (<10m0s)
12:49:42 tpcc.go:191: test status: finished tpc-c setup
12:49:42 tpcc.go:254: test worker status: running tpcc worker=0 warehouses=1000 ramp=5m0s duration=1h30m0s on {pgurl:1-3} (<1m0s)
12:49:42 admission_control_elastic_backup.go:90: test status: during: enabling admission control (<30s)
12:49:42 admission_control_elastic_backup.go:95: test status: during: creating full backup schedule to run every 20m (<1m0s)
12:49:42 admission_control_elastic_backup.go:104: test status: during: waiting for workload to finish (<1h30m0s)
14:24:50 test_runner.go:942: tearing down after success; see teardown.log

Here's a prometheus dump of an experiment run. To see it, run the contained ./prometheus-docker-run.sh and point to localhost:9090 through your grafana instance (brew install grafana; brew services start grafana + localhost:3000 if you're on macs). http://go.crdb.dev/p/backup-admission-control-grafana is the set of dashboards I've been using.

elastic-backup.zip

[andrewbaptist]

Reviewed 2 of 7 files at r1.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @irfansharif and @renatolabs)

---

pkg/cmd/roachtest/tests/admission_control_snapshot_overload.go line 92 at r1 (raw file):

			t.Status(fmt.Sprintf("setting up prometheus/grafana (<%s)", 2*time.Minute))
			{

Nit: Remove this change from this PR, or look at my other PR which pulls this into admission_control.go.

---

pkg/cmd/roachtest/tests/util.go line 121 at r1 (raw file):

		"admission.sql_kv_response.enabled",
		"admission.sql_sql_response.enabled",
		"admission.elastic_cpu.enabled",

Its not clear that we want to enable this in all tests where admission control is enabled. Calling either true or false for this method does not result in the same as not setting anything. I'm concerned that users of this method won't do the right thing.

[irfansharif]

Thanks for the speedy review!

bors r+

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andrewbaptist and @renatolabs)

---

pkg/cmd/roachtest/tests/admission_control_snapshot_overload.go line 92 at r1 (raw file):

Previously, andrewbaptist (Andrew Baptist) wrote…

Nit: Remove this change from this PR, or look at my other PR which pulls this into admission_control.go.

If you notice I'm using a different grafana dashboard for the backup roachtest vs. the snapshot one. Quickly skimming through #89324, I think instead of a shared constructor for prometheus.Config (which itself is a constructor param for setting up prometheus), perhaps we ought to add some smarts to setupPrometheusForRoachtest or create a "default prometheus.Config" that assumes: - last-node-houses-prom/workload-generator

• every-other-node-runs-crdb, and
• every-other-node-should-run-node-exporter

And then tests can use that + annotate it with their custom dashboards. I'll keep this patch as is and leave the refactoring to a future commit/#89324.

---

pkg/cmd/roachtest/tests/util.go line 121 at r1 (raw file):

Previously, andrewbaptist (Andrew Baptist) wrote…

Its not clear that we want to enable this in all tests where admission control is enabled. Calling either true or false for this method does not result in the same as not setting anything. I'm concerned that users of this method won't do the right thing.

The only uses of this are tests we already own or shall soon enough 😛
The current uses look ok to me.

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)