roachtest: add replica-circuit-breakers

[tbg]

his adds a roachtest that runs TPCC (with a low warehouse count and
--tolerate-errors), loses quorum half-way through, and checks the
prometheus metrics of the workload for fail-fast behavior.

Touches #33007.

Epic: none

Release note: None

[cockroach-teamcity]

This change is 

[erikgrinaker]

Nice!

[srosenberg]

Tour de force of roachtests!

Reviewed 1 of 1 files at r1, 3 of 3 files at r2, 5 of 5 files at r3, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @otan and @tbg)

---

pkg/cmd/roachtest/tests/replica_circuit_breaker.go, line 200 at r3 (raw file):

			},
		)
		t.L().Printf("under healthy cluster: ~%.2f qps", min)

Shouldn't we also log max? It could help with debugging in case we're not able to hit anything higher than min for whatever reason.

---

pkg/cmd/roachtest/tests/replica_circuit_breaker.go, line 206 at r3 (raw file):

		// Starting at one minute after quorum was lost, we should see at least a
		// steady trickle of errors, or a steady trickle of successes. (If
		// miraculously no range lost quorum it would be all successes, if they all

Should we fail if a miracle does occur? Conversely, how do we know that this test is not always miraculous? If we can guarantee a loss of quorum, then a simple assertion on the non-zero count of workload_tpcc_newOrder_error_total would suffice.

---

pkg/cmd/roachtest/tests/replica_circuit_breaker.go, line 221 at r3 (raw file):

}

func visitPromRangeQuery(

It's a fairly useful helper function which could be made general enough to be reused by other tests; i.e., assertions about the range could be parameterized.

---

pkg/cmd/roachtest/tests/replica_circuit_breaker.go, line 256 at r3 (raw file):

	for _, val := range mtr.(model.Matrix)[0].Values {
		n++
		visit(timeutil.Unix(int64(val.Timestamp/1000), 0), val)

Why not val.Timestamp.Time()?

---

pkg/roachprod/install/cluster_synced.go, line 2043 at r1 (raw file):

	if len(failed) > 0 {
		var err error
		for _, res := range failed {

Do we even need failed? CombineErrors could simply replace the corresponding append(failed, r).

[srosenberg]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @otan and @tbg)

---

pkg/cmd/roachtest/tests/replica_circuit_breaker.go, line 136 at r3 (raw file):

				r := sqlutils.MakeSQLRunner(c.Conn(ctx, t.L(), 1))
				queryTS := ts.Add(-10 * time.Second).Format("2006-01-02 15:04:05")
				q1 := `SELECT count(*) FROM tpcc.order_line AS OF SYSTEM TIME '` + queryTS + `'`

Not sure I follow why q1 is not using the stronger form of AOST, i.e., with_min_timestamp?

[tbg]

Rebased this PR, maybe I broke something. No longer passes.

$ ./pkg/cmd/roachtest/roachstress.sh -c 3 replica-circuit-breaker tag:weekly
=== RUN   replica-circuit-breaker#2
=== RUN   replica-circuit-breaker#3
=== RUN   replica-circuit-breaker#1
Grafana dashboard 0: http://34.75.146.40:3000/
Grafana dashboard 0: http://34.148.31.31:3000/
Grafana dashboard 0: http://34.73.60.89:3000/
--- FAIL: replica-circuit-breaker#3 (451.36s)
test artifacts and logs in: artifacts/b8a78987c1-dirty/043215/replica-circuit-breaker/run_3
(test_impl.go:297).Fatalf: error executing 'SELECT count(*) FROM tpcc.order_line AS OF SYSTEM TIME '2022-11-02 04:37:05'': pq: inbox communication error: rpc error: code = Canceled desc = context canceled
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_043715.018671211_n6_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=50 --histograms=perf/stats.json --tolerate-errors --ramp=5m0s --duration=10m0s --prometheus-port=0 --pprofport=33333  {pgurl:1-5} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor task failed: t.Fatal() was called
--- FAIL: replica-circuit-breaker#2 (453.09s)
test artifacts and logs in: artifacts/b8a78987c1-dirty/043215/replica-circuit-breaker/run_2
(test_impl.go:297).Fatalf: error executing 'SELECT count(*) FROM tpcc.order_line AS OF SYSTEM TIME '2022-11-02 04:37:06'': pq: inbox communication error: rpc error: code = Canceled desc = context canceled
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_043716.222616076_n6_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=50 --histograms=perf/stats.json --tolerate-errors --ramp=5m0s --duration=10m0s --prometheus-port=0 --pprofport=33333  {pgurl:1-5} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor task failed: t.Fatal() was called

[tbg]

@yuzefovich do you know which context got canceled here (and ideally which timeout it had on it)?

pq: inbox communication error: rpc error: code = Canceled desc = context canceled

[yuzefovich]

Unfortunately, I don't think we can know for sure. Most likely this error indicates that the gRPC stream connecting two parts of the distributed execution plan was broken for whatever reason, and on the "inbox" (consumer) side of the stream it is seen as "context canceled".

[tbg]

I thought I could fix this by setting the circuit breaker timeout to something like 4s but same failure mode, sadly. Since I don't have time to dig in, I'll try with some of the checks removed.