roachtest: failover variant for partial network partitions

[erikgrinaker]

We should add failover roachtest variants that benchmark range unavailability in the case of partial network partitions. There are two main variants, where all nodes can reach the liveness leaseholder: range leaseholder partitioned away from Raft leader, and SQL gateway partitioned away from range leaseholder.

See also internal doc.

Jira issue: CRDB-23045

Epic CRDB-25212

[blathers-crl]

cc @cockroachdb/replication

[erikgrinaker]

@andrewbaptist Note that we already have roachtest variants for asymmetric partitions, which is what's most relevant for #84289: failover/non-system/blackhole-recv and failover/non-system/blackhole-send.

cockroach/pkg/cmd/roachtest/tests/failover.go

Lines 43 to 51 in 42e6639

	r.Add(registry.TestSpec{
	Name: fmt.Sprintf("failover/non-system/%s", failureMode),
	Owner: registry.OwnerKV,
	Timeout: 30 * time.Minute,
	Cluster: r.MakeClusterSpec(7, spec.CPU(4)),
	Run: func(ctx context.Context, t test.Test, c cluster.Cluster) {
	runFailoverNonSystem(ctx, t, c, failureMode)
	},
	})

The pMax recovery time is graphed nightly, and looks terrible because we just don't handle these failures at all (60 seconds is the current maximum latency the tests measure).

• https://roachperf.crdb.dev/?filter=&view=failover%2Fnon-system%2Fblackhole-recv&tab=gce
• https://roachperf.crdb.dev/?filter=&view=failover%2Fnon-system%2Fblackhole-send&tab=gce

[erikgrinaker]

This is not done yet, the test in #95394 does not cover two important classes of partition: leader/leaseholder and leaseholder/gateway.

[erikgrinaker]

I'll pick this up as part of the Raft prevote work.