storage: add WAL failover configuration

[jbowens]

Introduce support for configuring a multi-store CockroachDB node to failover a
store's write-ahead log (WAL) to another store's data directory. Failing over
the write-ahead log may allow some operations against a store to continue to
complete despite temporary unavailability of the underlying storage.

Customers must opt into WAL failover by passing --wal-failover=among-stores
to cockroach start or setting the env var
COCKROACH_WAL_FAILOVER=among-stores. On start, cockroach will assign each
store another store to be its failover destination. Cockroach will begin
monitoring the latency of all WAL writes. If latency to the WAL exceeds the
value of the storage.wal_failover.unhealthy_op_threshold cluster setting,
Cockroach will attempt to write WAL entries to its secondary store's volume.

If a user wishes to disable WAL failover, they must restart the node setting
--wal-failover=disabled.

Close #119418.
Informs cockroachdb/pebble#3230
Epic: CRDB-35401

Release note (ops change): Introduces a new start option (--wal-failover or
COCKROACH_WAL_FAILOVER env var) to opt into failing over WALs between stores in
multi-store nodes. Introduces a new storage.wal_failover.unhealthy_op_threshold
cluster setting for configuring the latency threshold at which a WAL write is
considered unhealthy.

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[sumeerbhola]

Reviewed 11 of 11 files at r1, 8 of 8 files at r2, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @itsbilal and @jbowens)

---

pkg/server/config.go line 761 at r1 (raw file):

	defer storeEnvs.CloseAll()

	walFailoverConfig := storage.WALFailover(cfg.WALFailover, storeEnvs)

I don't see storage.WALFailover on master or in this commit.

---

pkg/storage/open.go line 292 at r2 (raw file):

					UnhealthyOperationLatencyThreshold: func() time.Duration {
						return walFailoverUnhealthyOpThreshold.Get(&cfg.Settings.SV)
					},

We should probably reduce FailoverOptions.UnhealthySamplingInterval to 25ms given the default for the cluster setting is 100ms. Polling every 25ms is cheap enough.
Also FailoverOptions.ElevatedWriteStallThresholdLag doesn't seem to have a default in Pebble (oversight?). Something like 60s may be ok.
And HealthyInterval default of 2min is too long, I think. Something like 15s seems more reasonable.

Of course, these are all guesses without experimental validation under real conditions.

[sumeerbhola]

This turned out very clean!

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @itsbilal and @jbowens)

[jbowens]

TFTR!

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @itsbilal and @sumeerbhola)

---

pkg/server/config.go line 761 at r1 (raw file):

Previously, sumeerbhola wrote…

I don't see storage.WALFailover on master or in this commit.

you found it, right?

---

pkg/storage/open.go line 292 at r2 (raw file):

Previously, sumeerbhola wrote…

We should probably reduce FailoverOptions.UnhealthySamplingInterval to 25ms given the default for the cluster setting is 100ms. Polling every 25ms is cheap enough.
Also FailoverOptions.ElevatedWriteStallThresholdLag doesn't seem to have a default in Pebble (oversight?). Something like 60s may be ok.
And HealthyInterval default of 2min is too long, I think. Something like 15s seems more reasonable.

Of course, these are all guesses without experimental validation under real conditions.

Put up cockroachdb/pebble#3413 to change them there

[sumeerbhola]

Reviewed 5 of 8 files at r4.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @itsbilal and @jbowens)

---

pkg/server/config.go line 761 at r1 (raw file):

Previously, jbowens (Jackson Owens) wrote…

you found it, right?

yep. But isn't it in the wrong commit, in that CockroachDB won't build after the first commit?

[jbowens]

TFTR!

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @itsbilal and @sumeerbhola)

---

pkg/server/config.go line 761 at r1 (raw file):

Previously, sumeerbhola wrote…

yep. But isn't it in the wrong commit, in that CockroachDB won't build after the first commit?

whoops, sorry, I somehow missed this. Moved it into second commit and verified that both commits build

[jbowens]

bors r=sumeerbhola

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[rickystewart]

bors r-

This looks like merge skew to me.

[craig]

Canceled.

[jbowens]

I believe the other commit within the batch (from #119885) contains the skew.

Edit: They both did

bors r=sumeerbhola

[craig]

Canceled.

[jbowens]

bors r=sumeerbhola

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)
• check_generated_code
• docker_image_amd64
• examples_orms
• macos_amd64_build
• macos_arm64_build
• windows_build