wal: implement failoverManager.{Obsolete,Stats} and log recycling

[sumeerbhola]

Informs #3230

Informs CRDB-35401

[cockroach-teamcity]

This change is 

[sumeerbhola]

Reviewable status: 0 of 8 files reviewed, 1 unresolved discussion (waiting on @jbowens)

---

wal/failover_manager_test.go line 496 at r1 (raw file):

// TODO(sumeer): test failoverManager.{List,Obsolete,Stats}
// TODO(sumeer): test failoverWriter.getLog

The todos to test the new code. Existing tests pass.
If you need these changes merged to get unblocked, I can write the tests in a followup PR.

[sumeerbhola]

Reviewable status: 0 of 11 files reviewed, 1 unresolved discussion (waiting on @jbowens)

---

wal/failover_manager_test.go line 496 at r1 (raw file):

Previously, sumeerbhola wrote…

The todos to test the new code. Existing tests pass.
If you need these changes merged to get unblocked, I can write the tests in a followup PR.

tests are done

[jbowens]

Reviewed 5 of 8 files at r1, 2 of 5 files at r2, 1 of 2 files at r3, all commit messages.
Reviewable status: 8 of 11 files reviewed, 2 unresolved discussions (waiting on @sumeerbhola)

---

wal/failover_manager.go line 602 at r3 (raw file):

	wm.mu.Lock()
	defer wm.mu.Unlock()
	wm.mu.closedWALs = append(wm.mu.closedWALs, wm.mu.ww.getLog())

wdyt of passing the logicalLogWithSizesEtc in to the writerClosed callback to make this control flow a little more explicit?

[sumeerbhola]

TFTR!

Reviewable status: 5 of 11 files reviewed, 2 unresolved discussions (waiting on @jbowens)

---

wal/failover_manager.go line 602 at r3 (raw file):

Previously, jbowens (Jackson Owens) wrote…

wdyt of passing the logicalLogWithSizesEtc in to the writerClosed callback to make this control flow a little more explicit?

Done

[sumeerbhola]

following test failure is unrelated:

--- FAIL: TestGeneratorRandom (0.08s)
    --- FAIL: TestGeneratorRandom/config=default (0.08s)
panic: could not generate unique key [recovered]
	panic: could not generate unique key [recovered]
	panic: could not generate unique key