wal,record: failover write path code

failover_manager.go contains the failoverManager (which implements
wal.Manager) for the write path, and helper classes. dirProber monitors
the primary dir when failed over to use the secondary, to decide when to
failback to the primary. failoverMonitor uses the latency and error seen
by the current *LogWriter, and probing state, to decide when to switch to
a different dir.

failover_writer.go contains the failoverWriter, that can switch across
a sequence of record.LogWriters.

record.LogWriter is changed to accommodate both standalone and failover
mode, without affecting the synchronization in standalone mode.
In failover mode there is some additional synchronization, in the
failoverWriter queue, which is not lock free, but is hopefully fast
enough given the fastpath will use read locks.

Informs cockroachdb#3230