storage,kv: tolerate corruption of sideloaded sstables

[jbowens]

If an AddSSTable's sstable is sideloaded and becomes corrupted (eg, due to a bad disk), the operator has no recourse other than to replace the node.

This issue is intended to track isolation of corruption of the raft log / sideloaded sstables, in contrast to #67568 which tracks recovery from corruption of already-applied state.

See #90834 for an example.

Jira issue: CRDB-21080

[blathers-crl]

cc @cockroachdb/replication

[erikgrinaker]

This is related to #75903, in that any failure to apply a Raft command will crash the node anyway. The proposed solution there is to cordon the replica (and then discard the faulty replica and upreplicate elsewhere, unless all replicas are faulty), which is likely the preferable approach here as well. That said, if the SST is corrupt then the disks are likely faulty, so we may not want to keep the node running and risk further corruption anyway.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!