storage: migrate RaftTombstoneKey to the unreplicated key space

[petermattis]

RaftTombstoneKey currently resides in the replicated range-local key space. This means that it will be included in snapshots and was being included (incorrectly) in consistency checks. #12131 added a hack to not include RaftTombstoneKey in consistency checks, but the real fix is to make it an unreplicated key.

Note that RaftTombstoneKey is special in that it is only supposed to exist if there is no corresponding replica data.

A migration would have to perform a one time loop over all of the tombstone keys for a store similar to what is done in Store.IterateRangeDescriptors.

[tbg]

Nominating for 2.1. This is a big wart I've been running into for the last couple of days. cc @awoods187.

[tbg]

For example, any invocation of ComputeStatsForRange is getting this wrong:

// ComputeStatsForRange computes the stats for a given range by
// iterating over all key ranges for the given range that should
// be accounted for in its stats.
func ComputeStatsForRange(
	d *roachpb.RangeDescriptor, e engine.Reader, nowNanos int64,
) (enginepb.MVCCStats, error) {
	iter := e.NewIterator(false)
	defer iter.Close()

	ms := enginepb.MVCCStats{}
	for _, r := range makeReplicatedKeyRanges(d) {
		msDelta, err := iter.ComputeStats(r.start, r.end, nowNanos)
		if err != nil {
			return enginepb.MVCCStats{}, err
		}
		ms.Add(msDelta)
	}
	return ms, nil
}

This isn't just some oddity, it's a full-on problem.

[tbg]

(This is called during splits, for example).

[tbg]

I'm doing this now.