backupccl: memory monitored restore processor erroneously restores deleted data

[stevendanna]

Describe the problem

The new memory monitored restored attempts to limit the number of SSTs added to a single iterator. When a given span has more SSTs that would be allowed by the memory monitor, it attempts to process one set of SSTs first, and then the second set.

This, however, poses a problem for deletions. Currently, we do not write deletions directly, rather we depend on the iterator never returning a deleted key. This assumption was correct when all SSTs related to a key were definitely in the same iterator. It is no longer correct when the SSTs for a given iterator can be split over multiple iterators.

Proposed work to fix this

To fix this we think we need to (1) ensure that all SSTs for a given layer are all inside the same iterator and (2) change our usage of the iterator to raise deletion tombstones (both point and range keys) and then explicitly write those deletions during the restore process.

• Unit test that reproduces the between-layer issue
• Unit test that reproduces in in-layer issue
• Pass layer information in RestoreSpanSpec
• Ensure all SSTs from a single layer are added to a restore iterator
• Raise point deletion tombstones in ReadAsOfInterator
• Raise range tombstones in ReadAsOfIterator
• Correctly write range tombstones during restore

See

• ccl/backupccl: add memory monitor to external SST iterators in restore #93324
• backupccl: disable restore data processor memory monitoring by default #103301

Jira issue: CRDB-27949

Epic CRDB-28050

[blathers-crl]

cc @cockroachdb/disaster-recovery