metamorphic: integrate IngestWithExcise #2885
Comments
bananabrick
added a commit
that referenced
this issue
Sep 22, 2023
Supports validation of virtual sstable block checksums during ingestion. It was easier to deduplicate the virtual sstables by file backing to ensure that the checksums for each file are only validated once. We could also implement a VirtualReader.ValidateVirtual function which only validates blocks which fall within virtual sstable bounds, but that seems like too much work for a feature which is off by default and was never turned on. This pr also prepares us for #2885. The metamorphic test might set `ValidateOnIngest` to true in which case we'll hit a panic.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Oct 31, 2023
This change deprecates and removes `generator.db` in the metamorphic package as it's duplicated by the `dbs` slice that was added to support multi-instance mode. All uses of `dbObjID` outside of tests are also updated to support multi-instance mode, and rely on deriving db IDs or having them passed-in instead of assuming one. Snapshot operations are also updated to track the underlying DB IDs, and are now re-enabled in multi-instance mode albeit with some limitations that will be addressed in cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 2, 2023
This change deprecates and removes `generator.db` in the metamorphic package as it's duplicated by the `dbs` slice that was added to support multi-instance mode. All uses of `dbObjID` outside of tests are also updated to support multi-instance mode, and rely on deriving db IDs or having them passed-in instead of assuming one. Snapshot operations are also updated to track the underlying DB IDs, and are now re-enabled in multi-instance mode albeit with some limitations that will be addressed in cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 3, 2023
This change deprecates and removes `generator.db` in the metamorphic package as it's duplicated by the `dbs` slice that was added to support multi-instance mode. All uses of `dbObjID` outside of tests are also updated to support multi-instance mode, and rely on deriving db IDs or having them passed-in instead of assuming one. Snapshot operations are also updated to track the underlying DB IDs, and are now re-enabled in multi-instance mode albeit with some limitations that will be addressed in cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 3, 2023
This change deprecates and removes `generator.db` in the metamorphic package as it's duplicated by the `dbs` slice that was added to support multi-instance mode. All uses of `dbObjID` outside of tests are also updated to support multi-instance mode, and rely on deriving db IDs or having them passed-in instead of assuming one. Snapshot operations are also updated to track the underlying DB IDs, and are now re-enabled in multi-instance mode albeit with some limitations that will be addressed in cockroachdb#2885.
itsbilal
added a commit
that referenced
this issue
Nov 3, 2023
This change deprecates and removes `generator.db` in the metamorphic package as it's duplicated by the `dbs` slice that was added to support multi-instance mode. All uses of `dbObjID` outside of tests are also updated to support multi-instance mode, and rely on deriving db IDs or having them passed-in instead of assuming one. Snapshot operations are also updated to track the underlying DB IDs, and are now re-enabled in multi-instance mode albeit with some limitations that will be addressed in #2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 8, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 8, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 8, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 9, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 10, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 10, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 10, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 14, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 15, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 15, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 15, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 16, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 16, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 29, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes cockroachdb#2885.
itsbilal
added a commit
that referenced
this issue
Nov 29, 2023
This change updates the metamorphic tests to support a new useExcise testOption and a new IngestAndExcise operation, which calls db.IngestAndExcise() if useExcise is true, or calls it with an sstable containing DeleteRange/RangeKeyDelete for the same span if useExcise is false. Furthermore, a new private option is added, efosAlwaysCreatesIterators. If the test option for useExcise (or the previously added useSharedReplicate) is enabled, one of these two things happen to guarantee EFOS determinism around excises: 1) If efosAlwaysCreatesIterators is true, the behaviour of EFOS is changed to always hold onto a readState and never return errors from NewIter. 2) If efosAlwaysCreatesIterators is false, regular EFOS behaviour is retained, but a flush is completed after every EFOS instantiation to guarantee that it becomes excise-immune right at instantiation. Note that regular snapshots are never created when useExcise or useSharedReplicate is true. Fixes #2885.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The excise operation is currently untested by the metamorphic tests. Integration could've caught issues like the one fixed in #2884.
The text was updated successfully, but these errors were encountered: