-
Notifications
You must be signed in to change notification settings - Fork 456
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert v1 to v2 range-del blocks on the fly #146
Conversation
I need to add testing for the v1 -> v2 transformation, which is why this is still marked [WIP]. The first commit is ready to go and something I just stumbled upon. I'm not entirely settled on that approach. Perhaps it would be better for |
bdc377f
to
bfc7eb3
Compare
Tests have been added. I did this by adding a private |
Provide stability for the keys returned from a blockIter across positioning calls (Seek*, First, Last, Prev, Next) when the block has a restart interval of 1. This is achieved by noticing when a key does not share any bytes with the previous key and changing the returned key to point directly into the block instead of the key buffer. This behavior was already being expected by `rangedel.Truncate` and by forthcoming code to transform v1 range-del blocks to v2 format on the fly.
RocksDB generates range-del blocks that contain range tombstones that are not fragmented and sorted. Pebble generates range-del blocks containing tombstones that are fragmented and sorted. The fragmented and sorted range-del blocks are suitable for serving from directly, while the unfragmented/unsorted blocks are not. We now transform RocksDB format range-del blocks into Pebble format blocks on the fly. The transformation is done via a hook that allows passing a block through a function after it has been read from disk but before it is added to the cache. Fixes #68
bfc7eb3
to
13d77b0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 2 of 2 files at r1, 5 of 5 files at r2, 2 of 2 files at r3.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ajkr)
TFTR! |
RocksDB generates range-del blocks that contain range tombstones that are
not fragmented and sorted. Pebble generates range-del blocks containing
tombstones that are fragmented and sorted. The fragmented and sorted
range-del blocks are suitable for serving from directly, while the
unfragmented/unsorted blocks are not. We now transform RocksDB format
range-del blocks into Pebble format blocks on the fly.
The transformation is done via a hook that allows passing a block through a
function after it has been read from disk but before it is added to the
cache.
Fixes #68
Provide key stability across blockIter positioning calls
Provide stability for the keys returned from a blockIter across positioning
calls (Seek*, First, Last, Prev, Next) when the block has a restart
interval of 1. This is achieved by noticing when a key does not share any
bytes with the previous key and changing the returned key to point directly
into the block instead of the key buffer. This behavior was already being
expected by
rangedel.Truncate
and by forthcoming code to transform v1range-del blocks to v2 format on the fly.