You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Coarse index generation takes as input the remote segment index and produces a kafka offset to file offset mapping with a step size equal to chunk size. There are two issues with the coarse index generation code:
It only examines the indices for file offset and kafka offsets. there is additional data in the _offsets fields. In certain cases where the segment is too small, the index fields may end up empty and all the data is contained in the offsets. This happens when the segment size is small enough that a write to the index does not take place. The coarse index should also examine the offset fields after it has gone through the index fields.
The write to coarse index uses mod comparison to determine which data to write to index, this can run into a bug where consecutive entries have the same mod value. A simpler approach is to keep a running sum and use it to write to coarse index.
The text was updated successfully, but these errors were encountered:
Coarse index generation takes as input the remote segment index and produces a kafka offset to file offset mapping with a step size equal to chunk size. There are two issues with the coarse index generation code:
_offsets
fields. In certain cases where the segment is too small, the index fields may end up empty and all the data is contained in the offsets. This happens when the segment size is small enough that a write to the index does not take place. The coarse index should also examine the offset fields after it has gone through the index fields.The text was updated successfully, but these errors were encountered: