Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: ensure the checkpoint decoder is regularly flushed #2960

Merged
merged 2 commits into from
Oct 25, 2024

Conversation

rtyler
Copy link
Member

@rtyler rtyler commented Oct 24, 2024

For checkpoint buffers that cannot fit into the batch size the checkpoint will be written with an insufficient number of bytes.

Unfortunately our tests didn't catch this and it only manifested on tables with very large transaction logs

@github-actions github-actions bot added the binding/rust Issues for the Rust crate label Oct 24, 2024
Copy link

codecov bot commented Oct 24, 2024

Codecov Report

Attention: Patch coverage is 0% with 60 lines in your changes missing coverage. Please review.

Project coverage is 72.25%. Comparing base (8ba8e08) to head (a08179d).
Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/protocol/checkpoints.rs 0.00% 58 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2960      +/-   ##
==========================================
- Coverage   72.36%   72.25%   -0.11%     
==========================================
  Files         131      131              
  Lines       40602    40660      +58     
  Branches    40602    40660      +58     
==========================================
- Hits        29380    29378       -2     
- Misses       9335     9393      +58     
- Partials     1887     1889       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

For checkpoint buffers that cannot fit into the batch size the
checkpoint will be written with an insufficient number of bytes.

Unfortunately our tests didn't catch this and it only manifested on
tables with very large transaction logs

Signed-off-by: R. Tyler Croy <[email protected]>
The default batch size is too large to build an on-demand "large table"
so the test is ignored by default

Signed-off-by: R. Tyler Croy <[email protected]>
@rtyler rtyler marked this pull request as ready for review October 24, 2024 20:24
@rtyler rtyler enabled auto-merge October 24, 2024 20:24
@rtyler rtyler added this pull request to the merge queue Oct 25, 2024
Merged via the queue into delta-io:main with commit e19e21f Oct 25, 2024
22 checks passed
@rtyler rtyler deleted the flush-the-pipez branch October 25, 2024 05:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/rust Issues for the Rust crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants