Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot load version properly if there exists corresponding checkpoint #1403

Closed
ognis1205 opened this issue May 29, 2023 · 3 comments
Closed
Labels
bug Something isn't working

Comments

@ognis1205
Copy link
Contributor

ognis1205 commented May 29, 2023

Environment

Delta-rs version:
v.0.11.0

Binding:
Python

Environment:

  • Cloud provider: N/A
  • OS: macOS Monterey Version 12.3.1
  • Other:

Bug

What happened:
The load_version does not work correctly when there exists the corresponding checkpoint of the user specified version. This is due to the condition is always evaluated as false.

What you expected to happen:
The load_version loads a specified version of a table particularly when there exists the corresponding checkpoint.

How to reproduce it:
Notebooks

More details:

 $ tree _delta_log 
_delta_log
├── 00000000000000000000.checkpoint.parquet
├── 00000000000000000000.json
├── 00000000000000000001.json
├── 00000000000000000002.json
├── 00000000000000000003.json
├── 00000000000000000004.json
├── 00000000000000000005.json
├── 00000000000000000006.json
├── 00000000000000000007.json
├── 00000000000000000008.json
├── 00000000000000000009.json
├── 00000000000000000010.checkpoint.parquet
├── 00000000000000000010.json
:
@rtyler
Copy link
Member

rtyler commented Jan 3, 2024

I believe this is fixed, can you verify please?

@alexwilcoxson-rel
Copy link
Contributor

I don't believe it is, if you look at load_version, if the requested version is a checkpoint the state will get updated from the checkpoint, then when update_incremental is called it performs a check to see which version to load up to, however the table state version will equal the version requested and "fail" the check falling into getting the whole table state.

After loading the checkpoint if the table's version is equal to the requested update_incremental could not be called by load_version or itself could exit early.

@ion-elgreco
Copy link
Collaborator

I can't reproduce this anymore, so closing the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants