fix: use table config target file size, expose target_file_size in python #2811
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ion-elgreco
requested review from
wjones127,
roeap,
rtyler and
hntd187
as code owners
ion-elgreco
force-pushed
the
fix/use_table_target_file_size
branch
from
1893da3
to
42f73b1
Compare
auto-merge was automatically disabled
August 21, 2024 20:30
Pull request was converted to draft
ion-elgreco
changed the title
ion-elgreco
force-pushed
the
fix/use_table_target_file_size
branch
from
aee6132
to
0388ab6
Compare
rtyler
reviewed
Aug 30, 2024
crates/core/src/operations/mod.rs
Outdated
| /// Get the target_file_size from the table configuration in the sates | ||
| /// If table_config does not exist (only can occur in the first write action) it takes | ||
| /// the configuration that was passed to the writerBuilder. | ||
| pub fn get_target_file_size( |
There was a problem hiding this comment.
I don't think this should become part of our public API, so I'm going to slap a crate level visibility on it.
There was a problem hiding this comment.
Woops, my bad, should have been internal indeed
rtyler
force-pushed
the
fix/use_table_target_file_size
branch
from
0388ab6
to
452f8a3
Compare
rtyler
previously approved these changes
Aug 30, 2024
rtyler
force-pushed
the
fix/use_table_target_file_size
branch
from
452f8a3
to
60a1080
Compare
rtyler
approved these changes
Sep 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
If the target_file_size is None but we have a snapshot, then we should take it from the table config, otherwise we try to fetch it from the provided configuration since the setting might be in there.
If you pass delta.targetFileSize to the configuration, this will be used even on first writes. Additionally target_file_size can be passed to the rust writer, which takes precedence over any table config that has been set
Related issues