Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added without_files flag to DeltaTable constructor #866

Merged
merged 5 commits into from
Oct 13, 2022

Conversation

MykhailoHevak
Copy link
Contributor

@MykhailoHevak MykhailoHevak commented Oct 7, 2022

Description

Added without_files flag to DeltaTable constructor. If the flag is set it will be called method without_files in the DeltaTableBuilder. And this allows users to control memory usage. Some append-only applications might have no need of tracking any files.

Documentation

Indicates whether DeltaTable should not track files. This defaults to "false" (files are tracked). Some append-only applications might have no need of tracking any files. So, DeltaTable will be loaded with significant memory reduction.

Copy link
Collaborator

@wjones127 wjones127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for submitting this! I have a few suggested changes.

What's the behavior of DeltaTable.file_uris() when without_files=True? Does it return empty? Or raise an error? Or load the files? We might want to add a unit test for that in Python.

python/deltalake/table.py Show resolved Hide resolved
python/deltalake/table.py Outdated Show resolved Hide resolved
@wjones127
Copy link
Collaborator

What's the behavior of DeltaTable.file_uris() when without_files=True? Does it return empty? Or raise an error? Or load the files? We might want to add a unit test for that in Python.

Just checked and it returns an empty list. We should document that for now, and consider changing that behavior in the future (#867).

@wjones127
Copy link
Collaborator

@MykhailoHevak looks like the merge you pushed overwrote the changes you accepted in the GitHub UI. Could you add those back?

@MykhailoHevak
Copy link
Contributor Author

@MykhailoHevak looks like the merge you pushed overwrote the changes you accepted in the GitHub UI. Could you add those back?

Oooops... I did it accidentally. Is it OK now?

@wjones127
Copy link
Collaborator

Oooops... I did it accidentally. Is it OK now?

The docstring change is still missing.

@MykhailoHevak MykhailoHevak requested review from wjones127 and removed request for rtyler, houqp, fvaleye and roeap October 13, 2022 15:45
@MykhailoHevak
Copy link
Contributor Author

Do I have to do something else?

python/src/lib.rs Outdated Show resolved Hide resolved
@wjones127 wjones127 merged commit 6492ab3 into delta-io:main Oct 13, 2022
@wjones127
Copy link
Collaborator

Thanks @MykhailoHevak!

@danielvrog
Copy link

danielvrog commented Oct 14, 2022

@wjones127 is there an estimation on when the new release is planned?
Thanks

P.S. closes #860

@MykhailoHevak MykhailoHevak deleted the without_files_flag branch October 14, 2022 16:55
@dennyglee
Copy link
Collaborator

Hey @MykhailoHevak I believe this is your first PR to the delta repo. Can you ping me (dennyglee) via the Delta Users slack so we can send you some swag? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants