-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Iceberg] Support procedure remove_orphan_files
#23267
[Iceberg] Support procedure remove_orphan_files
#23267
Conversation
e644017
to
a9066c5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will be nice to have! Just NITs from me
presto-iceberg/src/main/java/com/facebook/presto/iceberg/procedure/RemoveOrphanFiles.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/procedure/RemoveOrphanFiles.java
Outdated
Show resolved
Hide resolved
a9066c5
to
4bdb47b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the doc! Local doc build with the new table looks good. A minor suggested rephrase for active voice, and shortening for readability.
4bdb47b
to
0bf44a8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! (docs)
Pull updated branch, new local docs build, looks good. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few initial comments. Great to see us getting more parity with Iceberg's Spark procedures
presto-iceberg/src/main/java/com/facebook/presto/iceberg/procedure/RemoveOrphanFiles.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/procedure/RemoveOrphanFiles.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/procedure/RemoveOrphanFiles.java
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/procedure/RemoveOrphanFiles.java
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/procedure/RemoveOrphanFiles.java
Outdated
Show resolved
Hide resolved
0bf44a8
to
755e834
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two final things. Also, I would like to see a test which exercises the default expiry limit. I don't have a good idea on how to do that in a reasonable time frame within the test, so I am happy to leave as is if you think it would be difficult
presto-iceberg/src/main/java/com/facebook/presto/iceberg/procedure/RemoveOrphanFiles.java
Outdated
Show resolved
Hide resolved
...berg/src/test/java/com/facebook/presto/iceberg/procedure/TestRemoveOrphanFilesProcedure.java
Outdated
Show resolved
Hide resolved
755e834
to
be6df09
Compare
Seems it's indeed a bit difficult to test this scenario unless we change the implementation of |
Description
This PR support the procedure
remove_orphan_files
for iceberg. It can be used to remove files which are not referenced in any metadata files of an Iceberg table and can thus be considered "orphaned".See examples as follow:
db.sample
and older than specified timestamp::db.sample
and created 3 days ago (by default)::Motivation and Context
Support removing orphan files that are not referenced in any metadata files for Iceberg
Test Plan
TestRemoveOrphanFilesProcedure
Contributor checklist
Release Notes