Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add VACUUM operation as commit in transaction log #1728

Merged
merged 8 commits into from
Oct 30, 2023

Conversation

ion-elgreco
Copy link
Collaborator

@ion-elgreco ion-elgreco commented Oct 14, 2023

Description

Since deltalake 2.3.0 VACUUM operations are added in the transaction log, with VACUUM START, and VACUUM END. This PR will add these changes to the transaction log now for Delta-RS 🕺:

Examples of the commits:

{"commitInfo":{"timestamp":1697298818104,"operation":"VACUUM START","operationParameters":{"defaultRetentionMillis":"604800000","retentionCheckEnabled":"false","specifiedRetentionMillis":"0"},"clientVersion":"delta-rs.0.17.0","operationMetrics":{"numFilesToDelete":7, "sizeOfDataToDelete":1000}}}
{"commitInfo":{"timestamp":1697298818105,"operation":"VACUUM END","operationParameters":{"status":"COMPLETED"},"clientVersion":"delta-rs.0.17.0","operationMetrics":{"numDeletedFiles":7,"numVacuumedDirectories":0}}}

Related Issue(s)

Documentation

PR in delta repo: delta-io/delta#1552

@github-actions github-actions bot added binding/rust Issues for the Rust crate rust labels Oct 14, 2023
@github-actions
Copy link

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

@ion-elgreco ion-elgreco changed the title feature: add VACUUM operation as commit in transaction log feat: add VACUUM operation as commit in transaction log Oct 14, 2023
@github-actions github-actions bot added the binding/python Issues for the Python package label Oct 21, 2023
@rtyler rtyler enabled auto-merge (rebase) October 26, 2023 15:56
@ion-elgreco
Copy link
Collaborator Author

@rtyler or @wjones127 could you take a look at the last changes if it's ok now?

Copy link
Collaborator

@wjones127 wjones127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish there was a description somewhere of how this operation is meant to be used.

@rtyler rtyler merged commit eec0349 into delta-io:main Oct 30, 2023
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package binding/rust Issues for the Rust crate rust
Projects
None yet
Development

Successfully merging this pull request may close these issues.

VACUUM doesn't create a commit in transaction log
3 participants