Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add is_non_acquirable field to the dynamodb lock #429

Merged
merged 2 commits into from
Sep 17, 2021

Conversation

mosyp
Copy link
Contributor

@mosyp mosyp commented Sep 16, 2021

Description

If there's any outside of delta-rs writes to the table that are done with s3 backend and dynamodb lock, we want the mechanism to immediately stop delta-rs writes if required.

The example could be the dropping of a table. For example you acquire a lock and then try drop (or drop/create etc., something that modifies the delta log completely) a table. The dangerous part here is that the concurrent delta workers will still perform the write whenever the lock is available, because we effectively lock the rename operation.

To prevent that, this PR introduces a is_non_acquirable that will cause delta-rs writer to fail immediately (or to be precise return the specific error) when the remote lock has this field set. For this to work, the outside of delta-rs writer, which performs the dangerous operation, has to set this field by itself once the lock is acquired.

xianwill
xianwill previously approved these changes Sep 16, 2021
houqp
houqp previously approved these changes Sep 17, 2021
@mosyp mosyp dismissed stale reviews from houqp and xianwill via 3d88287 September 17, 2021 10:08
@mosyp mosyp enabled auto-merge (squash) September 17, 2021 10:09
@mosyp mosyp merged commit 484c374 into delta-io:main Sep 17, 2021
@mosyp mosyp deleted the add_is_non_acquirable_field branch September 17, 2021 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants