Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configurable number of retries for transaction commit loop #1595

Closed
cmackenzie1 opened this issue Aug 18, 2023 · 0 comments · Fixed by #1596
Closed

Configurable number of retries for transaction commit loop #1595

cmackenzie1 opened this issue Aug 18, 2023 · 0 comments · Fixed by #1596
Labels
enhancement New feature or request

Comments

@cmackenzie1
Copy link
Contributor

cmackenzie1 commented Aug 18, 2023 •
edited
Loading

Description

Use Case

During testing, we found that as we scale out writers up, the # of conflicts can exceed the 15 attempts. This value should be configurable.

https://github.com/delta-io/delta-rs/blob/main/rust/src/operations/transaction/mod.rs#L167

Additionally, allow a small random backoff in retries to increase the fairness amongst writers

Related Issue(s)
@cmackenzie1 cmackenzie1 added the enhancement New feature or request label Aug 18, 2023
cmackenzie1 added a commit to cmackenzie1-contrib/delta-rs that referenced this issue Aug 21, 2023
@cmackenzie1
feat: allow configurable number of commit attempts
22fa5d6 
This adds `transaction::commit_with_retries` function where
the number of attempts can be specified. The default behavior for
`transaction::commit` remains 15.

Closes delta-io#1595.
@cmackenzie1 cmackenzie1 mentioned this issue Aug 21, 2023
Merged
@cmackenzie1 cmackenzie1 changed the title Configurable number of retries and backoff for transaction commit loop Configurable number of retries for transaction commit loop Aug 21, 2023
rtyler pushed a commit that referenced this issue Aug 28, 2023
@cmackenzie1 @rtyler
feat: allow configurable number of commit attempts
42e7123 
This adds `transaction::commit_with_retries` function where
the number of attempts can be specified. The default behavior for
`transaction::commit` remains 15.

Closes #1595.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: allow configurable number of commit attempts cmackenzie1-contrib/delta-rs
1 participant
@cmackenzie1