Cap max RetryableAction wait time/timeout. #74940
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RetryableAction uses randomized and exponential back off. If unlucky, the randomization would cause a series of very short waits, which would double the bound every time, risking a subsequent very long wait. Now randomize between [bound/2, bound[. Closes elastic#70996
henningandersen
added
>bug
:Distributed Indexing/CRUD
elasticmachine
added
Team:Distributed (Obsolete)
|
Pinging @elastic/ml-core (Team:ML) |
|
Pinging @elastic/es-distributed (Team:Distributed) |
davidkyle
approved these changes
Jul 6, 2021
...n/ml/src/main/java/org/elasticsearch/xpack/ml/utils/persistence/ResultsPersisterService.java
Outdated
Show resolved
Hide resolved
benwtrent
approved these changes
Jul 12, 2021
|
Will this PR fix timeouts like this one? https://gradle-enterprise.elastic.co/s/qvkqnuh5ocffw |
I am afraid not, the symptoms of the failing |
…s/persistence/ResultsPersisterService.java Co-authored-by: David Kyle <[email protected]>
henningandersen
added
the
auto-backport
💔 Backport failed
To backport manually run: |
henningandersen
removed
the
auto-backport
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Aug 5, 2021
RetryableAction uses randomized and exponential back off. If unlucky, the randomization would cause a series of very short waits, which would double the bound every time, risking a subsequent very long wait. Now randomize between [bound/2, bound[. Closes elastic#70996
henningandersen
added a commit
that referenced
this pull request
Aug 5, 2021
RetryableAction uses randomized and exponential back off. If unlucky, the randomization would cause a series of very short waits, which would double the bound every time, risking a subsequent very long wait. Now randomize between [bound/2, bound[. Closes #70996
henningandersen
added a commit
that referenced
this pull request
Aug 5, 2021
RetryableAction uses randomized and exponential back off. If unlucky, the randomization would cause a series of very short waits, which would double the bound every time, risking a subsequent very long wait. Now randomize between [bound/2, bound[. Closes #70996
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
RetryableAction uses randomized and exponential back off. If unlucky,
the randomization would cause a series of very short waits, which would
double the bound every time, risking a subsequent very long wait. Now
randomize between [bound/2, bound[.
Closes #70996
This fixes
testAckedIndexingbecause it ran into a final very long wait time for one of the retries.I am not necessarily fixed on this solution, though it seems like a step in the right direction. An
alternative could be to cap the final wait time to meet the timeout, but it does add more deterministic/less random
behavior and the solution here should be good enough.