This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
Increase preparation timeout #4270
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pepyakin
added
A0-please_review
pepyakin
added
B1-releasenotes
and removed
B0-silent
ordian
approved these changes
Nov 15, 2021
drahnr
pushed a commit
that referenced
this pull request
Nov 15, 2021
* Increase preparation-timeout to 60 seconds * Adapt `pvf_preparation_time` metric to the new value
23 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Lately we've been observing preparation timeouts. Whereas in local testing, compilation of a hefty PVF should take <1s for multi-threaded compilation (and ≈3s for single-threaded), in the wild we were observing compilation speed of more like 5 seconds and more occasionally bumping into the 10s cutoff.
This timeout affects the maximum response time for backing and approvals work. However, it is amortized, and once compiled it will be cached for time the artifact is being used. 10s cutoff was chosen rather arbitrary. It is already more than backing, which has way less than 10s, so it will be missed anyway. Missing the deadline once in a while for approvals does not seem bad either.
At the same time, increasing the timeout does not seem to be very bad at this moment. Exploiting JIT bombs does not seem to be likely. Doing so will require big investment and the effects of the attack are questionable.
Priority is set to high, because we need to upgrade the nodes ASAP to minimize the time when nodes have different opinion on whether the code timed out or not.