This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
PVF host prechecking support v2 #4123
Merged
+310
−82
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
slumber
changed the base branch from
master
to
slumber-move-artifact-status-into-memory
slumber
added
B0-silent
pepyakin
reviewed
Oct 22, 2021
|
/cla run-cla-check |
|
Queueing command execution: run-cla-check |
|
@pepyakin Command execution has finished. |
Base automatically changed from
slumber-move-artifact-status-into-memory
to
master
October 22, 2021 16:38
|
/cla run-cla-check |
|
Queueing command execution: run-cla-check |
|
@azerella Command execution has finished. |
pepyakin
reviewed
Oct 27, 2021
pepyakin
approved these changes
Oct 27, 2021
Closed
|
checks failing |
drahnr
pushed a commit
that referenced
this pull request
Nov 15, 2021
* pvf host: store only compiled artifacts on disk * Correctly handle failed artifacts * Serialize result of PVF preparation uniquely * Set the artifact state depending on the result * Return the result of PVF preparation directly * Move PrepareError to the error module * Update doc comments * Update misleading comment * pvf host: turn off parallel compilation * pvf host: implement precheck requests * Fix warnings * Unnecessary clone * Add a note about timed out outcome * Revert the pool outcome handling behavior * Move the prepare result type into error mod * Test prepare done * fmt * Add an explanation to wasmtime config * Split pvf host test * Add precheck to dictionary Co-authored-by: Sergei Shulepov <[email protected]>
23 tasks
pepyakin
added a commit
that referenced
this pull request
Dec 14, 2021
We wanted to change niceness to accomodate the fact that some of the preparation tasks are low priority. For example, when a node sees that there is a new para was onboarded the node may start preparing right away. Since all other activities are more important, such as network I/O or validation of the backed candidates and preparation of the immediatelly needed PVFs. However, it turned out that this approach does not work: generally non-root processes can only decrease niceness and they cannot increase it to the previous value, as was assumed by the code. Apart from that, #4123 assumes all PVFs are prepared in the same way. Specifically, that if a PVF preparation failed before, then PVF pre-checking will also report that it was failed, even though it could happen that preparation failed due to being low-priority. In order to avoid such cases, we decided to simplify the whole preparation model. Preparation under low priority does not work well with that. Closes #4520
pepyakin
added a commit
that referenced
this pull request
Dec 14, 2021
We wanted to change niceness to accomodate the fact that some of the preparation tasks are low priority. For example, when a node sees that there is a new para was onboarded the node may start preparing right away. Since all other activities are more important, such as network I/O or validation of the backed candidates and preparation of the immediatelly needed PVFs. However, it turned out that this approach does not work: generally non-root processes can only decrease niceness and they cannot increase it to the previous value, as was assumed by the code. Apart from that, #4123 assumes all PVFs are prepared in the same way. Specifically, that if a PVF preparation failed before, then PVF pre-checking will also report that it was failed, even though it could happen that preparation failed due to being low-priority. In order to avoid such cases, we decided to simplify the whole preparation model. Preparation under low priority does not work well with that. Closes #4520
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make every prepare request single-threaded
Resolves #3857
Based on #3907
NOTE(@pepyakin): this PR disables parallel compilation for preparation. I conducted a couple of tests on different machines. It takes ~0.7-0.8ms to compile the code with
parallel_compilationon and around 3.5s without it.