[consensus] verify proposals in parallel, add counters #12209
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
⏱️ 5h 24m total CI duration on this PR
🚨 1 job on the last run was significantly faster/slower than expected
|
bchocho
requested review from
zekun000,
sasha8 and
ibalajiarun
as code owners
zekun000
approved these changes
Feb 23, 2024
| proof_with_status | ||
| .proofs | ||
| .par_iter() | ||
| .try_for_each(|proof| proof.verify(validator))?; |
There was a problem hiding this comment.
we may want to chunk them instead of every single one, spawn task is non trivial as well
There was a problem hiding this comment.
+1, rayon spawn is very expensive - especially done on small item. Use a min size with the parallel item.
ibalajiarun
reviewed
Feb 23, 2024
| @@ -85,17 +85,24 @@ impl UnverifiedEvent { | |||
| max_num_batches: usize, | |||
| max_batch_expiry_gap_usecs: u64, | |||
| ) -> Result<VerifiedEvent, VerifyError> { | |||
| let start_time = Instant::now(); | |||
There was a problem hiding this comment.
we can use monitor! macro instead?
There was a problem hiding this comment.
Not sure we want to create a separate one for each event. Isn't it a bit cleaner to have a separate HistogramVec?
bchocho
added a commit
that referenced
this pull request
Feb 24, 2024
ibalajiarun
approved these changes
Feb 24, 2024
sitalkedia
approved these changes
Feb 24, 2024
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
✅ Forge suite
|
✅ Forge suite
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
For blocks with many batches, using par_iter significantly improves proposal processing time.
The counters were used to find this issue in prep for previewnet.
Test Plan
Observe counters changed in forge. Existing forge tests.