Re-apply qs backpressure increase with buffer latency increase #13961
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As Quorum Store batches are bucketed, and we are looking to increase block limits, now is the time to reduce Quorum Store backpressure. We now allow 36K transactions outstanding. At 12K TPS, this is approximately 3 seconds worth of batches. For forge tests, a lot of the queuing shifts from mempool to POS-to-Proposal, so the limits need to be adjusted accordingly.
vusirikala
added
the
CICD:build-images
vusirikala
approved these changes
Jul 10, 2024
There was a problem hiding this comment.
As graceful overload test failed last time, I ran graceful overload test here. It seems to be working.
https://github.com/aptos-labs/aptos-core/actions/runs/9865912053
sitalkedia
reviewed
Jul 10, 2024
| @@ -85,8 +85,8 @@ impl Default for MempoolConfig { | |||
| system_transaction_timeout_secs: 600, | |||
| system_transaction_gc_interval_ms: 60_000, | |||
| broadcast_buckets: DEFAULT_BUCKETS.to_vec(), | |||
| eager_expire_threshold_ms: Some(10_000), | |||
| eager_expire_time_ms: 3_000, | |||
| eager_expire_threshold_ms: Some(15_000), | |||
There was a problem hiding this comment.
Can you explain the downside of increasing this?
There was a problem hiding this comment.
new constants make the logic like this:
if there is any transaction in the mempool (that never entered parking lot) that was there for longer than eager_expire_threshold_ms (which is increased from 10s -> 15s here) - that means we have a significant backlog.
if that is the case, we pull into batches only transactions that have at least eager_expire_time_ms time left (increased from 3s to 6s)
We've increased both, as with reduced QS backpressure, we see it taking more time from batch creation to batch being included in the block.
So the side-consequence of this change is that - if that during backlog, if you use expiration <6s, your transactions will be ignored (previously <3s).
There was a problem hiding this comment.
Sounds good to me. Slightly orthogonal but do we really need this logic after @vusirikala 's change to exclude expired transaction in QS backpressure calculation in https://github.com/aptos-labs/aptos-core/pull/13850/files. cc - @bchocho
There was a problem hiding this comment.
creating useless batches, and then requiring everyone to fetch them, just in order to throw them out - probably has negative effect on our overall throughput.
There was a problem hiding this comment.
@sitalkedia This is the run for graceful overload test with my changes and increasing QS backpressure limits without changing the eager_expire_threshold_ms config.
#13964 (comment)
sitalkedia
approved these changes
Jul 10, 2024
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
✅ Forge suite
|
✅ Forge suite
|
✅ Forge suite
|
zi0Black
pushed a commit
that referenced
this pull request
Jul 12, 2024
* [quorum store] reduce backpressure significantly for more TPS (#13558) As Quorum Store batches are bucketed, and we are looking to increase block limits, now is the time to reduce Quorum Store backpressure. We now allow 36K transactions outstanding. At 12K TPS, this is approximately 3 seconds worth of batches. For forge tests, a lot of the queuing shifts from mempool to POS-to-Proposal, so the limits need to be adjusted accordingly. * increase buffer for expiration in batch creation * adding buffers on inner traffic as well --------- Co-authored-by: Brian (Sunghoon) Cho <[email protected]>
igor-aptos
added a commit
that referenced
this pull request
Jul 12, 2024
* [quorum store] reduce backpressure significantly for more TPS (#13558) As Quorum Store batches are bucketed, and we are looking to increase block limits, now is the time to reduce Quorum Store backpressure. We now allow 36K transactions outstanding. At 12K TPS, this is approximately 3 seconds worth of batches. For forge tests, a lot of the queuing shifts from mempool to POS-to-Proposal, so the limits need to be adjusted accordingly. * increase buffer for expiration in batch creation * adding buffers on inner traffic as well --------- Co-authored-by: Brian (Sunghoon) Cho <[email protected]>
igor-aptos
added a commit
that referenced
this pull request
Jul 15, 2024
… (#13997) * [quorum store] reduce backpressure significantly for more TPS (#13558) As Quorum Store batches are bucketed, and we are looking to increase block limits, now is the time to reduce Quorum Store backpressure. We now allow 36K transactions outstanding. At 12K TPS, this is approximately 3 seconds worth of batches. For forge tests, a lot of the queuing shifts from mempool to POS-to-Proposal, so the limits need to be adjusted accordingly. * increase buffer for expiration in batch creation * adding buffers on inner traffic as well ---------
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
As Quorum Store batches are bucketed, and we are looking to increase block limits, now is the time to reduce Quorum Store backpressure.
We now allow 36K transactions outstanding. At 12K TPS, this is approximately 3 seconds worth of batches.
For forge tests, a lot of the queuing shifts from mempool to POS-to-Proposal, so the limits need to be adjusted accordingly.
Additionally, this increases time batches can take to be included in the block, so increasing eager expirations
Type of Change
Which Components or Systems Does This Change Impact?
How Has This Been Tested?
Key Areas to Review
Checklist