[execution window] further optimizations #15053

bchocho · 2024-10-23T05:26:28Z

Add window and block retrieval to block_tree, block_store and persistent_liveness_storage. Validator should be able to retrieve blocks and recover with blocks up to window start of the next block.
Execute the block window. Doesn't have any optimizations.
Filter committed transactions. Up until round-1 in prepare phase, and round-1 in execution phase. This requires using blocking txn provider to quickly provide shuffled txns in the execution phase.
incremental update of duplicate counts

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-10-23T05:26:31Z

⏱️ 2h 3m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
test-target-determinator	28m	🟩 🟩 🟩 🟩 🟩 (+1 more)
rust-images / rust-all	14m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
rust-cargo-deny	9m	🟩 🟩 🟩 🟩 🟩
check-dynamic-deps	5m	🟩 🟩 🟩 🟩 🟩
general-lints	2m	🟩 🟩 🟩 🟩 🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩 🟩
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+1 more)
file_change_determinator	57s	🟩 🟩 🟩 🟩 🟩
rust-images / rust-all	52s	🟥

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

par iter for filter add a max txns slighly bump up pipeline pending latency

Add junit support for forge This will allow us to onboard forge onto trunk flaky test detection Also make suite choice more clean Test Plan: running on PR

This reverts commit e05f5dc.

github-actions · 2024-11-15T18:33:01Z

✅ Forge suite `realistic_env_max_load` success on `f1ca5e9830f9fe8ddc25ad13951cc2613755af67`

two traffics test: inner traffic : committed: 14276.64 txn/s, latency: 2784.71 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 5400 ms), latency samples: 5428360
two traffics test : committed: 99.97 txn/s, latency: 2631.22 ms, (p50: 2200 ms, p70: 2300, p90: 3600 ms, p99: 8400 ms), latency samples: 1780
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.215, avg: 0.804", "ConsensusProposalToOrdered: max: 0.319, avg: 0.294", "ConsensusOrderedToCommit: max: 0.405, avg: 0.389", "ConsensusProposalToCommit: max: 0.695, avg: 0.682"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.05s no progress at version 2491602 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.33s no progress at version 2491600 (avg 8.33s) [limit 15].
Test Ok

bchocho added the CICD:run-forge-e2e-perf Run the e2e perf forge only label Oct 23, 2024

bchocho changed the base branch from main to brian/exec-window-exec-optimize-2 October 23, 2024 06:01