[Consensus Observer] Parallelize message processing logic. #15030

JoshLind · 2024-10-21T13:06:04Z

Description

This PR parallelizes message verification steps in consensus observer. Specifically, payload signature verification and payload digest verification. This should save about ~20ms in consensus observer lag for realistic_env_max_load (e.g., see: #15002)

Testing Plan

Existing test infrastructure.

trunk-io · 2024-10-21T13:06:09Z

⏱️ 2h 23m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / single-node-performance	47m	🟩 🟩
test-target-determinator	26m	🟩 🟩 🟩 🟩 🟩 (+1 more)
rust-cargo-deny	10m	🟩 🟩 🟩 🟩 🟩 (+1 more)
check-dynamic-deps	10m	🟩 🟩 🟩 🟩 🟩 (+2 more)
execution-performance / test-target-determinator	9m	🟩 🟩
check	7m	🟩 🟩
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
fetch-last-released-docker-image-tag	4m	🟩 🟩
general-lints	3m	🟩 🟩 🟩 🟩 🟩 (+1 more)
semgrep/ci	3m	🟩 🟩 🟩 🟩 🟩 (+2 more)
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

zekun000 · 2024-10-21T20:52:00Z

consensus/src/consensus_observer/network/observer_message.rs

+        // Verify all the reconstructed batches (in parallel)
+        batches_and_transactions
+            .into_par_iter()
+            .try_for_each(|(batch_info, transactions)| verify_batch(&batch_info, transactions))


nit: we may want to have a min_len to avoid too many small tasks

Done boss 😄

zekun000 · 2024-10-21T20:52:06Z

consensus/src/consensus_observer/network/observer_message.rs

-                    proof_of_store.info(),
+        payload_proofs
+            .par_iter()
+            .try_for_each(|proof| proof.verify(validator_verifier, &proof_cache))


github-actions · 2024-10-22T22:34:38Z

✅ Forge suite `realistic_env_max_load` success on `b7722208f9637a00747f4af6cbbb351661b44f23`

two traffics test: inner traffic : committed: 13751.09 txn/s, latency: 2888.40 ms, (p50: 2700 ms, p70: 3000, p90: 3000 ms, p99: 3600 ms), latency samples: 5228580
two traffics test : committed: 99.91 txn/s, latency: 1659.58 ms, (p50: 1500 ms, p70: 1600, p90: 1700 ms, p99: 9400 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.268, avg: 0.229", "QsPosToProposal: max: 1.275, avg: 1.182", "ConsensusProposalToOrdered: max: 0.331, avg: 0.302", "ConsensusOrderedToCommit: max: 0.425, avg: 0.400", "ConsensusProposalToCommit: max: 0.725, avg: 0.702"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.33s no progress at version 33432 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.73s no progress at version 2464955 (avg 8.73s) [limit 15].
Test Ok

github-actions · 2024-10-22T22:37:09Z

✅ Forge suite `compat` success on `b29f09f57e898d8d211c8bc3e303f6e50bba2266` ==> `b7722208f9637a00747f4af6cbbb351661b44f23`

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> b7722208f9637a00747f4af6cbbb351661b44f23 (PR)
1. Check liveness of validators at old version: b29f09f57e898d8d211c8bc3e303f6e50bba2266
compatibility::simple-validator-upgrade::liveness-check : committed: 14141.93 txn/s, latency: 2362.83 ms, (p50: 1900 ms, p70: 2100, p90: 3100 ms, p99: 9400 ms), latency samples: 544340
2. Upgrading first Validator to new version: b7722208f9637a00747f4af6cbbb351661b44f23
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6187.78 txn/s, latency: 4488.72 ms, (p50: 5000 ms, p70: 5300, p90: 5800 ms, p99: 6000 ms), latency samples: 115660
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6151.14 txn/s, latency: 5171.64 ms, (p50: 5600 ms, p70: 5900, p90: 6300 ms, p99: 7400 ms), latency samples: 207500
3. Upgrading rest of first batch to new version: b7722208f9637a00747f4af6cbbb351661b44f23
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6704.45 txn/s, latency: 4262.35 ms, (p50: 4900 ms, p70: 5100, p90: 5300 ms, p99: 5400 ms), latency samples: 125080
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6175.22 txn/s, latency: 5156.09 ms, (p50: 5300 ms, p70: 5400, p90: 7100 ms, p99: 7400 ms), latency samples: 203160
4. upgrading second batch to new version: b7722208f9637a00747f4af6cbbb351661b44f23
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8870.56 txn/s, latency: 3172.00 ms, (p50: 3400 ms, p70: 3900, p90: 4100 ms, p99: 4500 ms), latency samples: 156960
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8386.31 txn/s, latency: 3781.12 ms, (p50: 3700 ms, p70: 4200, p90: 5100 ms, p99: 6200 ms), latency samples: 279560
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> b7722208f9637a00747f4af6cbbb351661b44f23 passed
Test Ok

github-actions · 2024-10-22T22:42:03Z