[State Sync] Add sync_for_duration() mode.

[JoshLind]

Description

This PR adds basic sync_for_duration() support to state sync, allowing callers to block while state sync synchronizes the node for the specified duration. This will be required by consensus observer (CO), to implement a smarter fallback mechanism on failures to stay up-to-date. Integration with CO will happen in the next PR.

The PR offers the following commits:

1. Add a sync_for_duration() function to the consensus notifier.
2. Update state sync to handle the sync_for_duration() request.
3. Make sync_for_duration() return the latest synced ledger info on handover.

Testing Plan

New and existing test infrastructure. More will be added upon integration with CO.

[trunk-io]

⏱️ 53m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / test-target-determinator	9m	🟩 🟩
test-target-determinator	9m	🟩 🟩
check	8m	🟩 🟩
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
rust-cargo-deny	4m	⬜ 🟩 🟩
fetch-last-released-docker-image-tag	3m	🟩 🟩
check-dynamic-deps	3m	🟩 🟩 🟩
rust-move-tests	2m	🟩
general-lints	2m	🟩 🟩 🟩
rust-move-tests	2m	🟩
semgrep/ci	1m	🟩 🟩 🟩
file_change_determinator	32s	🟩 🟩 🟩
rust-move-tests	26s	⬜
file_change_determinator	22s	🟩 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / single-node-performance	10s	21m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on 1623844699bed725701d890949c6a67536ab0f32

two traffics test: inner traffic : committed: 13324.17 txn/s, latency: 2985.02 ms, (p50: 2700 ms, p70: 3000, p90: 3100 ms, p99: 12300 ms), latency samples: 5066160
two traffics test : committed: 100.02 txn/s, latency: 2606.52 ms, (p50: 2400 ms, p70: 2600, p90: 2700 ms, p99: 11200 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.251, avg: 0.222", "QsPosToProposal: max: 0.330, avg: 0.264", "ConsensusProposalToOrdered: max: 0.319, avg: 0.298", "ConsensusOrderedToCommit: max: 0.484, avg: 0.460", "ConsensusProposalToCommit: max: 0.782, avg: 0.759"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.16s no progress at version 966552 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.39s no progress at version 966550 (avg 8.30s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 7eeba4cd15892717741a614add1afde004c7855f ==> 1623844699bed725701d890949c6a67536ab0f32

Compatibility test results for 7eeba4cd15892717741a614add1afde004c7855f ==> 1623844699bed725701d890949c6a67536ab0f32 (PR)
Upgrade the nodes to version: 1623844699bed725701d890949c6a67536ab0f32
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1255.48 txn/s, submitted: 1259.77 txn/s, failed submission: 4.30 txn/s, expired: 4.30 txn/s, latency: 2562.05 ms, (p50: 2400 ms, p70: 2700, p90: 4400 ms, p99: 6000 ms), latency samples: 111020
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1188.66 txn/s, submitted: 1191.75 txn/s, failed submission: 3.09 txn/s, expired: 3.09 txn/s, latency: 2514.51 ms, (p50: 2400 ms, p70: 2700, p90: 4000 ms, p99: 5700 ms), latency samples: 107560
5. check swarm health
Compatibility test for 7eeba4cd15892717741a614add1afde004c7855f ==> 1623844699bed725701d890949c6a67536ab0f32 passed
Upgrade the remaining nodes to version: 1623844699bed725701d890949c6a67536ab0f32
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1089.19 txn/s, submitted: 1091.02 txn/s, failed submission: 1.83 txn/s, expired: 1.83 txn/s, latency: 3062.08 ms, (p50: 2400 ms, p70: 3100, p90: 5900 ms, p99: 7900 ms), latency samples: 95420
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 7eeba4cd15892717741a614add1afde004c7855f ==> 1623844699bed725701d890949c6a67536ab0f32

Compatibility test results for 7eeba4cd15892717741a614add1afde004c7855f ==> 1623844699bed725701d890949c6a67536ab0f32 (PR)
1. Check liveness of validators at old version: 7eeba4cd15892717741a614add1afde004c7855f
compatibility::simple-validator-upgrade::liveness-check : committed: 13064.83 txn/s, latency: 2562.11 ms, (p50: 1800 ms, p70: 2100, p90: 2700 ms, p99: 24100 ms), latency samples: 510820
2. Upgrading first Validator to new version: 1623844699bed725701d890949c6a67536ab0f32
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6097.97 txn/s, latency: 4752.50 ms, (p50: 5400 ms, p70: 6100, p90: 6500 ms, p99: 6600 ms), latency samples: 113580
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6293.63 txn/s, latency: 5133.07 ms, (p50: 5700 ms, p70: 5800, p90: 6200 ms, p99: 6500 ms), latency samples: 208280
3. Upgrading rest of first batch to new version: 1623844699bed725701d890949c6a67536ab0f32
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7030.05 txn/s, latency: 3997.72 ms, (p50: 4600 ms, p70: 4800, p90: 5000 ms, p99: 5100 ms), latency samples: 129200
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6638.95 txn/s, latency: 4852.14 ms, (p50: 4800 ms, p70: 5000, p90: 7200 ms, p99: 7600 ms), latency samples: 237440
4. upgrading second batch to new version: 1623844699bed725701d890949c6a67536ab0f32
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10403.36 txn/s, latency: 2649.68 ms, (p50: 2800 ms, p70: 3100, p90: 3400 ms, p99: 3600 ms), latency samples: 184460
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 9490.06 txn/s, latency: 3282.56 ms, (p50: 2900 ms, p70: 3200, p90: 6200 ms, p99: 8300 ms), latency samples: 352780
5. check swarm health
Compatibility test for 7eeba4cd15892717741a614add1afde004c7855f ==> 1623844699bed725701d890949c6a67536ab0f32 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking