[State Sync] Update synced version and synced epoch metrics.

[JoshLind]

Description

This PR updates the synced version and synced epoch metrics. We now track these metrics in two ways: (i) by reading the values directly from storage; and (ii) by incrementally summing up all processed transactions and epochs.

The first metric will be used as the "official" metric, and the second as a sanity check to ensure that there are no gaps in transaction and/or epoch processing (e.g., consensus dropping notifications meant for state sync).

Testing Plan

Existing test infrastructure.

[trunk-io]

⏱️ 43m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
test-target-determinator	13m	🟩 🟩 🟩
rust-cargo-deny	9m	🟩 🟩 🟩 🟩 🟩
check-dynamic-deps	4m	🟩 🟩 🟩 🟩 🟩
general-lints	2m	🟩 🟩 🟩 🟩 🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩 🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	1m	🟩
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩
file_change_determinator	33s	🟩 🟩 🟩
permission-check	21s	🟩 🟩 🟩 🟩 🟩
permission-check	17s	🟩 🟩 🟩 🟩 🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[msmouse]

Thanks boss

[github-actions]

✅ Forge suite realistic_env_max_load success on ab458fd347943ddd030ae8a20c5bd8ec92da3b9a

two traffics test: inner traffic : committed: 14221.81 txn/s, latency: 2790.59 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3200 ms), latency samples: 5407440
two traffics test : committed: 99.91 txn/s, latency: 1615.38 ms, (p50: 1400 ms, p70: 1500, p90: 1600 ms, p99: 7800 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.001, avg: 1.547", "ConsensusProposalToOrdered: max: 0.313, avg: 0.295", "ConsensusOrderedToCommit: max: 0.384, avg: 0.372", "ConsensusProposalToCommit: max: 0.677, avg: 0.667"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.91s no progress at version 2795907 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.48s no progress at version 2795905 (avg 8.48s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 23a2cd8419c1c283a4aa52c94e917621d39039aa ==> ab458fd347943ddd030ae8a20c5bd8ec92da3b9a

Compatibility test results for 23a2cd8419c1c283a4aa52c94e917621d39039aa ==> ab458fd347943ddd030ae8a20c5bd8ec92da3b9a (PR)
Upgrade the nodes to version: ab458fd347943ddd030ae8a20c5bd8ec92da3b9a
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1185.12 txn/s, submitted: 1188.01 txn/s, failed submission: 2.89 txn/s, expired: 2.89 txn/s, latency: 2585.77 ms, (p50: 2400 ms, p70: 2700, p90: 3900 ms, p99: 5100 ms), latency samples: 106700
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1019.45 txn/s, submitted: 1022.29 txn/s, failed submission: 2.84 txn/s, expired: 2.84 txn/s, latency: 2870.73 ms, (p50: 2400 ms, p70: 3000, p90: 5400 ms, p99: 6900 ms), latency samples: 93340
5. check swarm health
Compatibility test for 23a2cd8419c1c283a4aa52c94e917621d39039aa ==> ab458fd347943ddd030ae8a20c5bd8ec92da3b9a passed
Upgrade the remaining nodes to version: ab458fd347943ddd030ae8a20c5bd8ec92da3b9a
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1200.67 txn/s, submitted: 1202.87 txn/s, failed submission: 2.20 txn/s, expired: 2.20 txn/s, latency: 2498.59 ms, (p50: 2400 ms, p70: 2700, p90: 3800 ms, p99: 6200 ms), latency samples: 109100
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 23a2cd8419c1c283a4aa52c94e917621d39039aa ==> ab458fd347943ddd030ae8a20c5bd8ec92da3b9a

Compatibility test results for 23a2cd8419c1c283a4aa52c94e917621d39039aa ==> ab458fd347943ddd030ae8a20c5bd8ec92da3b9a (PR)
1. Check liveness of validators at old version: 23a2cd8419c1c283a4aa52c94e917621d39039aa
compatibility::simple-validator-upgrade::liveness-check : committed: 14940.32 txn/s, latency: 2249.42 ms, (p50: 2000 ms, p70: 2100, p90: 3100 ms, p99: 6000 ms), latency samples: 488860
2. Upgrading first Validator to new version: ab458fd347943ddd030ae8a20c5bd8ec92da3b9a
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 5856.11 txn/s, latency: 4794.89 ms, (p50: 5400 ms, p70: 5700, p90: 6000 ms, p99: 6400 ms), latency samples: 108220
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6144.22 txn/s, latency: 5278.87 ms, (p50: 5700 ms, p70: 5800, p90: 6900 ms, p99: 7100 ms), latency samples: 204580
3. Upgrading rest of first batch to new version: ab458fd347943ddd030ae8a20c5bd8ec92da3b9a
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5328.77 txn/s, latency: 5424.58 ms, (p50: 6200 ms, p70: 6500, p90: 6700 ms, p99: 6700 ms), latency samples: 103660
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5364.45 txn/s, latency: 6092.79 ms, (p50: 6700 ms, p70: 6900, p90: 7100 ms, p99: 7400 ms), latency samples: 192500
4. upgrading second batch to new version: ab458fd347943ddd030ae8a20c5bd8ec92da3b9a
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8691.81 txn/s, latency: 3259.14 ms, (p50: 3600 ms, p70: 3800, p90: 4200 ms, p99: 4400 ms), latency samples: 156420
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8810.27 txn/s, latency: 3581.81 ms, (p50: 3700 ms, p70: 4200, p90: 4900 ms, p99: 5200 ms), latency samples: 287060
5. check swarm health
Compatibility test for 23a2cd8419c1c283a4aa52c94e917621d39039aa ==> ab458fd347943ddd030ae8a20c5bd8ec92da3b9a passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking