[optqs] proposal status tracker metrics/logs

[ibalajiarun]

Description

Metrics and logs for proposal status tracker. Also, moves opt_qs_minimum_batch_age_usecs to config.

[trunk-io]

⏱️ 2h 7m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / single-node-performance	44m	🟩 🟩
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
execution-performance / test-target-determinator	9m	🟩 🟩
test-target-determinator	9m	🟩 🟩
rust-cargo-deny	8m	🟩 🟩 🟩 🟩
check	7m	🟩 🟩
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
check-dynamic-deps	5m	🟩 🟩 🟩 🟩
fetch-last-released-docker-image-tag	3m	🟩 🟩
general-lints	2m	🟩 🟩 🟩 🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩
file_change_determinator	42s	🟩 🟩 🟩 🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on 4778f22dc83cccfcffdff2fa2844a4e8afea6628

two traffics test: inner traffic : committed: 14225.17 txn/s, latency: 2797.93 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5408760
two traffics test : committed: 99.91 txn/s, latency: 1600.74 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 10300 ms), latency samples: 1720
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.066, avg: 1.633", "ConsensusProposalToOrdered: max: 0.324, avg: 0.297", "ConsensusOrderedToCommit: max: 0.364, avg: 0.354", "ConsensusProposalToCommit: max: 0.658, avg: 0.651"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.96s no progress at version 8845 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.61s no progress at version 2787422 (avg 8.61s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 58c5c53eaffe680eb2fc4cfbddf05a867fecadb9 ==> 4778f22dc83cccfcffdff2fa2844a4e8afea6628

Compatibility test results for 58c5c53eaffe680eb2fc4cfbddf05a867fecadb9 ==> 4778f22dc83cccfcffdff2fa2844a4e8afea6628 (PR)
Upgrade the nodes to version: 4778f22dc83cccfcffdff2fa2844a4e8afea6628
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1271.14 txn/s, submitted: 1275.01 txn/s, failed submission: 3.86 txn/s, expired: 3.86 txn/s, latency: 2381.20 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 6200 ms), latency samples: 111840
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1379.39 txn/s, submitted: 1382.08 txn/s, failed submission: 2.69 txn/s, expired: 2.69 txn/s, latency: 2191.03 ms, (p50: 2100 ms, p70: 2300, p90: 3600 ms, p99: 5100 ms), latency samples: 122900
5. check swarm health
Compatibility test for 58c5c53eaffe680eb2fc4cfbddf05a867fecadb9 ==> 4778f22dc83cccfcffdff2fa2844a4e8afea6628 passed
Upgrade the remaining nodes to version: 4778f22dc83cccfcffdff2fa2844a4e8afea6628
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1070.55 txn/s, submitted: 1072.82 txn/s, failed submission: 2.27 txn/s, expired: 2.27 txn/s, latency: 2826.19 ms, (p50: 2100 ms, p70: 2400, p90: 6000 ms, p99: 9900 ms), latency samples: 94400
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 58c5c53eaffe680eb2fc4cfbddf05a867fecadb9 ==> 4778f22dc83cccfcffdff2fa2844a4e8afea6628

Compatibility test results for 58c5c53eaffe680eb2fc4cfbddf05a867fecadb9 ==> 4778f22dc83cccfcffdff2fa2844a4e8afea6628 (PR)
1. Check liveness of validators at old version: 58c5c53eaffe680eb2fc4cfbddf05a867fecadb9
compatibility::simple-validator-upgrade::liveness-check : committed: 15393.33 txn/s, latency: 2181.46 ms, (p50: 1800 ms, p70: 2000, p90: 3000 ms, p99: 7200 ms), latency samples: 503520
2. Upgrading first Validator to new version: 4778f22dc83cccfcffdff2fa2844a4e8afea6628
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7987.05 txn/s, latency: 3542.38 ms, (p50: 3900 ms, p70: 4100, p90: 4300 ms, p99: 4300 ms), latency samples: 148060
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7785.27 txn/s, latency: 4089.27 ms, (p50: 4200 ms, p70: 4400, p90: 6000 ms, p99: 6200 ms), latency samples: 259220
3. Upgrading rest of first batch to new version: 4778f22dc83cccfcffdff2fa2844a4e8afea6628
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7059.91 txn/s, latency: 3815.60 ms, (p50: 4200 ms, p70: 4300, p90: 5000 ms, p99: 5100 ms), latency samples: 146440
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7685.31 txn/s, latency: 4103.82 ms, (p50: 4200 ms, p70: 4300, p90: 6500 ms, p99: 6800 ms), latency samples: 256000
4. upgrading second batch to new version: 4778f22dc83cccfcffdff2fa2844a4e8afea6628
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10755.58 txn/s, latency: 2590.07 ms, (p50: 2600 ms, p70: 2900, p90: 4000 ms, p99: 4200 ms), latency samples: 207020
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10803.59 txn/s, latency: 2913.13 ms, (p50: 2700 ms, p70: 2800, p90: 5500 ms, p99: 6900 ms), latency samples: 350000
5. check swarm health
Compatibility test for 58c5c53eaffe680eb2fc4cfbddf05a867fecadb9 ==> 4778f22dc83cccfcffdff2fa2844a4e8afea6628 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking