update randomness_stall_recovery to not be blocked by a db bug

[zjma]

No description provided.

[trunk-io]

⏱️ 2h total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / test-target-determinator	22m	🟩 🟩 🟩 🟩 🟩
test-target-determinator	18m	🟩 ⬜ 🟩 🟩 🟩
check	16m	🟩 ⬜ 🟩 🟩 🟩
rust-cargo-deny	10m	⬜ 🟩 🟩 🟩 🟩 (+1 more)
check-dynamic-deps	9m	🟩 🟩 🟩 🟩 🟩 (+2 more)
fetch-last-released-docker-image-tag	9m	🟩 ⬜ 🟩 🟩 🟩
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
semgrep/ci	3m	🟩 🟩 🟩 🟩 🟩 (+2 more)
general-lints	3m	🟩 🟩 🟩 🟩 🟩 (+1 more)
rust-doc-tests	2m	🟥
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	1m	🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / single-node-performance	10s	17m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on dbb44ea10a393a07293e42fb8b50e59b2c65cb3d

two traffics test: inner traffic : committed: 14301.31 txn/s, latency: 2774.90 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5437680
two traffics test : committed: 99.93 txn/s, latency: 1523.73 ms, (p50: 1500 ms, p70: 1600, p90: 1700 ms, p99: 1900 ms), latency samples: 1680
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.009, avg: 1.564", "ConsensusProposalToOrdered: max: 0.323, avg: 0.294", "ConsensusOrderedToCommit: max: 0.390, avg: 0.381", "ConsensusProposalToCommit: max: 0.684, avg: 0.675"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.81s no progress at version 2542352 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.35s no progress at version 2542350 (avg 8.35s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> dbb44ea10a393a07293e42fb8b50e59b2c65cb3d

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> dbb44ea10a393a07293e42fb8b50e59b2c65cb3d (PR)
Upgrade the nodes to version: dbb44ea10a393a07293e42fb8b50e59b2c65cb3d
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1089.18 txn/s, submitted: 1092.39 txn/s, failed submission: 3.21 txn/s, expired: 3.21 txn/s, latency: 2925.78 ms, (p50: 2400 ms, p70: 3300, p90: 5200 ms, p99: 6600 ms), latency samples: 95000
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1187.97 txn/s, submitted: 1190.75 txn/s, failed submission: 2.77 txn/s, expired: 2.77 txn/s, latency: 2783.95 ms, (p50: 2100 ms, p70: 3000, p90: 4900 ms, p99: 6800 ms), latency samples: 94240
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> dbb44ea10a393a07293e42fb8b50e59b2c65cb3d passed
Upgrade the remaining nodes to version: dbb44ea10a393a07293e42fb8b50e59b2c65cb3d
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1195.52 txn/s, submitted: 1198.83 txn/s, failed submission: 3.30 txn/s, expired: 3.30 txn/s, latency: 2475.22 ms, (p50: 2400 ms, p70: 2700, p90: 3500 ms, p99: 5700 ms), latency samples: 108620
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> dbb44ea10a393a07293e42fb8b50e59b2c65cb3d

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> dbb44ea10a393a07293e42fb8b50e59b2c65cb3d (PR)
1. Check liveness of validators at old version: b29f09f57e898d8d211c8bc3e303f6e50bba2266
compatibility::simple-validator-upgrade::liveness-check : committed: 14432.12 txn/s, latency: 2033.87 ms, (p50: 1900 ms, p70: 2100, p90: 2400 ms, p99: 3100 ms), latency samples: 549520
2. Upgrading first Validator to new version: dbb44ea10a393a07293e42fb8b50e59b2c65cb3d
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 5557.69 txn/s, latency: 4909.01 ms, (p50: 5600 ms, p70: 5700, p90: 5800 ms, p99: 5900 ms), latency samples: 112380
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6021.12 txn/s, latency: 5287.49 ms, (p50: 5900 ms, p70: 6000, p90: 7100 ms, p99: 7600 ms), latency samples: 207180
3. Upgrading rest of first batch to new version: dbb44ea10a393a07293e42fb8b50e59b2c65cb3d
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5749.66 txn/s, latency: 4892.41 ms, (p50: 5300 ms, p70: 6000, p90: 6200 ms, p99: 6400 ms), latency samples: 106320
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5830.12 txn/s, latency: 5543.59 ms, (p50: 6100 ms, p70: 6300, p90: 6700 ms, p99: 7300 ms), latency samples: 194900
4. upgrading second batch to new version: dbb44ea10a393a07293e42fb8b50e59b2c65cb3d
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8073.99 txn/s, latency: 3396.54 ms, (p50: 3500 ms, p70: 4200, p90: 5100 ms, p99: 5300 ms), latency samples: 137620
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 7861.02 txn/s, latency: 4003.04 ms, (p50: 3500 ms, p70: 4900, p90: 5900 ms, p99: 7600 ms), latency samples: 260040
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> dbb44ea10a393a07293e42fb8b50e59b2c65cb3d passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking