[QuorumStore] fix insert/expire issue

[gelash]

Fix the issue of not holding the lock while checking expiration and inserting a new entry (that can extend expiration but could end up truncating).

Simple test that replicates the panic without the fix.

More tests (more coverage, concurrent, proptests, injecting pause) coming separately.

[github-actions]

✅ Forge suite compat success on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 6cec025cc0e79e4725e032062a00cde9847dc89b

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 6cec025cc0e79e4725e032062a00cde9847dc89b (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8279 TPS, 4616 ms latency, 6100 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 6cec025cc0e79e4725e032062a00cde9847dc89b
compatibility::simple-validator-upgrade::single-validator-upgrade : 4730 TPS, 8482 ms latency, 11000 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 6cec025cc0e79e4725e032062a00cde9847dc89b
compatibility::simple-validator-upgrade::half-validator-upgrade : 4982 TPS, 7967 ms latency, 10100 ms p99 latency,no expired txns
4. upgrading second batch to new version: 6cec025cc0e79e4725e032062a00cde9847dc89b
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7139 TPS, 5332 ms latency, 9600 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 6cec025cc0e79e4725e032062a00cde9847dc89b passed
Test Ok

• Grafana dashboard
• Humio Logs
• Pyroscope Profiling
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite land_blocking success on 6cec025cc0e79e4725e032062a00cde9847dc89b

performance benchmark with full nodes : 6085 TPS, 6512 ms latency, 9300 ms p99 latency,(!) expired 140 out of 2598500 txns
Test Ok

• Grafana dashboard
• Humio Logs
• Pyroscope Profiling
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 6cec025cc0e79e4725e032062a00cde9847dc89b

Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 6cec025cc0e79e4725e032062a00cde9847dc89b (PR)
Upgrade the nodes to version: 6cec025cc0e79e4725e032062a00cde9847dc89b
framework_upgrade::framework-upgrade::full-framework-upgrade : 6486 TPS, 5903 ms latency, 8800 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 6cec025cc0e79e4725e032062a00cde9847dc89b passed
Test Ok

• Grafana dashboard
• Humio Logs
• Pyroscope Profiling
• Test runner output
• Test run is land-blocking