change(mempool): Re-verify mempool transactions after a chain fork, rather than re-downloading them all

[teor2345]

Motivation

Zebra drops all its mempool transactions when there is a chain fork. But it would be faster and put less load on the network if we re-verified them instead.

I discovered this bug while testing the getblocktemplate RPC.

Complex Code or Requirements

We need to make sure we clear the mempool, then re-verify all the stored or pending transactions.

Otherwise, we could break a consensus rule.

Solution

• Re-verify mempool transactions after a fork, rather than re-downloading them all
• Update tests for mempool transaction re-verification after forks
• Update README based on recent mitigations for some issues
  • also includes some mitigations from PR fix(rpc): Wait for 3 minutes to check Zebra is synced to the tip, rather than 2 #5840

Related bug fixes:

• Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs
  • In the previous design, we could drop the mempool Downloads without cancelling all the tasks

Related cleanups:

• Use a marker struct for cancelling mempool tasks
• Make mempool proptests easier to debug

This doesn't need to be behind a feature, because it's a bug fix for the production mempool as well.

Review

This is a routine bug fix.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

[codecov]

Codecov Report

Merging #5841 (3cfde8c) into main (f1e3fda) will increase coverage by 0.01%.
The diff coverage is 68.42%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5841      +/-   ##
==========================================
+ Coverage   78.70%   78.72%   +0.01%     
==========================================
  Files         308      308              
  Lines       38786    38836      +50     
==========================================
+ Hits        30528    30573      +45     
- Misses       8258     8263       +5     

[oxarbitrage]

Looks good, thanks for the fix.