fix(sync): fix testnet syncer loop on large Orchard blocks

[teor2345]

Motivation

Testnet block 1860741 contains a 600 kB Orchard transaction, which can take a long time to verify.

This causes a syncer loop:

• the syncer requests that block again
• which causes a duplicate request error,
• which restarts the syncer loop,
• which re-verifies the same block.

Eventually, all the CPUs are busy verifying the same block, and Zebra does not make any sync progress.

Edit: actually I think there's just one verification going at one time. The CPU bug could be something different.

Solution

• Ignore the duplicate request error, and allow the block to verify
  • Make all download errors into enum variants
  • Check all syncer request errors to see if we want to ignore them

Related fixes:

• Check genesis errors to log info or warn messages

We might also need to increase the verification timeout on slower machines (PR #4156).

Review

This bug blocks syncing on testnet past block 1860741, so it is a high priority.

Reviewer Checklist

• Code implements Specs and Designs
• Tests for Expected Behaviour
• Tests for Errors

Follow Up Work

Better syncer errors

[oxarbitrage]

Looks good, should help.

[dconnolly]

👍