Fix "synced block height too far ahead of the tip: dropped downloaded block"

[teor2345]

Motivation

Sometimes Zebra downloads a block that is a long way ahead of the state tip. Currently, we're dropping those blocks, which can waste a lot of network bandwidth.

There are two cases we need to handle here:

1. the block is near the estimated tip, but the state tip is a long way from the estimated tip
2. the block is close to the state tip

Here is how we currently handle them:

1. drop the block, because verification would timeout anyway, and download it again in an hour or two
2. drop the block, and download it again a few minutes later

For the second case, we could pause extending tips instead.

Designs

Syncer:

• add a state pipeline limit to the syncer, and wait for it after waiting for the download & verify lookahead limit
  • await the current state tip height changing in a loop, until we are under the limit
• only return a "too far ahead" error when the block is a very long way ahead of the tip

If we also get these errors from the inbound downloads, we could increase its limit, or just turn down the log level. (We can't pause the inbound downloads, because they are gossiped.)

Related Work

• Finalizing blocks appears to be slow when checkpointing large blocks #3438
• Fix slowness when syncing near the tip #3375
• Zebra validation can be very slow on macOS, and cause very slow syncs #3042

[ftm1000]

Hey team! Please add your planning poker estimate with ZenHub @conradoplg @dconnolly @gustavovalverde @jvff @oxarbitrage @teor2345 @upbqdn

[teor2345]

This isn't causing us any issues right now.