Full synchronization test times out in CI

[jvff]

Motivation

The full chain synchronization test is currently timing out in CI. We should find a way to speed up synchronization so that it doesn't time out.

[jvff]

One possible solution is to:

• increase the checkpoint height, so that those blocks get validated and finalized more quickly - change(verifier): Update Zebra's block hash checkpoints #4183

[teor2345]

Some other possible solutions are:

Quick tweaks, likely to help

• optimise Rust code at link time - change(rust): Use link-time optimisation in release builds #4184
• increase (or maybe decrease) the number of block download retries - fix(sync): Add an extra block retry, to speed up the initial sync #4185
• move database writes to a blocking thread, so they don't block network futures - 2. fix(perf): When writing blocks to disk, don't block other async tasks #4199
• move large database reads to a blocking thread, so they don't block network futures

Quick tweaks, likely to help, might cause bugs on slower networks

• decrease (or maybe increase) the handshake timeout - fix(perf): Decrease the peer handshake timeout to 3 seconds, to speed up the initial sync #4212
• decrease (or maybe increase) the network request timeout - change(perf): Decrease the request timeout to 15 seconds, to speed up the initial sync #4213

Quick tweaks, might help

• reduce the default hedge to 15 seconds, to increase the initial sync retry speed - covered by change(perf): Decrease the request timeout to 15 seconds, to speed up the initial sync #4213
• launch the full sync test in another cloud region that's closer to more Zcash peers

Out of scope

Needs separate tickets, priorities, scheduling:

Larger changes, likely to help

• move hash calculation and signature verification to a blocking thread, so they don't block network futures
  • we should only do this if the similar disk writes change helps
• profiling to see where Zebra is waiting - tokio-console Integrate tokio-console #2112