Range sync rate limit exhaust when execution node is syncing

[nazarhussain]

Describe the bug

When beacon node start and tries to sync from genesis, it usually fall back to range sync because head of connected peers are too far in the future.

1. It tries to call BeaconBlocksByRange method to get blocks from connected peer.
2. If the execution engine in the meanwhile is syncing it failed to process the blocks with

Batch process error id=Finalized, startEpoch=4, status=Processing, code=BLOCK_ERROR_EXECUTION_ERROR, execStatus=UNSAFE_OPTIMISTIC_STATUS, errorMessage=not safe to import SYNCING payload within 128 of currentSlot

3. It does not stop with this error and keep calling BeaconBlocksByRange continuously.
4. This exhaust the rate limit for BeaconBlocksByRange which is 1024 blocks per 10s.
   

   lodestar/packages/beacon-node/src/network/reqresp/rateLimit.ts

   Line 31 in 5047410

   byPeer: {quota: MAX_REQUEST_BLOCKS, quotaTimeMs: 10_000},

5. Till the connected execution engine sync, beacon node can't proceed sync further because of rate limit on BeaconBlocksByRange.

Expected behavior

It should be able to proceed with range sync after waiting the execution node to be synced.

Possible solution could be to wait for the execution engine to sync before initiating multiple concurrent BeaconBlocksByRange requests.

Steps to reproduce

1. Run a node pair and wait for couple of epochs
2. Run the second node pair in sequence that beacon node and execution node starts same time.

Additional context

Look for the logs of range-sync-node-beacon-lodestar.log

multi-fork.zip

Operating system

Linux

Lodestar version or commit hash

unstable