report failed requests from RPC after being clear from rate limiter but not yet sent.

[jxs]

Issue Addressed

When RPC::send_request() is called it checks if a request can be sent by assessing its own self_limiter, if a request can be sent it will be immediately added to events if not it will be queued on self_limiter.

When RPC receives notice that a peer has disconnected it gets the list of pending requests to the disconnected peer from self_limiter to report them as failed requests, but it doesn't check requests that might have been directly added to events.

This PR addresses that, by transforming those pending requests into reports of failed requests.

[pawanjay176]

Nice

[dapplion]

LGTM! This should address the issue we saw on 5.2.0 with this sequence of logs?

Jun 12 09:01:33.528 DEBG Sending BlobsByRoot Request, id: SingleLookupReqId { lookup_id: 38, req_id: 41 }, peer: 16Uiu2HAkutTnUn36ype96onoudLfnNBzXbTxc5yfN47Nj117egBr, blob_indices: [0, 1, 2, 3, 4, 5], block_root: 0x5a5f0e6471eebb31b288995b279b0a9faafd6fff0d706ba29b590240ec2fd47f, method: BlobsByRoot, service: sync, module: network::sync::network_context:432
..
Jun 12 09:01:36.025 DEBG Request exceeds the rate limit, wait_time_ms: 1963, peer_id: 16Uiu2HAkutTnUn36ype96onoudLfnNBzXbTxc5yfN47Nj117egBr, request: MetaData request, service: libp2p_rpc, service: libp2p, module: lighthouse_network::rpc:368
Jun 12 09:01:36.026 DEBG RPC Error, direction: Incoming, score: -10, peer_id: 16Uiu2HAkutTnUn36ype96onoudLfnNBzXbTxc5yfN47Nj117egBr, client: Lighthouse: version: v5.1.3-3058b96, os_version: x86_64-linux, err: RPC response was an error: Rate limited with reason: Wait 1.963549303s, protocol: metadata, service: libp2p, module: lighthouse_network::peer_manager:489
Jun 12 09:01:36.027 DEBG Peer transitioned to forced disconnect score state, past_score_state: Healthy, score: -20.00, peer_id: 16Uiu2HAkutTnUn36ype96onoudLfnNBzXbTxc5yfN47Nj117egBr, service: libp2p, module: lighthouse_network::peer_manager::peerdb:1085
Jun 12 09:01:36.027 DEBG Peer Manager disconnecting peer, reason: Bad Score, peer_id: 16Uiu2HAkutTnUn36ype96onoudLfnNBzXbTxc5yfN47Nj117egBr, service: libp2p, module: lighthouse_network::service:1690
Jun 12 09:01:36.164 DEBG Peer disconnected, peer_id: 16Uiu2HAmB4ZFcUmeRy3jpFsdQszQh8w4fUYpnWrSh4y3jRYAiRJq, service: libp2p, module: lighthouse_network::peer_manager::network_behaviour:308
Jun 12 09:01:36.164 DEBG Received disconnected message, peer_id: 16Uiu2HAmB4ZFcUmeRy3jpFsdQszQh8w4fUYpnWrSh4y3jRYAiRJq, service: sync, module: network::sync::manager:646
..
Jun 12 09:17:35.021 WARN Notify the devs, a sync lookup is stuck, ancestor_lookup: SingleBlockLookup { 
    id: 38, 
    blob_request_state: BlobRequestState { 
        state: SingleLookupRequestState { 
            state: Downloading(41), 
            failed_processing: 0, 
            failed_downloading: 0 
        } 
    }, 

[AgeManning]

It's not clear where that message got dropped. We've looked into it, it could be from this PR or one in the handler that I added: #5945

[AgeManning]

@Mergifyio queue

[mergify]

queue

🛑 The pull request has been removed from the queue default

Pull request #5942 has been dequeued by a dequeue command.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

[jimmygchen]

@mergify unqueue

[mergify]

unqueue

✅ The pull request has been removed from the queue default

[jimmygchen]

@mergify requeue

[mergify]

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request

[mergify]

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 1503f7d