Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PeerState connection slot is not released on network error #3758

Open
P1sar opened this issue Feb 15, 2024 · 0 comments
Open

PeerState connection slot is not released on network error #3758

P1sar opened this issue Feb 15, 2024 · 0 comments
Assignees
Labels
A-debug issue requires detective debug work to figure out what's going wrong. P-low this is mostly nice to have. S-network issues related to the dot/network package. T-bug this issue covers unexpected and/or wrong behaviour.

Comments

@P1sar
Copy link
Member

P1sar commented Feb 15, 2024

Describe the bug

When node is running it writes all outgoing and incoming connection in to the slots that has limited capacity. But when libp2phost.connect happens with error there is no fallback route for us to free occupied connection slot inside PeerState.
Error happens pereodically here.
Another problem is that apprently before libp2p ibrary version 32.2 there is a bug that does not bubbleups an error so this happens unnoticeably this statement also requires verification.

Mentioned below log was done within this branch with additional info logs added

Expected Behavior

IF libp2p error happens we should free occupied perstate slot.
here

Current Behavior

PerrState slot is not getting released when libp2p.connect error hapens

 failed to negotiate security protocol: error reading handshake message: noise: message is too short

Possible Solution

Basically handshake error is out of scope of this issue and generally we just need to add communication between network.service and peerstate so we can free occupied slot in case of error.

To Reproduce

Steps to reproduce the behaviour:
Run integration test

  1. MODE=rpc go test ./tests/rpc/system_integration_test.go -v
  2. libp2p version should be 32.2 before that error happens silently without bubbling up

Log output

Log Output ``` system_integration_test.go:54: Node alice-0 is ready alice-0 2024-02-13T20:32:07+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:07+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:07+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset alice-0 2024-02-13T20:32:09+01:00 INFO peer count 0 pkg=network alice-0 2024-02-13T20:32:09+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:09+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:09+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset alice-0 2024-02-13T20:32:10+01:00 INFO HANDLE PEER FOUND BY MDNS {12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: [/ip4/127.0.0.1/tcp/7005 /ip4/192.168.1.133/tcp/7005]} pkg=network 2024-02-13T20:32:10+01:00 INFO PESRSET ADD PEER 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset 2024-02-13T20:32:10+01:00 INFO Inserting peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset 2024-02-13T20:32:10+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:10+01:00 INFO tryOutgoing outgoing slot increase for 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset 2024-02-13T20:32:10+01:00 INFO Sent connect message to peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset 2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset 2024-02-13T20:32:10+01:00 INFO PROCESS NEW MESSAGE {Status:0 setID:0 PeerID:12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7} pkg=network 2024-02-13T20:32:10+01:00 INFO PROCESS MESSAGE CONNECT {Status:0 setID:0 PeerID:12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7} pkg=network alice-0 2024-02-13T20:32:10+01:00 WARN failed to open connection for peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: failed to dial: failed to dial 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: all dials failed * [/ip4/127.0.0.1/tcp/7005] failed to negotiate security protocol: error reading handshake message: noise: message is too short * [/ip4/192.168.1.133/tcp/7005] failed to negotiate security protocol: error reading handshake message: noise: message is too short pkg=network alice-0 2024-02-13T20:32:10+01:00 INFO built block 1 with hash 0xdf9ea7f6c66d5ffc3193087cfd6477cc53b26f7443af48eb2a40bdb6ce6f1bce, state root 0xec9b9073553a22b406a9d390768b484767f75eb1f762a335bd0f8008f54e34fd, epoch 0 and slot 284642121 pkg=babe alice-0 2024-02-13T20:32:10+01:00 INFO 🔨 finalised block #0 (0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8), round 1, set id 0 pkg=state system_integration_test.go:54: Node bob-1 is ready alice-0 2024-02-13T20:32:11+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:11+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset 2024-02-13T20:32:11+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset alice-0 2024-02-13T20:32:12+01:00 INFO peer count 0 pkg=network alice-0 2024-02-13T20:32:13+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:13+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset ```

Specification

  • go version: 1.22
  • gossamer version:
  • gossamer commit tag:
  • gossamer commit hash:
  • operating system:
  • additional links:
@P1sar P1sar added T-bug this issue covers unexpected and/or wrong behaviour. S-network issues related to the dot/network package. A-debug issue requires detective debug work to figure out what's going wrong. labels Feb 15, 2024
@P1sar P1sar self-assigned this Feb 15, 2024
@P1sar P1sar changed the title Libp2p connect handshake error and outgoing slot permanent occupation PerrState connection slot is not released on network error Feb 15, 2024
@P1sar P1sar changed the title PerrState connection slot is not released on network error PeerState connection slot is not released on network error Feb 15, 2024
@P1sar P1sar added the P-low this is mostly nice to have. label Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-debug issue requires detective debug work to figure out what's going wrong. P-low this is mostly nice to have. S-network issues related to the dot/network package. T-bug this issue covers unexpected and/or wrong behaviour.
Projects
None yet
Development

No branches or pull requests

1 participant