You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A-debugissue requires detective debug work to figure out what's going wrong.P-lowthis is mostly nice to have.S-networkissues related to the dot/network package.T-bugthis issue covers unexpected and/or wrong behaviour.
When node is running it writes all outgoing and incoming connection in to the slots that has limited capacity. But when libp2phost.connect happens with error there is no fallback route for us to free occupied connection slot inside PeerState.
Error happens pereodically here.
Another problem is that apprently before libp2p ibrary version 32.2 there is a bug that does not bubbleups an error so this happens unnoticeably this statement also requires verification.
Mentioned below log was done within this branch with additional info logs added
Expected Behavior
IF libp2p error happens we should free occupied perstate slot. here
Current Behavior
PerrState slot is not getting released when libp2p.connect error hapens
failed to negotiate security protocol: error reading handshake message: noise: message is too short
Possible Solution
Basically handshake error is out of scope of this issue and generally we just need to add communication between network.service and peerstate so we can free occupied slot in case of error.
To Reproduce
Steps to reproduce the behaviour:
Run integration test
MODE=rpc go test ./tests/rpc/system_integration_test.go -v
libp2p version should be 32.2 before that error happens silently without bubbling up
Log output
Log Output
```
system_integration_test.go:54: Node alice-0 is ready
alice-0 2024-02-13T20:32:07+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:07+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:07+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
alice-0 2024-02-13T20:32:09+01:00 INFO peer count 0 pkg=network
alice-0 2024-02-13T20:32:09+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:09+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:09+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
alice-0 2024-02-13T20:32:10+01:00 INFO HANDLE PEER FOUND BY MDNS {12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: [/ip4/127.0.0.1/tcp/7005 /ip4/192.168.1.133/tcp/7005]} pkg=network
2024-02-13T20:32:10+01:00 INFO PESRSET ADD PEER 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset
2024-02-13T20:32:10+01:00 INFO Inserting peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset
2024-02-13T20:32:10+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:10+01:00 INFO tryOutgoing outgoing slot increase for 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset
2024-02-13T20:32:10+01:00 INFO Sent connect message to peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset
2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset
2024-02-13T20:32:10+01:00 INFO PROCESS NEW MESSAGE {Status:0 setID:0 PeerID:12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7} pkg=network
2024-02-13T20:32:10+01:00 INFO PROCESS MESSAGE CONNECT {Status:0 setID:0 PeerID:12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7} pkg=network
alice-0 2024-02-13T20:32:10+01:00 WARN failed to open connection for peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: failed to dial: failed to dial 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: all dials failed
* [/ip4/127.0.0.1/tcp/7005] failed to negotiate security protocol: error reading handshake message: noise: message is too short
* [/ip4/192.168.1.133/tcp/7005] failed to negotiate security protocol: error reading handshake message: noise: message is too short pkg=network
alice-0 2024-02-13T20:32:10+01:00 INFO built block 1 with hash 0xdf9ea7f6c66d5ffc3193087cfd6477cc53b26f7443af48eb2a40bdb6ce6f1bce, state root 0xec9b9073553a22b406a9d390768b484767f75eb1f762a335bd0f8008f54e34fd, epoch 0 and slot 284642121 pkg=babe
alice-0 2024-02-13T20:32:10+01:00 INFO 🔨 finalised block #0 (0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8), round 1, set id 0 pkg=state
system_integration_test.go:54: Node bob-1 is ready
alice-0 2024-02-13T20:32:11+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:11+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset
2024-02-13T20:32:11+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset
alice-0 2024-02-13T20:32:12+01:00 INFO peer count 0 pkg=network
alice-0 2024-02-13T20:32:13+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:13+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset
```
Specification
go version: 1.22
gossamer version:
gossamer commit tag:
gossamer commit hash:
operating system:
additional links:
The text was updated successfully, but these errors were encountered:
P1sar
added
T-bug
this issue covers unexpected and/or wrong behaviour.
S-network
issues related to the dot/network package.
A-debug
issue requires detective debug work to figure out what's going wrong.
labels
Feb 15, 2024
P1sar
changed the title
Libp2p connect handshake error and outgoing slot permanent occupation
PerrState connection slot is not released on network error
Feb 15, 2024
P1sar
changed the title
PerrState connection slot is not released on network error
PeerState connection slot is not released on network error
Feb 15, 2024
A-debugissue requires detective debug work to figure out what's going wrong.P-lowthis is mostly nice to have.S-networkissues related to the dot/network package.T-bugthis issue covers unexpected and/or wrong behaviour.
Describe the bug
When node is running it writes all outgoing and incoming connection in to the slots that has limited capacity. But when libp2phost.connect happens with error there is no fallback route for us to free occupied connection slot inside PeerState.
Error happens pereodically here.
Another problem is that apprently before libp2p ibrary version 32.2 there is a bug that does not bubbleups an error so this happens unnoticeably this statement also requires verification.
Mentioned below log was done within this branch with additional info logs added
Expected Behavior
IF libp2p error happens we should free occupied perstate slot.
here
Current Behavior
PerrState slot is not getting released when libp2p.connect error hapens
Possible Solution
Basically handshake error is out of scope of this issue and generally we just need to add communication between network.service and peerstate so we can free occupied slot in case of error.
To Reproduce
Steps to reproduce the behaviour:
Run integration test
Log output
Log Output
``` system_integration_test.go:54: Node alice-0 is ready alice-0 2024-02-13T20:32:07+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:07+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:07+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset alice-0 2024-02-13T20:32:09+01:00 INFO peer count 0 pkg=network alice-0 2024-02-13T20:32:09+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:09+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:09+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset alice-0 2024-02-13T20:32:10+01:00 INFO HANDLE PEER FOUND BY MDNS {12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: [/ip4/127.0.0.1/tcp/7005 /ip4/192.168.1.133/tcp/7005]} pkg=network 2024-02-13T20:32:10+01:00 INFO PESRSET ADD PEER 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset 2024-02-13T20:32:10+01:00 INFO Inserting peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset 2024-02-13T20:32:10+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset 2024-02-13T20:32:10+01:00 INFO tryOutgoing outgoing slot increase for 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset 2024-02-13T20:32:10+01:00 INFO Sent connect message to peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset 2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset 2024-02-13T20:32:10+01:00 INFO PROCESS NEW MESSAGE {Status:0 setID:0 PeerID:12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7} pkg=network 2024-02-13T20:32:10+01:00 INFO PROCESS MESSAGE CONNECT {Status:0 setID:0 PeerID:12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7} pkg=network alice-0 2024-02-13T20:32:10+01:00 WARN failed to open connection for peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: failed to dial: failed to dial 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: all dials failed * [/ip4/127.0.0.1/tcp/7005] failed to negotiate security protocol: error reading handshake message: noise: message is too short * [/ip4/192.168.1.133/tcp/7005] failed to negotiate security protocol: error reading handshake message: noise: message is too short pkg=network alice-0 2024-02-13T20:32:10+01:00 INFO built block 1 with hash 0xdf9ea7f6c66d5ffc3193087cfd6477cc53b26f7443af48eb2a40bdb6ce6f1bce, state root 0xec9b9073553a22b406a9d390768b484767f75eb1f762a335bd0f8008f54e34fd, epoch 0 and slot 284642121 pkg=babe alice-0 2024-02-13T20:32:10+01:00 INFO 🔨 finalised block #0 (0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8), round 1, set id 0 pkg=state system_integration_test.go:54: Node bob-1 is ready alice-0 2024-02-13T20:32:11+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:11+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset 2024-02-13T20:32:11+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset alice-0 2024-02-13T20:32:12+01:00 INFO peer count 0 pkg=network alice-0 2024-02-13T20:32:13+01:00 INFO Allocating slots for 0 pkg=peerset 2024-02-13T20:32:13+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset ```Specification
The text was updated successfully, but these errors were encountered: