gossip responder error: Invalid argument

[leoluk]

Problem

Invalid Argument errors in the log:

[2020-02-06T20:29:05.676271464Z WARN  solana_core::streamer] gossip responder error: IO(Os { code: 22, kind: InvalidInput, message: "Invalid argument" })

strace yields this:

[pid  6557] sendto(3, "\2\0\0\0\335\31\203v\367\272&\313\335+F\215NE5oM\361\373\343&\270\307O\213f\237\305"..., 768, MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EINVAL (Invalid argument)
[pid  6557] sendto(3, "\2\0\0\0\335\31\203v\367\272&\313\335+F\215NE5oM\361\373\343&\270\307O\213f\237\305"..., 554, MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EINVAL (Invalid argument)
[pid  6557] sendto(3, "\2\0\0\0\335\31\203v\367\272&\313\335+F\215NE5oM\361\373\343&\270\307O\213f\237\305"..., 560, MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EINVAL (Invalid argument)
[pid  6557] sendto(3, "\2\0\0\0\335\31\203v\367\272&\313\335+F\215NE5oM\361\373\343&\270\307O\213f\237\305"..., 554, MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EINVAL (Invalid argument)

fd 3 is SOCK_DGRAM so I would expect dest_addr to not be zero:

solana-va  6144  solana    3u     IPv4             270906      0t0    UDP *:8239 

[mvines]

@pgarg66 - can you please get to the bottom of this log warning. Perhaps it's just a message that should be suppressed if it's not actually interesting. I've seen this message show up on my nodes at various times as well

[pgarg66]

@mvines it seems the error would occur when gossip is trying to reply to a peer with dest IP 0, and port 0. Looking thru the code, these are the two places where it can originate from

solana/core/src/cluster_info.rs

Line 1255 in 0d5c123

let _ignore_disconnect = response_sender.send(rsp);

solana/core/src/cluster_info.rs

Line 1298 in 0d5c123

let _ignore_disconnect = response_sender.send(rsp);

In these scenarios, the error is not fatal. We can either check the dest IP/port before sending the response (i.e. drop it if the IP/Port is 0/0), or we can suppress the warning.

@sagar-solana any thoughts?

[mvines]

We can either check the dest IP/port before sending the response (i.e. drop it if the IP/Port is 0/0), or we can suppress the warning.

Dropping a response to 0.0.0.0/0 makes sense to me. I think I'd rather do that then suppress the warning, as the warning is telling us that something weird is happening (sending a response to 0.0.0.0/0 that is!)

[pgarg66]

We can either check the dest IP/port before sending the response (i.e. drop it if the IP/Port is 0/0), or we can suppress the warning.

Dropping a response to 0.0.0.0/0 makes sense to me. I think I'd rather do that then suppress the warning, as the warning is telling us that something weird is happening (sending a response to 0.0.0.0/0 that is!)

Sounds good to me. I'll push a PR sometime today. Meanwhile, @sagar-solana , LMK if my assertion doesn't look correct.

[sagar-solana]

That looks correct. I think the problem is that for some reason gossip is working with invalid addresses. Gossip shouldn't be working with peers it can't reach. We see them in the output from spy as well.