Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connection tracking of VXLAN UDP packets must be disabled #2015

Closed
yoheiueda opened this issue Aug 22, 2024 · 0 comments · Fixed by #2016 or #2023
Closed

Connection tracking of VXLAN UDP packets must be disabled #2015

yoheiueda opened this issue Aug 22, 2024 · 0 comments · Fixed by #2016 or #2023
Assignees
@yoheiueda

Comments

@yoheiueda
Copy link
Member

The Linux connection tracking system (conntrack) always tracks UDP packets as connections by default. This is OK for usual UDP communications, but does not work with VXLAN UDP packets, since a destination port of each VXLAN UDP packet is fixed (4789) in both directions, and a source port of each packet is randomly selected. This UDP packet flow does not creates a connection flow, and every UDP packet of a VXLAN tunnel is recognized as a first packet of a new UDP stream.

We can confirm this behavior with the conntrack command. We can see a lot of "UNREPLIED" conntrack entries as follows.

# conntrack -L | grep "udp" | grep "UNREPLIED" | grep "dport=4789"
conntrack v1.4.6 (conntrack-tools): 50 flow entries have been shown.
udp      17 26 src=192.168.122.185 dst=192.168.122.121 sport=34196 dport=4789 [UNREPLIED] src=192.168.122.121 dst=192.168.122.185 sport=4789 dport=34196 mark=0 use=1
udp      17 27 src=192.168.122.121 dst=192.168.122.185 sport=46394 dport=4789 [UNREPLIED] src=192.168.122.185 dst=192.168.122.121 sport=4789 dport=46394 mark=0 use=1
udp      17 25 src=192.168.122.121 dst=192.168.122.185 sport=52605 dport=4789 [UNREPLIED] src=192.168.122.185 dst=192.168.122.121 sport=4789 dport=52605 mark=0 use=1
udp      17 27 src=192.168.122.121 dst=192.168.122.185 sport=59264 dport=4789 [UNREPLIED] src=192.168.122.185 dst=192.168.122.121 sport=4789 dport=59264 mark=0 use=1
udp      17 25 src=192.168.122.185 dst=192.168.122.121 sport=59949 dport=4789 [UNREPLIED] src=192.168.122.121 dst=192.168.122.185 sport=4789 dport=59949 mark=0 use=1
udp      17 25 src=192.168.122.185 dst=192.168.122.121 sport=57929 dport=4789 [UNREPLIED] src=192.168.122.121 dst=192.168.122.185 sport=4789 dport=57929 mark=0 use=1
udp      17 25 src=192.168.122.121 dst=192.168.122.185 sport=57557 dport=4789 [UNREPLIED] src=192.168.122.185 dst=192.168.122.121 sport=4789 dport=57557 mark=0 use=1
udp      17 25 src=192.168.122.185 dst=192.168.122.121 sport=50136 dport=4789 [UNREPLIED] src=192.168.122.121 dst=192.168.122.185 sport=4789 dport=50136 mark=0 use=1
udp      17 25 src=192.168.122.121 dst=192.168.122.185 sport=38657 dport=4789 [UNREPLIED] src=192.168.122.185 dst=192.168.122.121 sport=4789 dport=38657 mark=0 use=1
udp      17 27 src=192.168.122.185 dst=192.168.122.121 sport=58652 dport=4789 [UNREPLIED] src=192.168.122.121 dst=192.168.122.185 sport=4789 dport=58652 mark=0 use=1
udp      17 25 src=192.168.122.121 dst=192.168.122.185 sport=49527 dport=4789 [UNREPLIED] src=192.168.122.185 dst=192.168.122.121 sport=4789 dport=49527 mark=0 use=1

When conntrack table fills up, it may affect other network connections that needs NAT.

This issue is reported in detail in projectcalico/calico#8934, and Calico has already implemented a solution for it.
@yoheiueda yoheiueda self-assigned this Aug 22, 2024
@yoheiueda yoheiueda mentioned this issue Aug 23, 2024
Merged
yoheiueda added a commit to yoheiueda/cloud-api-adaptor that referenced this issue Aug 30, 2024
@yoheiueda
podnetwork: Disable connection tracking of VXLAN UDP packets
400ac23 
This patch disables connection tracking of VXLAN UDP packets to
prevent the conntrack table from filing up.

The Linux connection tracking system (conntrack) always tracks UDP
packets as connections by default. This is OK for usual UDP
communications, but does not work with VXLAN UDP packets, since a
destination port of each VXLAN UDP packet is fixed (4789) in both
directions, and a source port of each packet is randomly selected.
This UDP packet flow does not creates a connection flow, and every
UDP packet of a VXLAN tunnel is recognized as a first packet of a
new UDP stream.

If conntrack table fills up, it may affect other network
connections that needs NAT. This patch will alleviate such problems.

Fixes confidential-containers#2015

Signed-off-by: Yohei Ueda <[email protected]>
@yoheiueda yoheiueda mentioned this issue Aug 30, 2024
Merged
yoheiueda added a commit to yoheiueda/cloud-api-adaptor that referenced this issue Sep 6, 2024
@yoheiueda
podnetwork: Disable connection tracking of VXLAN UDP packets
b0abc1a 
This patch disables connection tracking of VXLAN UDP packets to
prevent the conntrack table from filing up in both worker node
and per pod VMs.

The Linux connection tracking system (conntrack) always tracks UDP
packets as connections by default. This is OK for usual UDP
communications, but does not work with VXLAN UDP packets, since a
destination port of each VXLAN UDP packet is fixed (4789) in both
directions, and a source port of each packet is randomly selected.
This UDP packet flow does not creates a connection flow, and every
UDP packet of a VXLAN tunnel is recognized as a first packet of a
new UDP stream.

If conntrack table fills up, it may affect other network
connections that needs NAT. This patch will alleviate such problems.

Fixes confidential-containers#2015

Signed-off-by: Yohei Ueda <[email protected]>
bpradipt pushed a commit that referenced this issue Sep 6, 2024
@yoheiueda @bpradipt
podnetwork: Disable connection tracking of VXLAN UDP packets
2a79d39 
This patch disables connection tracking of VXLAN UDP packets to
prevent the conntrack table from filing up in both worker node
and per pod VMs.

The Linux connection tracking system (conntrack) always tracks UDP
packets as connections by default. This is OK for usual UDP
communications, but does not work with VXLAN UDP packets, since a
destination port of each VXLAN UDP packet is fixed (4789) in both
directions, and a source port of each packet is randomly selected.
This UDP packet flow does not creates a connection flow, and every
UDP packet of a VXLAN tunnel is recognized as a first packet of a
new UDP stream.

If conntrack table fills up, it may affect other network
connections that needs NAT. This patch will alleviate such problems.

Fixes #2015

Signed-off-by: Yohei Ueda <[email protected]>
@yoheiueda yoheiueda mentioned this issue Oct 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yoheiueda yoheiueda

Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@yoheiueda