The network speed of the pod cannot reach the speed of the baremetal, only half the speed of the baremetal server

[ming12713]

calico version: v3.26.1
kubernetes version: v1.26.6
calico installation spec

spec:
  calicoNetwork:
    bgp: Enabled
    hostPorts: Enabled
    ipPools:
    - blockSize: 26
      cidr: 10.244.0.0/16
      disableBGPExport: false
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled
      nodeSelector: all()
    linuxDataplane: Iptables
    multiInterfaceMode: None
    nodeAddressAutodetectionV4:
      firstFound: true

iperf3 testing in pod

1. iperf3 server pod running on airflow01 node with 10g nic
2. iperf3 client pod running on airflow02 node with 10g nic
   

iperf3 testing in baremetal node

1. iperf3 server running on airflow01 node with 10g nic
2. iperf3 client running on airflow02 node with 10g nic
   

[sridhartigera]

@ming12713 I am not sure if I understand this correctly. Baremetal has 10G NICs and iperf output is ~10Gbps. Am I missing something?

[ming12713]

@sridhartigera
Yes, the baremetal servers have 10G network card, and when using iperf to test on the baremetal servers, the speed reaches 10G. However, when pods are running on these baremetal servers using the Calcio plugin, point-to-point network testing does not achieve 10G. The two pods being tested are located on different baremetal server

[lwr20]

Halving of throughput generally indicates MTU issues. Often throughput is limited by maximum packets-per-second, and if there is an MTU issue in the path, that will lead to fragmentation and a doubling of the number of packets (which halves throughput)

See https://docs.tigera.io/calico/latest/networking/configuring/mtu

[ming12713]

@lwr20 thanks
I've set the MTU to 8950 and then used iperf to perform bandwidth testing between different pods. However, the bandwidth still doesn't reach the bandwidth of the baremeta servers.

[lwr20]

OK, and what's the MTU of the network interface between the nodes (eth0 or whatever)? And the MTU set on any routers between the nodes?

[ming12713]

@lwr20 thanks, baremeta server network interface mtu is 9000,nodes connected through 10G switch, and the 10G switch default setting of 1500 mtu.

[lwr20]

That doesn't sound good - if the server has 9000 MTU, switch should also have MTU=9000.

But on the other hand, that's the same for both baremetal and pod-pod case, so its clearly not the cause of this issue.

You sound like you're using VXLAN encapsulation (since you mentioned 8950 MTU setting in Calico). Do you need VXLAN encap at all in this scenario? ISTR there was a recent linux kernel bug with VXLAN checksum offloading. Can you try without VXLAN to establish if the problem is related to VXLAN or not?

[ming12713]

yes ,i use VXLANCrossSubnet encap, i think I've encountered the same bug you mentioned.
releate issue #7974

[lwr20]

I don't think that's the VXLAN checksum offload issue, that's a kernel crash, isn't it? The VXLAN checksum offload issue "just" causes dropped packets (I think)

Based on #4727 (comment)
Can you try setting featureDetectOverride: "ChecksumOffloadBroken=true" in the default FelixConfiguration and see if that fixes the issue please?

[ming12713]

@lwr20
I changed the encapsulation VXLANCossSubnet mode to IPIPCrossSubnet, and after testing, I found that the bandwidth still couldn't be fully utilized. IPIP performs slightly worse than VXLAN in terms of performance.

[onesb23]

I don't think that's the VXLAN checksum offload issue, that's a kernel crash, isn't it? The VXLAN checksum offload issue "just" causes dropped packets (I think)

Based on #4727 (comment) Can you try setting featureDetectOverride: "ChecksumOffloadBroken=true" in the default FelixConfiguration and see if that fixes the issue please?

This fixed the vxlan calico underspeed issue for me.

[mazdakn]

@ming12713 have you tried the fix that @lwr20 mentioned above?

[ming12713]

@ming12713 have you tried the fix that @lwr20 mentioned above?
no