Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable checksum offload on the IPIP tunnel like we do for VXLAN #7807

Closed
cyclinder opened this issue Jun 20, 2023 · 4 comments · Fixed by #8031
Closed

Disable checksum offload on the IPIP tunnel like we do for VXLAN #7807

cyclinder opened this issue Jun 20, 2023 · 4 comments · Fixed by #8031

Comments

@cyclinder
Copy link
Contributor

Expected Behavior

root@controller:~/cyclinder# ethtool -k tunl0 | grep tx-checksumming
tx-checksumming: on

As a part of projectcalico/felix#2811, I think we need also disable checksum offload on the IPIP tunnel for kernels <v5.7.

We encountered this problem in our production, the clients used IPIP mode, the clients found that the pod could not tcp communicate with across nodes, and finally solved it by executing the following command in every node:

ethtool -K tunl0 tx off

Current Behavior

calico only disable checksum offload on the VXLAN tunnel.

Possible Solution

Steps to Reproduce (for bugs)

Context

Similar reports: https://dev.hsrn.nyu.edu/hsrn-projects/kubernetes-bare-metal/-/issues/7

Your Environment

  • Calico version
  • Orchestrator version (e.g. kubernetes, mesos, rkt):
  • Operating System and version:
  • Link to your project (optional):
@mazdakn
Copy link
Member

mazdakn commented Jul 11, 2023

Hey @cyclinder , I don't think Felix disables IPIP tunnel checksum offload by default. Generally speaking, we recommend using vxlan instead of ipip as we have seen many issues with ipip due to lack of attention it's getting in different platform/NICs. There was also a major change to IPIP kernel v5.7 (or maybe v5.8) that made the situation worse.

@cyclinder
Copy link
Contributor Author

Thanks for the tip. @mazdakn

I think we have many IPIP users. and If calico supports IPIP mode, I think we should fix it, so should we also disables IPIP tunnel checksum offload by default?

There was also a major change to IPIP kernel v5.7 (or maybe v5.8) that made the situation worse.

Could you tell me more details about this? thanks.

@cyclinder
Copy link
Contributor Author

Hi @fasaxc, Should we disable checksum offload on the IPIP tunnel? If yes, I can work for this.

@fasaxc
Copy link
Member

fasaxc commented Sep 11, 2023

Yes, I think that's a good idea. Let's disable it by default. You could wire it up to the "ChecksumOffloadBroken" feature flag so that it can be overridden later for testing. (Then later if we find it's fixed we can dd autodetection in for the kernels where it works.)

@caseydavenport caseydavenport changed the title Is calico disable checksum offload on the IPIP tunnel? Disable checksum offload on the IPIP tunnel like we do for VXLAN Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants