k3s on rhel 8 network/dns probleme and metrics not work #5013
Labels
Comments
|
Thanks for reporting this. I'm trying with rhel 8.4 and I see things working. Please make sure that the port 8472 is not being blocked. Could you please verify the following:
|
|
Hi @manuelbuil , the interface is present, all port is open firewalld is disabled other track rancher/windows#96 |
Are you using vSphere? |
Can you check if the communication of pods in the same node work? |
Here is the output of lsmod (I haven't modprobe anything): |
|
@manuelbuil thanks for your reply, ok so is not modprob yes i use vsphere i can ping pods to anothers pods on same nodes and other kubectl exec -i -t dnsutils -- nslookup kubernetes.default command terminated with exit code 1 logs for coredns pods nothing tcpdump in 8.2 in 8.3 |
|
Could you please try execute these two commands: In all nodes and show me the output? Thanks |
|
pods master 1 master 2 worker 1 worker 2 worker 3 front 1 front 2 |
|
Thanks, so it works in |
|
Can you please show me the output of |
|
@manuelbuil yes is run in worker 2 see |
|
Can you try to ping coredns from other nodes? |
|
Master 1 Node front 1 worker 1 from pod running on worker 3 |
|
According to the iptables rules you showed me the coredns IP is |
|
Could you run the following:
And show me the output? |
|
And also |
|
@manuelbuil yes restarted nodes we see clearly there is no communication |
Thanks. There is something strange going on. Can you try again the dig command but now:
|
|
@manuelbuil ok sorry is verbose Node 1 Node 02 |
|
Oh ==> you might be hitting a kernel bug that affects udp + vxlan when using the offloading feature of the kernel. We saw it in Ubuntu but thought it was fixed in RHEL ==> rancher/rke2#1541 Could you please try disabling the offloading in all nodes? Execute this command |
|
worked ! @manuelbuil Thanks for helping me debug thank you ! |
Thanks for helping and your quick response! Something we need to fix in flannel upstream |
|
Note that there are issues with RHEL 8 and vmware. There is one related to vxlan which maybe is the root cause for our issue ==> https://docs.vmware.com/en/VMware-vSphere/6.7/rn/esxi670-202111001.html#esxi670-202111401-bg-resolved |
|
I think I'm having a very similar issue, but it's a single host deployment of k3s onto a RHEL 8.4 system. The node is deployed in AWS EC2, and I'm just trying to get a light cluster started up using k3s. However, none of the pods that I deploy to k3s can reach the apiserver ( including coredns it appears ). |
HI! Can you give us the output of: |
|
Sorry, maybe I did something wrong. I created a new instance, disabled nm-cloud-setup.service and nm-cloud-setup.timer, restarted the instance, and installed k3s and now it seems to work,. I'm pretty sure the last time I disabled both services and it still wasn't functioning. |
|
Same issue still happening with RHEL8.6. Pod communication entirely broken. added the above fix to crontab to band-aid things so it survives reboot.
This fix should be posted in the readme to avoid headaches. It took a bit of digging to find this issue. |
|
Is EL8 still shipping a kernel with broken vxlan tx checksum offload? |
Yes it is. Not unusual for Redhat since they typically move at the speed of molasses when making changes. I had three new VMs with a fresh install of 8.6 and still broken. Same bad udp issue when I used tcpdump. The ethtool change fixed it. |
|
I can confirm the problem and ethtool fix on RHEL 8.6. The following have known broken kernels on VMWare vSphere with the vxlan tx checksum offload bug:
Also, I believe the problem manifests when VMs cross VMWare vSphere hosts, not when they're on a single host. The following is known good:
I don't know about 8.5. But it was fixed in 8.4. This is a another regression. |
|
Hi, we did some tests again with RHEL8.3 and and we see a really strange problem with vsphere. cluster with RHEL8.3 and vm hardware 11 work, the bad checksum is present but they are no impact in cluster cluster in RHEL8.3 and vm hardware in 15 to 19 not work, problème with resolution dns (tested with rke2 and k3s) This probleme is know in openshift and fixed in uptream |
|
This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions. |
We encountered an issue where the
#!/usr/bin/env bash
# Maximum wait time in seconds (e.g., 300 seconds = 5 minutes)
MAX_WAIT=300
WAIT_INTERVAL=10
ELAPSED_TIME=0
while ! ip link show flannel.1 &> /dev/null; do
sleep $WAIT_INTERVAL
ELAPSED_TIME=$((ELAPSED_TIME + WAIT_INTERVAL))
if [ $ELAPSED_TIME -ge $MAX_WAIT ]; then
echo "Timed out waiting for flannel.1 interface to become ready."
exit 1
fi
done
# Now that flannel.1 is up, run the ethtool command
ethtool -K flannel.1 tx-checksum-ip-generic off
|
It can also be done by setting |
|
We just experienced this same problem running k3s v1.30.5+k3s1 on Ubuntu 20.04.6 LTS Kernel Version 5.15.0-124-generic running on Vsphere |
|
Yes. the vmware virtual nic driver is known to have bugs in its checksum offload support, and will generate packets with bad checksums. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello I try to make k3s work in a redhat 8.4 but I encounter network or dns problems, I checked the modprob as well as sysctl but nothing happens maybe is flannel problem ?
firewalld and selinux disabled
nm-cloud-setup.service nm-cloud-setup.timer no present
k3s installed by script https://get.k3s.io
work fine in rhel 7.9
Environmental Info:
K3s Version:
k3s version v1.22.5+k3s1 (405bf79)
go version go1.16.10
Node(s) CPU architecture, OS, and Version:
Linux vldsocfg01 4.18.0-305.25.1.el8_4.x86_64 #1 SMP Mon Oct 18 14:34:11 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
2 master
3 node
2 node only front (traefik / metallb / haproxy )
Describe the bug:
pods crash with dns resolution probleme
coredns:
longhorn:
metrics:
in logs k3s see metrics error
lsmod
Module Size Used by
xt_state 16384 0
veth 28672 0
nf_conntrack_netlink 49152 0
xt_recent 20480 6
xt_statistic 16384 21
xt_nat 16384 44
ip6t_MASQUERADE 16384 1
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 172032 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nft_chain_nat 16384 8
ipt_MASQUERADE 16384 5
vxlan 65536 0
ip6_udp_tunnel 16384 1 vxlan
udp_tunnel 20480 1 vxlan
nfnetlink_log 20480 1
nft_limit 16384 1
ipt_REJECT 16384 5
nf_reject_ipv4 16384 1 ipt_REJECT
xt_limit 16384 0
xt_NFLOG 16384 1
xt_physdev 16384 2
xt_conntrack 16384 21
xt_mark 16384 25
xt_multiport 16384 4
xt_addrtype 16384 7
nft_counter 16384 329
xt_comment 16384 296
nft_compat 20480 550
nf_tables 172032 884 nft_compat,nft_counter,nft_chain_nat,nft_limit
ip_set 49152 0
nfnetlink 16384 5 nft_compat,nf_conntrack_netlink,nf_tables,ip_set,nfnetlink_log
iptable_nat 16384 0
nf_nat 45056 5 ip6t_MASQUERADE,ipt_MASQUERADE,xt_nat,nft_chain_nat,iptable_nat
nf_conntrack 172032 8 xt_conntrack,nf_nat,ip6t_MASQUERADE,xt_state,ipt_MASQUERADE,xt_nat,nf_conntrack_netlink,ip_vs
nf_defrag_ipv6 20480 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack
cfg80211 835584 0
rfkill 28672 2 cfg80211
vsock_loopback 16384 0
vmw_vsock_virtio_transport_common 32768 1 vsock_loopback
vmw_vsock_vmci_transport 32768 1
vsock 45056 5 vmw_vsock_virtio_transport_common,vsock_loopback,vmw_vsock_vmci_transport
sunrpc 540672 1
intel_rapl_msr 16384 0
intel_rapl_common 24576 1 intel_rapl_msr
isst_if_mbox_msr 16384 0
isst_if_common 16384 1 isst_if_mbox_msr
nfit 65536 0
libnvdimm 192512 1 nfit
crct10dif_pclmul 16384 1
crc32_pclmul 16384 0
ghash_clmulni_intel 16384 0
rapl 20480 0
vmw_balloon 24576 0
joydev 24576 0
pcspkr 16384 0
vmw_vmci 86016 2 vmw_balloon,vmw_vsock_vmci_transport
i2c_piix4 24576 0
br_netfilter 24576 0
bridge 192512 1 br_netfilter
stp 16384 1 bridge
llc 16384 2 bridge,stp
overlay 135168 4
ip_tables 28672 1 iptable_nat
xfs 1515520 7
libcrc32c 16384 5 nf_conntrack,nf_nat,nf_tables,xfs,ip_vs
sr_mod 28672 0
cdrom 65536 1 sr_mod
sd_mod 53248 4
t10_pi 16384 1 sd_mod
sg 40960 0
ata_generic 16384 0
vmwgfx 368640 1
crc32c_intel 24576 1
drm_kms_helper 233472 1 vmwgfx
syscopyarea 16384 1 drm_kms_helper
sysfillrect 16384 1 drm_kms_helper
sysimgblt 16384 1 drm_kms_helper
fb_sys_fops 16384 1 drm_kms_helper
ata_piix 36864 0
ttm 114688 1 vmwgfx
serio_raw 16384 0
libata 270336 2 ata_piix,ata_generic
drm 569344 4 vmwgfx,drm_kms_helper,ttm
vmxnet3 65536 0
vmw_pvscsi 28672 8
dm_mod 151552 21
fuse 151552 1
iptables 1.8.4
in sysctl
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
Steps To Reproduce:
tried to delete iptables package of real for use iptables from k3s but same result
UPDATE:
with params --flannel-backend=host-gw is work, but is good fix ?
ingress not work with host-gw because front node is not in same network of worker
The text was updated successfully, but these errors were encountered: