Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flannel CNI / DNS Issues with in-cluster Service Names when "hostNetwork: true" #6880

Closed
reefland opened this issue Feb 3, 2023 · 8 comments

Comments

@reefland
Copy link

reefland commented Feb 3, 2023

Environmental Info:
K3s Version:

k3s version v1.25.4+k3s1 (0dc63334)
go version go1.19.3

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04.1 / Bare Metal

$ uname -a
Linux k3s01 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

$ sudo kubectl get nodes
NAME    STATUS   ROLES                       AGE    VERSION
k3s01   Ready    control-plane,etcd,master   309d   v1.25.4+k3s1
k3s02   Ready    control-plane,etcd,master   282d   v1.25.4+k3s1
k3s03   Ready    control-plane,etcd,master   289d   v1.25.4+k3s1
k3s04   Ready    worker                      55d    v1.25.4+k3s1
k3s05   Ready    worker                      55d    v1.25.4+k3s1

Describe the bug:
When I tried to enable hostNetwork: true for a pod, that pod is no longer able to resolve in-cluster DNS names. If I set both hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet then neither internal nor external DNS names work.

With hostNetwork: false then both in-cluster and external DNS names work as expected.

Steps To Reproduce:
Using default flannel:

$ sudo  cat /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist

{
  "name":"cbr0",
  "cniVersion":"1.0.0",
  "plugins":[
    {
      "type":"flannel",
      "delegate":{
        "hairpinMode":true,
        "forceAddress":true,
        "isDefaultGateway":true
      }
    },
    {
      "type":"portmap",
      "capabilities":{
        "portMappings":true
      }
    }
  ]
}

From Control-Plane Service:

ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    server \
        '--cluster-init' \
        '--token' \
        '[REDACTED]' \
        '--disable' \
        'traefik' \
        '--kube-apiserver-arg=feature-gates=MixedProtocolLBService=true' \
        '--kubelet-arg=image-gc-high-threshold=65' \
        '--kubelet-arg=image-gc-low-threshold=50' \
        '--disable' \
        'local-storage' \
        '--disable' \
        'servicelb' \
        '--tls-san=192.168.10.239' 

From Agent Service:

ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    agent \
        '--kubelet-arg=image-gc-low-threshold=50' \
        '--kubelet-arg=image-gc-high-threshold=65' 

Expected behavior:
I expected to be able to resolve in-cluster and external DNS names with hostNetwork enabled.


No Host Networking: In-Cluster & External DNS Work

$ kubectl run curl --image=radial/busyboxplus:curl --rm=true --stdin=true --tty=true --restart=Never 

[ root@curl:/ ]$ cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local [REDACTED] localdomain
nameserver 10.43.0.10
options ndots:5

[ root@curl:/ ]$ curl http://grafana.monitoring.svc.cluster.local
<a href="/grafana/login">Found</a>.

[ root@curl:/ ]$ curl https://www.google.com | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14054    0 14054    0     0   164k      0 --:--:-- --:--:-- --:--:--  198k
16

Just hostNetwork: true: In-Cluster Fails but External DNS Works

This seems normal as the nameserver IP is the DNS for this LAN segment (for external DNS).

$ kubectl run curl --image=radial/busyboxplus:curl --rm=true --stdin=true --tty=true --restart=Never \
 --overrides='{"apiVersion": "v1", "spec": {"hostNetwork": true }}'
 
[ root@k3s05:/ ]$ cat /etc/resolv.conf
search [REDACTED] localdomain
nameserver 192.168.10.1

[ root@k3s05:/ ]$ curl http://grafana.monitoring.svc.cluster.local
curl: (6) Couldn't resolve host 'grafana.monitoring.svc.cluster.local'

[ root@k3s05:/ ]$ curl https://www.google.com | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14061    0 14061    0     0   190k      0 --:--:-- --:--:-- --:--:--  193k
16

Both hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet: In-Cluster & External DNS Fail

$ kubectl run curl --image=radial/busyboxplus:curl --rm=true --stdin=true --tty=true --restart=Never \
  --overrides='{"apiVersion": "v1", "spec": {"hostNetwork": true, "dnsPolicy": "ClusterFirstWithHostNet"}}'

[ root@k3s05:/ ]$ cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local [REDACTED] localdomain
nameserver 10.43.0.10
options ndots:5

[ root@k3s05:/ ]$ curl http://grafana.monitoring.svc.cluster.local
curl: (6) Couldn't resolve host 'grafana.monitoring.svc.cluster.local'

[ root@k3s05:/ ]$ curl https://www.google.com
curl: (6) Couldn't resolve host 'www.google.com'

While DNS fails, I can still curl the service directly and pod directly:

[ root@k3s05:/ ]$ curl http://10.43.174.7
<a href="/grafana/login">Found</a>.

[ root@k3s05:/ ]$ curl http://10.42.2.14:3000
<a href="/grafana/login">Found</a>.

Not clear if this is helpful:

$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done | uniq

[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
.:53
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] plugin/reload: Running configuration SHA512 = b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d
CoreDNS-1.9.4
linux/amd64, go1.19.1, 1f0a41a
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
$ sudo k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin:
- sha256sum: good
- links: aux/ip6tables should link to iptables-detect.sh (fail)
- links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
- links: aux/ip6tables-save should link to iptables-detect.sh (fail)
- links: aux/iptables should link to iptables-detect.sh (fail)
- links: aux/iptables-restore should link to iptables-detect.sh (fail)
- links: aux/iptables-save should link to iptables-detect.sh (fail)

System:
- /var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/aux iptables v1.8.6 (legacy): ok
- swap: disabled
- routes: default CIDRs 10.42.0.0/16 or 10.43.0.0/16 already routed

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-58-generic
info: reading kernel config from /boot/config-5.15.0-58-generic ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- /usr/sbin/apparmor_parser
apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: 6 (fail)
@brandond
Copy link
Member

brandond commented Feb 7, 2023

@reefland
Copy link
Author

reefland commented Feb 7, 2023

I just upgraded to latest available on 1.25, and have better results:

$ k get nodes                                                                                                                                              127 ↵
NAME    STATUS   ROLES                       AGE    VERSION
k3s01   Ready    control-plane,etcd,master   312d   v1.25.6+k3s1
k3s02   Ready    control-plane,etcd,master   286d   v1.25.6+k3s1
k3s03   Ready    control-plane,etcd,master   293d   v1.25.6+k3s1
k3s04   Ready    worker                      58d    v1.25.6+k3s1
k3s05   Ready    worker                      58d    v1.25.6+k3s1

I can now curl internal service:

$ kubectl run curl --image=radial/busyboxplus:curl --rm=true --stdin=true --tty=true --restart=Never \
  --overrides='{"apiVersion": "v1", "spec": {"hostNetwork": true, "dnsPolicy": "ClusterFirstWithHostNet"}}'
  
[ root@k3s04:/ ]$ curl http://grafana.monitoring.svc.cluster.local
<a href="/grafana/login">Found</a>.

No change to /etc/resolv.conf:

[ root@k3s04:/ ]$ cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local [REDACTED] localdomain
nameserver 10.43.0.10
options ndots:5

I didn't see anything obvious within the release notes that would explain this. I do have a single node test cluster on the side I can still reproduce the original issue as it is still on v1.25.4+k3s1 if that would be helpful.

@rbrtbnfgl
Copy link
Contributor

@brandond could it be related to the containerd bug?

@baowj-678
Copy link

I also encountered this problem. Only pods on the node that runing coredns-xxxx can visited In-Cluster DNS.
My k3s Cluster is:
k3s version v1.24.8+k3s1 (648004e4)
go version go1.18.8

@brandond
Copy link
Member

brandond commented Feb 7, 2023

@baowj-678 that usually indicates a problem with your kernel dropping vxlan packets between nodes. This is caused by a bug in the kernel checksum offload code. See the ethtool command at flannel-io/flannel#1279

@rbrtbnfgl
Copy link
Contributor

rbrtbnfgl commented Feb 7, 2023

That issue was solved from flannel v0.20.2. K3s v1.24.8 it's using v0.20.1 and also v1.25.4 uses that version then both issues could be related to that.

@baowj-678
Copy link

Thanks, I solved my problem by following this issue flannel-io/flannel#1279.

@reefland
Copy link
Author

reefland commented Feb 7, 2023

Thanks all, looks like upgrade of K3S updated the embedded flannel which resolved my issue. Closing this.

@reefland reefland closed this as completed Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants