[BUG] DNS fix (K3D_FIX_DNS=1) breaks DNS resolution for external domain names (Internet)

[giulianopz]

What did you do

• How was the cluster created?

  • k3d cluster create test1 (default is K3D_FIX_DNS=1)

• What did you do afterwards?
  It's not possible to reach servers external to the cluster due to DNS resolution being broken:

~$ kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
~$ k exec -it dnsutils -- nslookup google.com
Server:         10.43.0.10
Address:        10.43.0.10#53

Non-authoritative answer:
Name:   google.com.homenet.telecomitalia.it 
Address: 127.0.0.1
~$ k exec -it dnsutils -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local homenet.telecomitalia.it
nameserver 10.43.0.10
options ndots:5

Please, note that homenet.telecomitalia.it is the search domain in the resolv.conf of the host machine:

:~$ cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 8.8.8.8 
nameserver 8.8.4.4
nameserver 127.0.0.53
search homenet.telecomitalia.it

What did you expect to happen

External domain names should not be resolved to 127.0.0.1.

Screenshots or terminal output

If I disable the DNS fix setting K3D_FIX_DNS to 0, everything works as expected:

~$ K3D_FIX_DNS=0 k3d cluster create test2
~$ kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
~$ k exec -it dnsutils -- nslookup google.com
Server:         10.43.0.10
Address:        10.43.0.10#53

Non-authoritative answer:
Name:   google.com
Address: 216.58.204.142
Name:   google.com
Address: 2a00:1450:4002:414::200e
~$ k exec -it dnsutils -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5

Which OS & Architecture

arch: x86_64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
name: docker
os: Ubuntu 22.04.5 LTS
ostype: linux
version: 27.2.1

Which version of k3d

k3d version v5.7.4
k3s version v1.30.4-k3s1 (default)

Which version of docker

Client: Docker Engine - Community
 Version:           27.2.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        9e34c9b
 Built:             Fri Sep  6 12:08:15 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.2.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       8b539b8
  Built:            Fri Sep  6 12:08:15 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

---

I know that the DNS fix causes the script k3d-entrypoint-dns.sh to be registered as a hook for the server node, altering the iptables rules and resolv.conf file inside the container.

This is supposed to 'forward DNS queries to your local machine, e.g. to use your local company DNS', quoting the changelog. But I don't need this feature since I'm not behind a company VPN.

Why is this feature enabled by default? Is this the intended behavior or do I have some issues with local machine configuration?

[iwilltry42]

Hey! Thanks for this extensive report :)

Why is this feature enabled by default? Is this the intended behavior or do I have some issues with local machine configuration?

Because many people had this problem and no one ever (until now) reported any issue with it (it's also on by default in e.g. kind).
So it made sense to enable it by default.

I have a similar setup (k3d version, docker version, ubuntu version) and a local search domain in the resolv.conf - I don't have any problem at all.

I'm also honestly unsure where I would start debugging this. Probably by playing with dig and DNS lookup traces.