[BUG] CoreDNS failed to patch on cluster creation

[praveern]

What did you do

I'm following the steps on this page: https://codeburst.io/creating-a-local-development-kubernetes-cluster-with-k3s-and-traefik-proxy-7a5033cb1c2d

• How was the cluster created?

  • k3d cluster create devcluster --api-port 127.0.0.1:6443 -p 80:80@loadbalancer -p 443:443@loadbalancer --k3s-server-arg "--no-deploy=traefik"

• What did you do afterwards?
  Nothing

What did you expect to happen

I was expecting to be able to simply visit localhost:6443 like the page specifies. And definitely was not expecting a "failed to patch CoreDNS" error.

Screenshots or terminal output

• Cluster creation terminal output
  INFO[0000] Prep: Network
  INFO[0000] Created network 'k3d-devcluster'
  INFO[0000] Created volume 'k3d-devcluster-images'
  INFO[0001] Creating node 'k3d-devcluster-server-0'
  INFO[0005] Pulling image 'docker.io/rancher/k3s:v1.20.2-k3s1'
  INFO[0012] Creating LoadBalancer 'k3d-devcluster-serverlb'
  INFO[0013] Pulling image 'docker.io/rancher/k3d-proxy:v4.2.0'
  INFO[0015] Starting cluster 'devcluster'
  INFO[0015] Starting servers...
  INFO[0015] Starting Node 'k3d-devcluster-server-0'
  INFO[0019] Starting agents...
  INFO[0019] Starting helpers...
  INFO[0019] Starting Node 'k3d-devcluster-serverlb'
  INFO[0020] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
  WARN[0021] Failed to patch CoreDNS ConfigMap to include entry '172.18.0.1 host.k3d.internal': Exec process in node 'k3d-devcluster-server-0' failed with exit code '1'
  INFO[0021] Successfully added host record to /etc/hosts in 2/2 nodes
  INFO[0021] Cluster 'devcluster' created successfully!
  INFO[0021] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
  INFO[0021] You can now use it like this:
  kubectl config use-context k3d-devcluster
  kubectl cluster-info

• Visiting "localhost:6443" and output:
  'Client sent an HTTP request to an HTTPS server.'

• Visiting "https://localhost:6443" and output:
  {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
  }

Which OS & Architecture

• Windows 10 Pro amd64, with Docker Desktop installed and WSL2 enabled - Ubuntu 20.04.1 LTS Distro

Which version of k3d

• k3d version v4.2.0
• k3s version v1.20.2-k3s1 (default)

Which version of docker

output of docker version

• Client: Docker Engine - Community
  Cloud integration: 1.0.7
  Version: 20.10.2
  API version: 1.41
  Go version: go1.13.15
  Git commit: 2291f61
  Built: Mon Dec 28 16:17:34 2020
  OS/Arch: linux/amd64
  Context: default
  Experimental: true

• Server: Docker Engine - Community
  Engine:
  Version: 20.10.2
  API version: 1.41 (minimum version 1.12)
  Go version: go1.13.15
  Git commit: 8891c58
  Built: Mon Dec 28 16:15:28 2020
  OS/Arch: linux/amd64
  Experimental: false
  containerd:
  Version: 1.4.3
  GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
  runc:
  Version: 1.0.0-rc92
  GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
  docker-init:
  Version: 0.19.0
  GitCommit: de40ad0

docker info

• Client:
  Context: default
  Debug Mode: false
  Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.5.0)

• Server:
  Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
  Images: 2
  Server Version: 20.10.2
  Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Cgroup Version: 1
  Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
  Swarm: inactive
  Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
  runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
  init version: de40ad0
  Security Options:
  seccomp
  Profile: default
  Kernel Version: 5.4.72-microsoft-standard-WSL2
  Operating System: Docker Desktop
  OSType: linux
  Architecture: x86_64
  CPUs: 12
  Total Memory: 12.3GiB
  Name: docker-desktop
  ID: "not shared to protect information"
  Docker Root Dir: /var/lib/docker
  Debug Mode: false
  Registry: https://index.docker.io/v1/
  Labels:
  Experimental: false
  Insecure Registries:
  127.0.0.0/8
  Live Restore Enabled: false

[murphye]

I am getting this too

k3d cluster create --api-port 6550 --registry-create --k3s-server-arg '--no-deploy=traefik' -p "8081:80@loadbalancer" localhost-1
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-localhost-1'            
INFO[0000] Created volume 'k3d-localhost-1-images'      
INFO[0000] Creating node 'k3d-localhost-1-registry'     
INFO[0000] Successfully created registry 'k3d-localhost-1-registry' 
INFO[0001] Creating node 'k3d-localhost-1-server-0'     
INFO[0001] Creating LoadBalancer 'k3d-localhost-1-serverlb' 
INFO[0001] Starting cluster 'localhost-1'               
INFO[0001] Starting servers...                          
INFO[0001] Starting Node 'k3d-localhost-1-server-0'     
INFO[0006] Starting agents...                           
INFO[0006] Starting helpers...                          
INFO[0006] Starting Node 'k3d-localhost-1-registry'     
INFO[0006] Starting Node 'k3d-localhost-1-serverlb'     
INFO[0006] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
WARN[0008] Failed to patch CoreDNS ConfigMap to include entry '172.22.0.1 host.k3d.internal': Exec process in node 'k3d-localhost-1-server-0' failed with exit code '1' 
INFO[0008] Successfully added host record to /etc/hosts in 3/3 nodes 
INFO[0009] Cluster 'localhost-1' created successfully!  
INFO[0009] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false 

Update: I recreated my k3d cluster several times. Most of the time it had this error, and sometimes not.

[iwilltry42]

Hi @praveern & @murphye , the warning is "only" a warning on purpose as it's just a "nice-to-have" addition and does not affect the usability of the cluster in any way.

@praveern, as said before, the warning regarding CoreDNS does not affect the expected usability of the cluster in any way 👍

I was expecting to be able to simply visit localhost:6443 like the page specifies. And definitely was not expecting a "failed to patch CoreDNS" error.

Where is that written on the linked page?
localhost:6443 brings you to the Kubernetes API-Server, which expects/requires a secure connection (https, as the error says) and authentication (kubeconfig) and is usually used via kubectl and not visited via browser.

[fgielow]

I depend on this domain to be setup so I can access resources from my local machine. I've been using this snippet to have that set-up:

# I am using WSL2, so I need to invoke powershell, if you are on native linux, just use ifconfig to create the interface: sudo ifconfig lo: 172.30.0.1 netmask 255.0.0.0 up
powershell.exe -Command "Start-Process powershell -ArgumentList 'New-NetIPAddress -InterfaceAlias Loopback -IPAddress 172.30.0.1 -PrefixLength 24' -Verb RunAs"

kubectl -n kube-system get configmap coredns -o yaml > coredns.yaml
cat coredns.yaml | sed -e 's/NodeHosts: |/NodeHosts: |\n    172.30.0.1 host.k3d.internal/g' > coredns-patched.yaml
kubectl -n kube-system apply -f coredns-patched.yaml 
rm coredns.yaml coredns-patched.yaml

Then I programatically look for that WARN message, and I execute this snippet myself if that's found. Sometimes that is set-up automatically, but most of the times it's not, so I rely on this workaround.

[iwilltry42]

Okay, let's try to figure out why it's failing for you in the first place and we may need to introduce a flag that sets this step (which is currently considered nice-to-have) as required (with proper error and retries in that case).

[memelet]

Yes, I too pretty much depend on this feature. It has only started failing for me in 4.4.6.

[iwilltry42]

So there is no pattern in when/how/why this is failing. It doesn't seem to depend on the k3d or the k3s version. For some it even only fails, if the loadbalancer is disabled.
Since we couldn't figure out anything that's triggering it, I just took the step to make this process a little more robust in general for v5.0.0.
The code for this is already in main-v5 and will be released in the next major version, so stay tuned!
I'll mark this issue as DONE, but only close it with the upcoming release.

[jamesdcarroll]

I was getting this error too and was about to comment as such. Thinking better of it though I downloaded the latest version moving from 4.4.3 to 4.4.6. And now it works as expected.