CoreDNS image incorrectly resolved when using --image-repository and can't use InitConfiguration on existing clusters?

[SGStino]

What keywords did you search in kubeadm issues before filing this one?

#2714 <- misinterpreted as a support request & closed instead of bug report
kubernetes-sigs/kubespray#8182 <- similar bug in kubespray, fixed.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

• kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:24:08Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
• kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4", GitCommit:"95ee5ab382d64cfe6c28967f36b53970b8374491", GitTreeState:"clean", BuildDate:"2022-08-17T18:52:53Z", GoVersion:"go1.18.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): N/A
• Cloud provider or hardware configuration: Metal, N/A
• OS (e.g. from /etc/os-release): Ubuntu 20.04, Ubuntu 22.04
• Kernel (e.g. uname -a): 5.4.0-124-generic & 5.4.0-124-generic
• Container runtime (CRI) (e.g. containerd, cri-o): cri-o 1.21.7 & cri-o 1.22.0
• Container networking plugin (CNI) (e.g. Calico, Cilium): cilium
• Others: N/A

What happened?

$ kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.24.4
k8s.gcr.io/kube-controller-manager:v1.24.4
k8s.gcr.io/kube-scheduler:v1.24.4
k8s.gcr.io/kube-proxy:v1.24.4
k8s.gcr.io/pause:3.7
k8s.gcr.io/etcd:3.5.3-0
k8s.gcr.io/coredns/coredns:v1.8.6

$ kubeadm config images list --image-repository=repo.apps.local
repo.apps.local/kube-apiserver:v1.24.4
repo.apps.local/kube-controller-manager:v1.24.4
repo.apps.local/kube-scheduler:v1.24.4
repo.apps.local/kube-proxy:v1.24.4
repo.apps.local/pause:3.7
repo.apps.local/etcd:3.5.3-0
repo.apps.local/coredns:v1.8.6

What you expected to happen?

$ kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.24.4
k8s.gcr.io/kube-controller-manager:v1.24.4
k8s.gcr.io/kube-scheduler:v1.24.4
k8s.gcr.io/kube-proxy:v1.24.4
k8s.gcr.io/pause:3.7
k8s.gcr.io/etcd:3.5.3-0
k8s.gcr.io/coredns/coredns:v1.8.6

$ kubeadm config images list --image-repository=repo.apps.local
repo.apps.local/kube-apiserver:v1.24.4
repo.apps.local/kube-controller-manager:v1.24.4
repo.apps.local/kube-scheduler:v1.24.4
repo.apps.local/kube-proxy:v1.24.4
repo.apps.local/pause:3.7
repo.apps.local/etcd:3.5.3-0
repo.apps.local/coredns/coredns:v1.8.6

How to reproduce it (as minimally and precisely as possible)?

$ kubeadm config images list --image-repository=repo.apps.local

Anything else we need to know?

Workaround is now to tell the pull-through repository to handle coredns differently than other images by adding k8s.gcr.io/coredns as extra upstream fall-back after k8s.gcr.io/ or to manually pull, re-tag and push it.

Using the init configuration lets you set the imageRepository for dns and etcd individually:
https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-ImageMeta

but that has an imageRepository and imageTag field, but I want to keep the path coredns/coredns, which isn't configurable in the configuration

[neolit123]

it's by design for backwards compatibility and the image path can differ when using image repo.

please read
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images