-
Notifications
You must be signed in to change notification settings - Fork 715
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CoreDNS image cannot be pulled when use --image-repository #2714
Comments
Please see
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images
The difference in image path is by design.
/close
/kind support
|
@neolit123: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I think coredns not be resolved correctly with root@master:/etc/containerd/certs.d# kubeadm config images list --image-repository=next-repo-host
next-repo-host/kube-apiserver:v1.24.2
next-repo-host/kube-controller-manager:v1.24.2
next-repo-host/kube-scheduler:v1.24.2
next-repo-host/kube-proxy:v1.24.2
next-repo-host/pause:3.7
next-repo-host/etcd:3.5.3-0
next-repo-host/coredns:v1.8.6
|
/reopen |
@wnow20: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
See https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images If the repo is not the official k8s gcr it doesn't the coredns subpath |
Same here. it isn't possible to pull coredns image with custom image repository. |
more description on stackoverflow: |
this section explains how to handle it: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images
and steps below. |
Here is a demo.
BTW, you can generate the init.yaml by |
This is absurd. This issue must be resolved. We're encountering this trying to create 1.21 K8s cluster with --image-repository pointing at registry.k8s.io |
This is example for anyone wondering how to write a kubeadm config file (docs): apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
imageRepository: my-private-registry/registry.k8s.io
dns:
imageRepository: my-private-registry/registry.k8s.io/coredns You will also need to edit containerd config to use sandbox image from your private repo:
Then run |
The old k8s.gcr.io Kubernetes image registry has been frozen in Apr, 2023, in favor of registry.k8s.io. Although old images are still present in the old registry, we have recently started witnessing failures (e.g., [1]) in the k8s-1.16-kernel-4.19 Jenkins test during the provisioning phase, with errors along the lines of: [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.15-0: output: Error response from daemon: Head "https://k8s.gcr.io/v2/etcd/manifests/3.3.15-0": unable to decode token response: invalid character '<' looking for beginning of value Let's attempt to address this error by explicitly configuring the usage of the newer registry, until v1.13 goes EOL and we can finally get rid of these tests. Additionally, we explicitly specify the coredns image repository, as it seems that in certain versions (v1.21 in particular) it otherwise defaults to using an incorrect path (i.e., without the coredns subpath) if a custom registry is specified [2]: ERROR ImagePull]: failed to pull image registry.k8s.io/coredns:v1.8.0: output: Error response from daemon: manifest for registry.k8s.io/coredns:v1.8.0 not found: manifest unknown: Failed to fetch "v1.8.0" [1]: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1323/console [2]: kubernetes/kubeadm#2714 Signed-off-by: Marco Iorio <[email protected]>
The old k8s.gcr.io Kubernetes image registry has been frozen in Apr, 2023, in favor of registry.k8s.io. Although old images are still present in the old registry, we have recently started witnessing failures (e.g., [1]) in the k8s-1.16-kernel-4.19 Jenkins test during the provisioning phase, with errors along the lines of: [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.15-0: output: Error response from daemon: Head "https://k8s.gcr.io/v2/etcd/manifests/3.3.15-0": unable to decode token response: invalid character '<' looking for beginning of value Let's attempt to address this error by explicitly configuring the usage of the newer registry, until v1.13 goes EOL and we can finally get rid of these tests. Additionally, we explicitly specify the coredns image repository, as it seems that in certain versions (v1.21 in particular) it otherwise defaults to using an incorrect path (i.e., without the coredns subpath) if a custom registry is specified [2]: ERROR ImagePull]: failed to pull image registry.k8s.io/coredns:v1.8.0: output: Error response from daemon: manifest for registry.k8s.io/coredns:v1.8.0 not found: manifest unknown: Failed to fetch "v1.8.0" Finally, let's hard-code the coredns version for older k8s versions, as coredns older than v1.8.0 seems to follow yet another versioning scheme in the registry. [1]: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1323/console [2]: kubernetes/kubeadm#2714 Signed-off-by: Marco Iorio <[email protected]>
The old k8s.gcr.io Kubernetes image registry has been frozen in Apr, 2023, in favor of registry.k8s.io. Although old images are still present in the old registry, we have recently started witnessing failures (e.g., [1]) in the k8s-1.16-kernel-4.19 Jenkins test during the provisioning phase, with errors along the lines of: [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.15-0: output: Error response from daemon: Head "https://k8s.gcr.io/v2/etcd/manifests/3.3.15-0": unable to decode token response: invalid character '<' looking for beginning of value Let's attempt to address this error by explicitly configuring the usage of the newer registry, until v1.13 goes EOL and we can finally get rid of these tests. Additionally, we explicitly specify the coredns image repository, as it seems that in certain versions (v1.21 in particular) it otherwise defaults to using an incorrect path (i.e., without the coredns subpath) if a custom registry is specified [2]: ERROR ImagePull]: failed to pull image registry.k8s.io/coredns:v1.8.0: output: Error response from daemon: manifest for registry.k8s.io/coredns:v1.8.0 not found: manifest unknown: Failed to fetch "v1.8.0" Finally, let's hard-code the coredns version for older k8s versions, as coredns older than v1.8.0 seems to follow yet another versioning scheme in the registry. [1]: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1323/console [2]: kubernetes/kubeadm#2714 Signed-off-by: Marco Iorio <[email protected]>
Worked for me! |
What keywords did you search in kubeadm issues before filing this one?
coredns image
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use
kubeadm version
):kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:20:54Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:
kubectl version
): kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:20:54Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}uname -a
):Linux master-01 3.10.0-1160.66.1.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Wed May 18 16:02:34 UTC 2022 x86_64 x86_64 x86_64 GNU/LinuxWhat happened?
kubeadm config images list
output:...
...
k8s.gcr.io/coredns/coredns:v1.8.6
BUT when
kubeadm config images pull --image-repository=harbor.xxx.tech/k8s-gcr-proxy
ERROR:
failed to pull image "harbor.xxx.tech/k8s-gcr-proxy/coredns:v1.8.6"
the image path should be harbor.xxx.tech/k8s-gcr-proxy/coredns/coredns:v1.8.6 instead of
harbor.xxx.tech/k8s-gcr-proxy/coredns:v1.8.6
What you expected to happen?
this command
kubeadm config images pull --image-repository=harbor.xxx.tech/k8s-gcr-proxy
should be able to pull coredns imageHow to reproduce it (as minimally and precisely as possible)?
clean machine
install kubeadm, kubelet and kubectl
execute
kubeadm config images pull --image-repository=harbor.xxx.tech/k8s-gcr-proxy
Anything else we need to know?
harbor.xxx.tech/k8s-gcr-proxy
is a registry proxy using https://github.com/goharbor/harborThe text was updated successfully, but these errors were encountered: