Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

registry.k8s.io/pause:3.6 does not use the address specified by --image-repository when executing kubeadm init #2851

Closed
Max-Qiu opened this issue Mar 29, 2023 · 14 comments
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@Max-Qiu
Copy link

Max-Qiu commented Mar 29, 2023

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

What happened?

  1. use kubeadm init ...... --image-repository registry.aliyuncs.com/google_containers , it show :
    [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory /etc/kubernetes/manifests". This can take up to 4m0s
    [kubelet-check] Initial timeout of 40s passed.
  2. then use systemctl status -l kubelet.service , it show :
    remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image "registry.k8s.io/pause:3.6": failed to pull image "registry.k8s.io/pause:3.6": failed to pull and unpack image "registry.k8s.io/pause:3.6": failed to resolve reference "registry.k8s.io/pause:3.6": failed to do request: Head "https://registry.k8s.io/v2/pause/manifests/3.6\": net/http: TLS handshake timeout"

What did you expect to happen?

I hope that registry.k8s.io/pause:3.6 will be downloaded through the address specified by --image-repository

How can we reproduce it (as minimally and precisely as possible)?

  1. Block registry.k8s.io
  2. Execute kubeadm init --control-plane-endpoint=192.168.220.101 --kubernetes-version v1.26.3 --pod-network-cidr=10.244.0.0/16 --image-repository registry.aliyuncs.com/google_containers

Anything else we need to know?

I need to execute the following code on each node to solve the problem

ctr -n k8s.io i pull registry.aliyuncs.com/google_containers/pause:3.6
ctr -n k8s.io i tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6

Kubernetes version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

OS version

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ uname -a
Linux node1 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
@pacoxu
Copy link
Member

pacoxu commented Mar 29, 2023

Check your containerd configuration.

@neolit123
Copy link
Member

see

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#override-pause-image-containerd

/kind support

@k8s-ci-robot k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Mar 29, 2023
@LxxxSec
Copy link

LxxxSec commented May 5, 2023

我也遇到了和你类似的问题,我的解决方案如下:

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

接着编辑/etc/containerd/config.toml文件,把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可,如下方代码所示(pause版本因人而异):

sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

接着重启服务,重新init即可:

systemctl restart containerd.service

@luyajin
Copy link

luyajin commented Jul 24, 2023

我修改了这个配置怎么一直都不生效呢

@luyajin
Copy link

luyajin commented Jul 24, 2023

sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

@linonetwo
Copy link

Is this a bug? i'm using docker, not containerd , how to fix it on user side?

@chendave
Copy link
Member

Is this a bug? i'm using docker, not containerd , how to fix it on user side?

this is not a bug, it's just a matter of configuration, shouldn't docker use containerd as the runtime by default?

@neolit123
Copy link
Member

Is this a bug? i'm using docker, not containerd , how to fix it on user side?

k8s maintainers recommend to move to containerd or cri-o.
docker (via cri-dockerd) is no longer well supported.

@kingwingfly
Copy link

systemctl restart containerd.service

义父!!!!!!!!!!!!!!!!!

@zhkmxx9302013
Copy link

我也遇到了和你类似的问题,我的解决方案如下:

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

接着编辑/etc/containerd/config.toml文件,把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可,如下方代码所示(pause版本因人而异):

sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

接着重启服务,重新init即可:

systemctl restart containerd.service

困扰了半个月...换了一堆源,原来在这里。。

@jiangxiaoqiang
Copy link

我也遇到了和你类似的问题,我的解决方案如下:

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

接着编辑/etc/containerd/config.toml文件,把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可,如下方代码所示(pause版本因人而异):

sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

接着重启服务,重新init即可:

systemctl restart containerd.service

我也是这个问题,不过是在集群运行过程中无法拉取。修改了配置不生效不知道是什么原因。

@Kensleycc
Copy link

Kensleycc commented Aug 28, 2024

我也是这个问题,不过是在集群运行过程中无法拉取。修改了配置不生效不知道是什么原因。

给后来的朋友记录一下我的情况

【环境】

使用cri为cri-dockerd,已经修改/usr/lib/systemd/system/cri-docker.service

cat /usr/lib/systemd/system/cri-docker.service
...
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=<Repository Path>/pause:3.8

【问题】

masterkubeadm init没问题,worker kubeadm join 之后,对应的 kube-proxy pod 一直处于FailedCreatePodSandBox .

Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               6m33s                 default-scheduler  Successfully assigned kube-system/kube-proxy-h2k9q to kendev2
  Warning  FailedCreatePodSandBox  80s (x26 over 6m42s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "registry.k8s.io/pause:3.9": Error response from daemon: Get "https://registry.k8s.io/v2/": dial tcp 34.96.108.209:443: connect: connection refused

【Solution】

最后定位原因是worker未重启cri-dockerd进程使配置生效:

# systemctl status cri-docker.service 

● cri-docker.service - CRI Interface for Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-08-27 11:44:00 UTC; 21h ago
TriggeredBy: ● cri-docker.socket
       Docs: https://docs.mirantis.com
   Main PID: 103962 (cri-dockerd)
      Tasks: 11
     Memory: 18.6M
     CGroup: /system.slice/cri-docker.service
             └─103962 /usr/bin/cri-dockerd --container-runtime-endpoint fd://

重启后worker重新join即可:

systemctl restart cri-docker.service

systemctl status cri-docker.service  
● cri-docker.service - CRI Interface for Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-08-28 09:19:19 UTC; 2s ago
TriggeredBy: ● cri-docker.socket
       Docs: https://docs.mirantis.com
   Main PID: 2045345 (cri-dockerd)
      Tasks: 8
     Memory: 10.5M
     CGroup: /system.slice/cri-docker.service
             └─2045345 /usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=<Repository Path>/pause:3.8

@neolit123
Copy link
Member

you probably need to tell cri-dockerd to use exactly that image.
maybe there is a cri-dockerd config for that. try asking the cri-docker maintainers.

@WQL782795
Copy link

我也遇到了和你类似的问题,我的解决方案如下:

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

接着编辑/etc/containerd/config.toml文件,把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可,如下方代码所示(pause版本因人而异):

sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

接着重启服务,重新init即可:

systemctl restart containerd.service

我也遇到了和你类似的问题,我的解决方案如下:

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

接着编辑/etc/containerd/config.toml文件,把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可,如下方代码所示(pause版本因人而异):

sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

接着重启服务,重新init即可:

systemctl restart containerd.service

义父+1!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Development

No branches or pull requests