kubeadm init Airgap ENV

[debankur1]

Kubelet is not picking up the pause version set in the config file on kubeadm init,though it is getting set in the kubelet args
but kubelet is looking for registry.k8s.io/pause:3.6

● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Fri 2023-07-07 19:46:15 UTC; 5min ago
Docs: http://kubernetes.io/docs/
Main PID: 11922 (kubelet)
Tasks: 21 (limit: 23961)
Memory: 48.6M
CGroup: /system.slice/kubelet.service
└─11922 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=PRIVATE_REPO/pause:3.7>

Jul 07 19:52:02 snc-tca kubelet[11922]: E0707 19:52:02.197162 11922 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image "registry.k8.io>
Jul 07 19:52:02 snc-tca kubelet[11922]: E0707 19:52:02.197245 11922 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox image "registry.k8s.io>
Jul 07 19:52:02 snc-tca kubelet[11922]: E0707 19:52:02.197278 11922 kuberuntime_manager.go:815] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.6\": failed to pull image "registry.k8s.io/pause:3.6">
~
~
~
~

K8s Version : 1.24.14

kubeadm Config File :

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    pod-infra-container-image: PRIVATE_REPO/pause:3.7
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.24.14
imageRepository: PRIVATE_REPO
controllerManager:
  imageRepository: PRIVATE_REPO
  imageTag: v1.9.3
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    imageRepository: PRIVATE_REPO
    imageTag: v3.5.6
networking:
  podSubnet: "10.244.0.0/16"

Also configured the containerd config

[plugins]
[plugins."io.containerd.grpc.v1.cri"]
disable_apparmor = true
sandbox_image = "PRIVATE_REPO/pause:3.7"

[neolit123]

if the kubelet flag is --pod-infra-container-image=PRIVATE_REPO/pause:3.7 and containerd has sandbox_image = "PRIVATE_REPO/pause:3.7" then the image should always come from private repo.

maybe there is a problem with the containerd config... make sure its loaded.
try asking for help on support channels instead.

/support

[github-actions]

Hello, @debankur1 🤖 👋

You seem to have troubles using Kubernetes and kubeadm.
Note that our issue trackers should not be used for providing support to users.
There are special channels for that purpose.

Please see:

• https://github.com/kubernetes/kubeadm#support

[github-actions]

Hello, @debankur1 🤖 👋

You seem to have troubles using Kubernetes and kubeadm.
Note that our issue trackers should not be used for providing support to users.
There are special channels for that purpose.

Please see:

• https://github.com/kubernetes/kubeadm#support