Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition] #13872

Orygeunik · 2022-03-28T21:00:15Z

What Happened?

Hello!
I'm tried to enable ingress addon, but takes error such as from title
partial reference for issue #13841
Originally posted by @spowelljr in #13841 (comment)

root@orion:~# minikube addons enable ingress --alsologtostderr
I0328 21:23:12.286390 1954403 out.go:297] Setting OutFile to fd 1 ...
I0328 21:23:12.286699 1954403 out.go:349] isatty.IsTerminal(1) = true
I0328 21:23:12.286760 1954403 out.go:310] Setting ErrFile to fd 2...
I0328 21:23:12.286819 1954403 out.go:349] isatty.IsTerminal(2) = true
I0328 21:23:12.287050 1954403 root.go:315] Updating PATH: /root/.minikube/bin
W0328 21:23:12.287255 1954403 root.go:293] Error reading config file at /root/.minikube/config/config.json: open /root/.minikube/config/config.json: no such file or directory
I0328 21:23:12.288046 1954403 config.go:176] Loaded profile config "minikube": Driver=none, ContainerRuntime=docker, KubernetesVersion=v1.23.4-rc.0
I0328 21:23:12.288111 1954403 addons.go:65] Setting ingress=true in profile "minikube"
I0328 21:23:12.288159 1954403 addons.go:153] Setting addon ingress=true in "minikube"
I0328 21:23:12.288490 1954403 host.go:66] Checking if "minikube" exists ...
I0328 21:23:12.289084 1954403 exec_runner.go:51] Run: systemctl --version
I0328 21:23:12.294233 1954403 kubeconfig.go:92] found "minikube" server: "https://192.168.188.160:8443"
I0328 21:23:12.294262 1954403 api_server.go:165] Checking apiserver status ...
I0328 21:23:12.294304 1954403 exec_runner.go:51] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0328 21:23:12.305745 1954403 exec_runner.go:51] Run: sudo egrep ^[0-9]+:freezer: /proc/1951927/cgroup
W0328 21:23:12.314202 1954403 api_server.go:176] unable to find freezer cgroup: sudo egrep ^[0-9]+:freezer: /proc/1951927/cgroup: exit status 1
stdout:

stderr:
I0328 21:23:12.314232 1954403 api_server.go:240] Checking apiserver healthz at https://192.168.188.160:8443/healthz ...
I0328 21:23:12.319308 1954403 api_server.go:266] https://192.168.188.160:8443/healthz returned 200:
ok
I0328 21:23:12.334410 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
I0328 21:23:12.344282 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0328 21:23:12.354291 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0328 21:23:12.354596 1954403 addons.go:348] installing /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.354654 1954403 exec_runner.go:144] found /etc/kubernetes/addons/ingress-deploy.yaml, removing ...
I0328 21:23:12.354670 1954403 exec_runner.go:207] rm: /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.354731 1954403 exec_runner.go:151] cp: memory --> /etc/kubernetes/addons/ingress-deploy.yaml (15567 bytes)
I0328 21:23:12.354865 1954403 exec_runner.go:51] Run: sudo cp -a /tmp/minikube3233579437 /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.363407 1954403 exec_runner.go:51] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.936710 1954403 addons.go:386] Verifying addon ingress=true in "minikube"
I0328 21:23:12.999722 1954403 out.go:176] 
🔎  Verifying ingress addon...
I0328 21:23:13.000806 1954403 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0328 21:23:13.044867 1954403 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0328 21:23:13.044897 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:13.548198 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:14.048591 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:14.547859 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
... cut ...
I0328 21:29:13.050823 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:29:13.050850 1954403 kapi.go:108] duration metric: took 6m0.050047714s to wait for app.kubernetes.io/name=ingress-nginx ...
I0328 21:29:13.063110 1954403 out.go:176]

W0328 21:29:13.063269 1954403 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
W0328 21:29:13.063341 1954403 out.go:241]

W0328 21:29:13.072474 1954403 out.go:241]

Attach the log file

logs.txt

Operating System

Other (Debian 11.3)

Driver

none

The text was updated successfully, but these errors were encountered:

spowelljr · 2022-03-28T21:12:34Z

Hi @Orygeunik, thanks for reporting your issue with minikube!

Just looking at the logs I'm seeing bind: address already in use and Operation cannot be fulfilled on jobs.batch "ingress-nginx-admission-create": the object has been modified.

I also see that the cluster was originally created with 1.24.0 and now you're using 1.25.2.

The original cluster might have discrepancies with the updated version, I'd recommend starting with deleting the existing clusters minikube delete --all and then trying starting minikube again and enabling ingress again and let me know if that resolves your issue or your get different errors, thanks!

Orygeunik · 2022-03-28T21:26:12Z

Thanks for quick answer!

At first i want to remark that this error was received me on version 1.24 and i was hopes that update fixit it :)

So i was executed command minikube delete --all (little question, how to update minikube without errors like this?) and again run command minikube addons enable ingress --alsologtostderr
But errror ocured:

W0329 00:26:38.460857 1983999 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

Orygeunik · 2022-03-28T21:32:51Z

root@orion:~# minikube addons enable ingress --alsologtostderr
I0329 00:20:36.650345 1983999 out.go:297] Setting OutFile to fd 1 ...
I0329 00:20:36.652417 1983999 out.go:349] isatty.IsTerminal(1) = true
I0329 00:20:36.652433 1983999 out.go:310] Setting ErrFile to fd 2...
I0329 00:20:36.652492 1983999 out.go:349] isatty.IsTerminal(2) = true
I0329 00:20:36.652659 1983999 root.go:315] Updating PATH: /root/.minikube/bin
I0329 00:20:36.653128 1983999 config.go:176] Loaded profile config "minikube": Driver=none, ContainerRuntime=docker, KubernetesVersion=v1.23.4-rc.0
I0329 00:20:36.653186 1983999 addons.go:65] Setting ingress=true in profile "minikube"
I0329 00:20:36.653203 1983999 addons.go:153] Setting addon ingress=true in "minikube"
I0329 00:20:36.653407 1983999 host.go:66] Checking if "minikube" exists ...
I0329 00:20:36.653684 1983999 exec_runner.go:51] Run: systemctl --version
I0329 00:20:36.682942 1983999 kubeconfig.go:92] found "minikube" server: "https://192.168.188.160:8443"
I0329 00:20:36.683005 1983999 api_server.go:165] Checking apiserver status ...
I0329 00:20:36.683048 1983999 exec_runner.go:51] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0329 00:20:36.730755 1983999 exec_runner.go:51] Run: sudo egrep ^[0-9]+:freezer: /proc/1983293/cgroup
W0329 00:20:36.754976 1983999 api_server.go:176] unable to find freezer cgroup: sudo egrep ^[0-9]+:freezer: /proc/1983293/cgroup: exit status 1
stdout:

stderr:
I0329 00:20:36.755079 1983999 api_server.go:240] Checking apiserver healthz at https://192.168.188.160:8443/healthz ...
I0329 00:20:36.759621 1983999 api_server.go:266] https://192.168.188.160:8443/healthz returned 200:
ok
I0329 00:20:36.770284 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
I0329 00:20:36.779802 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0329 00:20:36.788875 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0329 00:20:36.789150 1983999 addons.go:348] installing /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.789188 1983999 exec_runner.go:144] found /etc/kubernetes/addons/ingress-deploy.yaml, removing ...
I0329 00:20:36.789207 1983999 exec_runner.go:207] rm: /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.789274 1983999 exec_runner.go:151] cp: memory --> /etc/kubernetes/addons/ingress-deploy.yaml (15567 bytes)
I0329 00:20:36.789416 1983999 exec_runner.go:51] Run: sudo cp -a /tmp/minikube101386137 /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.795590 1983999 exec_runner.go:51] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:38.395316 1983999 exec_runner.go:84] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml: (1.599607702s)
I0329 00:20:38.395385 1983999 addons.go:386] Verifying addon ingress=true in "minikube"
I0329 00:20:38.409437 1983999 out.go:176] 🔎  Verifying ingress addon...
🔎  Verifying ingress addon...
I0329 00:20:38.410507 1983999 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0329 00:20:38.433520 1983999 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0329 00:20:38.433657 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:20:38.937061 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:20:39.437494 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
...cut...
I0329 00:26:38.440491 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:26:38.440518 1983999 kapi.go:108] duration metric: took 6m0.030023123s to wait for app.kubernetes.io/name=ingress-nginx ...
I0329 00:26:38.460716 1983999 out.go:176]

W0329 00:26:38.460857 1983999 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
W0329 00:26:38.460873 1983999 out.go:241]

W0329 00:26:38.462117 1983999 out.go:241]

new logs.txt

Orygeunik · 2022-04-01T13:07:41Z

I'm sorry @spowelljr, anything other need?

spowelljr · 2022-04-20T18:38:43Z

Hi @Orygeunik, sorry for the delayed response.

After the ingress fails to enable, you can run the command kubectl get pods -A and you should see something like the following:

NAMESPACE       NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-8dnv9       0/1     Completed   0          3m14s
ingress-nginx   ingress-nginx-admission-patch-2x7q2        0/1     Completed   1          3m14s
ingress-nginx   ingress-nginx-controller-cc8496874-xxjhc   1/1     Running     0          3m15s
kube-system     coredns-64897985d-zlbf8                    1/1     Running     0          5m3s
kube-system     etcd-minikube                              1/1     Running     0          5m17s
kube-system     kube-apiserver-minikube                    1/1     Running     0          5m15s
kube-system     kube-controller-manager-minikube           1/1     Running     0          5m15s
kube-system     kube-proxy-lzrng                           1/1     Running     0          5m3s
kube-system     kube-scheduler-minikube                    1/1     Running     0          5m15s
kube-system     storage-provisioner                        1/1     Running     0          5m14s

Then if you do kubectl describe pod -n ingress-nginx <pod-name> you should be able to see all of the logs within the pod, I'm assuming one of those ingress pods is going to be failing to start and will have useful errors in there.

Orygeunik · 2022-04-21T15:37:55Z

Hello @spowelljr

I was rerun again deleting and enabling, such as in comment: #13872 (comment)

And i have below situation:

root@orion:~# kubectl get pods -A
NAMESPACE       NAME                                       READY   STATUS    RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-8nlcb       0/1     Pending   0          7m
ingress-nginx   ingress-nginx-admission-patch-jp8wz        0/1     Pending   0          7m
ingress-nginx   ingress-nginx-controller-cc8496874-9lhhr   0/1     Pending   0          7m
kube-system     coredns-64897985d-5hzh5                    0/1     Pending   0          7m18s
kube-system     etcd-orion                                 1/1     Running   6          7m30s
kube-system     kube-apiserver-orion                       1/1     Running   7          7m30s
kube-system     kube-controller-manager-orion              1/1     Running   7          7m30s
kube-system     kube-proxy-pl8nc                           1/1     Running   0          7m19s
kube-system     kube-scheduler-orion                       1/1     Running   6          7m30s
kube-system     storage-provisioner                        0/1     Pending   0          7m24s

kubectl describe pod -n ingress-nginx ingress-nginx-admission-create-8nlcb

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  31s (x390 over 6h40m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

kubectl describe pod -n ingress-nginx ingress-nginx-admission-patch-jp8wz

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  86s (x390 over 6h41m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

kubectl describe pod -n ingress-nginx ingress-nginx-controller-cc8496874-9lhhr

Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  35s (x396 over 6h45m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

16871560 · 2022-05-09T09:43:24Z

@Orygeunik do you solve this problem , I got the same issue. oops....

16871560 · 2022-05-09T09:44:39Z

Orygeunik · 2022-05-09T11:57:08Z

do you solve this problem , I got the same issue. oops....

@16871560 no, the problem still persisted

16871560 · 2022-05-10T03:25:05Z

@Orygeunik I reinstall 1.23.2. it seems work well.

Orygeunik · 2022-05-10T17:11:51Z

@16871560
So, is in my case it no results

root@orion:~# minikube start --driver=none --network-plugin=cni --bootstrapper=kubeadm --kubernetes-version=latest
😄  minikube v1.23.2 on Debian 11.3

root@orion:~# minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.3
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
🔎  Verifying ingress addon...

❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

i think so problem around driver, i used "none" and it's a source of problems :(

Orygeunik · 2022-05-10T18:14:57Z

And so, for tests i was changed driver to "docker" and minikube versio to "v1.23.2" and... it's work

root@orion:~# minikube start --driver=docker --bootstrapper=kubeadm --kubernetes-version=latest --force
😄  minikube v1.23.2 on Debian 11.3

root@orion:~# minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.3
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

More proofs:

root@orion:~# minikube start --driver=docker --bootstrapper=kubeadm --kubernetes-version=latest --force
😄  minikube v1.25.2 on Debian 11.3

root@orion:~# minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

culprit is driver - Q.E.D. (and idk why)

k8s-triage-robot · 2022-08-08T18:47:09Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

peachestao · 2022-08-28T08:04:53Z

@Orygeunik I also encountered this problem, and then check the pod error log, and prompt "0 / 1 nodes are available: 1 node (s) didn't match pod's node affinity / selector Preemption: 0 / 1 nodes are available: 1 preemption is not helpful for scheduling ", i find pod's label "minikube.k8s.io/primary=true" does not exist in the minikube node. later, the problem is solved by running the command:"minikube label node minikube minikube.k8s.io/primary=true"

k8s-triage-robot · 2022-09-27T08:37:35Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

oluranticode · 2022-10-04T18:58:44Z

Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

please can anyone help me, I ca't enable ingress on minikube

I'm using mac book Apple M1 chip

klaases · 2022-10-10T23:19:09Z

please can anyone help me, I ca't enable ingress on minikube

Hi @oluranticode – for additional assistance, please consider reaching out to the minikube community:

https://minikube.sigs.k8s.io/community/

We also offer support through Slack, Groups, and office hours.

manan · 2022-10-29T01:13:23Z

I have the same issue. It only fails with --driver=none but works otherwise.

techxplrorer · 2022-11-09T07:19:43Z

Hi @Orygeunik, sorry for the delayed response.

After the ingress fails to enable, you can run the command kubectl get pods -A and you should see something like the following:

NAMESPACE       NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-8dnv9       0/1     Completed   0          3m14s
ingress-nginx   ingress-nginx-admission-patch-2x7q2        0/1     Completed   1          3m14s
ingress-nginx   ingress-nginx-controller-cc8496874-xxjhc   1/1     Running     0          3m15s
kube-system     coredns-64897985d-zlbf8                    1/1     Running     0          5m3s
kube-system     etcd-minikube                              1/1     Running     0          5m17s
kube-system     kube-apiserver-minikube                    1/1     Running     0          5m15s
kube-system     kube-controller-manager-minikube           1/1     Running     0          5m15s
kube-system     kube-proxy-lzrng                           1/1     Running     0          5m3s
kube-system     kube-scheduler-minikube                    1/1     Running     0          5m15s
kube-system     storage-provisioner                        1/1     Running     0          5m14s

Then if you do kubectl describe pod -n ingress-nginx <pod-name> you should be able to see all of the logs within the pod, I'm assuming one of those ingress pods is going to be failing to start and will have useful errors in there.

idris@xplorer:~$ kubectl describe pods ingress-nginx-controller-755dfbfc65-s68pr -n ingress-nginx
Name: ingress-nginx-controller-755dfbfc65-s68pr
Namespace: ingress-nginx
Priority: 0
Service Account: ingress-nginx
Node: minikube/192.168.58.2
Start Time: Wed, 09 Nov 2022 14:24:15 +0800
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
gcp-auth-skip-secret=true
pod-template-hash=755dfbfc65
Annotations:
Status: Pending
IP: 172.17.0.5
IPs:
IP: 172.17.0.5
Controlled By: ReplicaSet/ingress-nginx-controller-755dfbfc65
Containers:
controller:
Container ID:
Image: k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 80/TCP, 443/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--watch-ingress-without-class=true
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
--udp-services-configmap=$(POD_NAMESPACE)/udp-services
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-755dfbfc65-s68pr (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jngmg (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-jngmg:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
minikube.k8s.io/primary=true
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 49m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-755dfbfc65-s68pr to minikube
Warning FailedMount 48m (x8 over 49m) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
Warning FailedMount 47m kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert kube-api-access-jngmg]: timed out waiting for the condition
Normal Pulling 40m (x3 over 46m) kubelet Pulling image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8"
Warning Failed 38m (x3 over 43m) kubelet Error: ErrImagePull
Warning Failed 38m (x4 over 43m) kubelet Error: ImagePullBackOff
Warning Failed 13m (x8 over 43m) kubelet Failed to pull image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8": rpc error: code = Unknown desc = context deadline exceeded
Normal BackOff 4m2s (x101 over 43m) kubelet Back-off pulling image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8"

spowelljr · 2022-11-09T14:44:31Z

rpc error: code = Unknown desc = context deadline exceeded this is a sign that the operation is taking longer than 2 minutes. The image k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f81 is not huge, but on a machine with slow internet it's possible it's taking longer than 2 minutes to download it.

Could you try running time docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 and see how long it takes to download the image on your machine.

You could also try deleting your minikube cluster (minikube delete) and then start it again with the --extra-config=kubelet.runtime-request-timeout=5m flag which will increase the timeout for the operation to 5 minutes.

techxplrorer · 2022-11-10T17:12:24Z

@spowelljr Hey! Thanks for your quick response and I followed your steps as below yet it fails to pull the image.

Time taken to pull image

Deleted minikube cluster and increases timeout to 5 mins & 10 mins

Describe Pods

idris@xplorer:~$ kubectl describe pods ingress-nginx-controller-5959f988fd-mqtrp -n ingress-nginx
Name: ingress-nginx-controller-5959f988fd-mqtrp
Namespace: ingress-nginx
Priority: 0
Service Account: ingress-nginx
Node: minikube/192.168.58.2
Start Time: Fri, 11 Nov 2022 00:57:09 +0800
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
gcp-auth-skip-secret=true
pod-template-hash=5959f988fd
Annotations:
Status: Pending
IP: 172.17.0.3
IPs:
IP: 172.17.0.3
Controlled By: ReplicaSet/ingress-nginx-controller-5959f988fd
Containers:
controller:
Container ID:
Image: k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 80/TCP, 443/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--watch-ingress-without-class=true
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
--udp-services-configmap=$(POD_NAMESPACE)/udp-services
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-5959f988fd-mqtrp (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8qvz9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-8qvz9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
minikube.k8s.io/primary=true
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 8m3s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-5959f988fd-mqtrp to minikube
Warning FailedMount 7m31s (x7 over 8m3s) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
Normal Pulling 2m24s (x3 over 6m58s) kubelet Pulling image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8"
Warning Failed 25s (x3 over 4m59s) kubelet Failed to pull image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8": rpc error: code = Unknown desc = context deadline exceeded
Warning Failed 25s (x3 over 4m59s) kubelet Error: ErrImagePull
Normal BackOff 13s (x3 over 4m59s) kubelet Back-off pulling image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8"
Warning Failed 13s (x3 over 4m59s) kubelet Error: ImagePullBackOff

marcoferri72 · 2022-11-18T12:13:49Z

As workaround I resolved pre-dowloading problematic image direcly on minikube machine via docker.
First connect to minikube container:
docker exec -it minikube /bin/bash or simply minikube ssh
now download docker image:
docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8
and then retry to enable ingress addons
minikube addons enable ingress

hn-tma · 2022-11-23T05:05:54Z

As workaround I resolved pre-dowloading problematic image direcly on minikube machine via docker. First connect to minikube container: docker exec -it minikube /bin/bash or simply minikube ssh now download docker image: docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 and then retry to enable ingress addons minikube addons enable ingress

I come here from the similar issue 13841. The aforementioned fix does not work for me. I am using minikube 1.28.0 and kubectl version v1.25.4/v1.25.3 (client/server).

k8s-triage-robot · 2022-12-23T05:28:19Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot · 2022-12-23T05:28:24Z

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ClownWang-1217 · 2023-03-13T08:40:20Z

cause I used minikube, what I got error log like below:

the final solution:
minikube delete
export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24
minikube start
then:
minikube addons enable ingress

💡 ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
💡 After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343
▪ Using image registry.k8s.io/ingress-nginx/controller:v1.5.1
▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343
🔎 Verifying ingress addon...
🌟 The 'ingress' addon is enabled
hope it can help you!

fahimjason · 2023-03-14T11:38:53Z

cause I used minikube, what I got error log like below:

the final solution: minikube delete export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24 minikube start then: minikube addons enable ingress

bulb ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS bulb After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1" black_small_square Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 black_small_square Using image registry.k8s.io/ingress-nginx/controller:v1.5.1 black_small_square Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 mag_right Verifying ingress addon... star2 The 'ingress' addon is enabled hope it can help you!

Thank you so much @ClownWang-1217. It worked for me.

Jen-apjen · 2023-05-04T13:37:22Z

Still issue persists. I am using in offline environment in redhat 8.6.
Can someone suggest working minikube and kubernetes version?

rhighs · 2023-05-06T19:18:19Z

The error persists still to this day, I'm using minikube v1.29.0

saphemmy · 2023-07-21T11:11:14Z

My version of minikube:
minikube version
minikube version: v1.29.0
This solved it for me: because minikube already had some loaded profiles within its clusters.
minikube delete --all
minikube start
minikube addons enable ingress
minikube addons list

fozouni · 2023-08-05T09:25:11Z

My OS is a Windows and I had the same issue. This is what I've done to get rid of this problem:

1- Run Docker desktop.
2- docker pull nginx/nginx-ingress:latest
3- minikube image load nginx/nginx-ingress:latest
4- minikube addons enable ingress
Done!
This worked for me.

mnguyen081002 · 2023-10-06T16:48:47Z

My version of minikube: minikube version minikube version: v1.29.0 This solved it for me: because minikube already had some loaded profiles within its clusters. minikube delete --all minikube start minikube addons enable ingress minikube addons list

This worked for me !

KaiJiaBrother · 2023-11-23T09:50:12Z

You can try this: #10544 (comment)

rosscreighton · 2024-07-10T23:36:21Z

cause I used minikube, what I got error log like below:

the final solution: minikube delete export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24 minikube start then: minikube addons enable ingress

💡 ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS 💡 After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1" ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.5.1 ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 🔎 Verifying ingress addon... 🌟 The 'ingress' addon is enabled hope it can help you!

@ClownWang-1217 why does this work?

Rany-yilian · 2024-08-11T10:58:46Z

My version of minikube: minikube version minikube version: v1.29.0 This solved it for me: because minikube already had some loaded profiles within its clusters. minikube delete --all minikube start minikube addons enable ingress minikube addons list

this is worked for me,too

sahilgupta88 · 2024-10-13T15:09:39Z

My version of minikube: minikube version minikube version: v1.29.0 This solved it for me: because minikube already had some loaded profiles within its clusters. minikube delete --all minikube start minikube addons enable ingress minikube addons list

Thanks! This worked for me.

HYChou0515 · 2024-11-23T05:49:05Z

I have a similar error message.

I have to start minikube using conternaind (as described here)

minikube start --container-runtime containerd

spowelljr added kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it. addon/ingress labels Mar 28, 2022

kubernetes deleted a comment Apr 4, 2022

spowelljr removed the triage/needs-information Indicates an issue needs more information in order to work on it. label Apr 20, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 8, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 27, 2022

hn-tma mentioned this issue Nov 23, 2022

Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition] #13841

Closed

k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 23, 2022

kundan2707 mentioned this issue Jan 30, 2023

not able to set up ingress on minikube #15737

Closed

rhighs mentioned this issue May 6, 2023

Ingress enable issue #16456

Closed

mnguyen081002 mentioned this issue Oct 6, 2023

minikube addons enable ingress fails #17372

Closed

T-Lakshmi mentioned this issue Mar 13, 2024

Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: metric-server is not a valid addon #18363

Closed

Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition] #13872

Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition] #13872

Comments

Orygeunik commented Mar 28, 2022 • edited Loading

What Happened?

Attach the log file

Operating System

Driver

spowelljr commented Mar 28, 2022 • edited Loading

Orygeunik commented Mar 28, 2022 • edited Loading

Orygeunik commented Mar 28, 2022

Orygeunik commented Apr 1, 2022

spowelljr commented Apr 20, 2022 • edited Loading

Orygeunik commented Apr 21, 2022

16871560 commented May 9, 2022

16871560 commented May 9, 2022

Orygeunik commented May 9, 2022

16871560 commented May 10, 2022

Orygeunik commented May 10, 2022

Orygeunik commented May 10, 2022 • edited Loading

k8s-triage-robot commented Aug 8, 2022

peachestao commented Aug 28, 2022 • edited Loading

k8s-triage-robot commented Sep 27, 2022

oluranticode commented Oct 4, 2022

klaases commented Oct 10, 2022

manan commented Oct 29, 2022

techxplrorer commented Nov 9, 2022

spowelljr commented Nov 9, 2022 • edited Loading

techxplrorer commented Nov 10, 2022

marcoferri72 commented Nov 18, 2022 • edited Loading

hn-tma commented Nov 23, 2022

k8s-triage-robot commented Dec 23, 2022

k8s-ci-robot commented Dec 23, 2022

ClownWang-1217 commented Mar 13, 2023

fahimjason commented Mar 14, 2023

Jen-apjen commented May 4, 2023

rhighs commented May 6, 2023

saphemmy commented Jul 21, 2023

fozouni commented Aug 5, 2023 • edited Loading

mnguyen081002 commented Oct 6, 2023

KaiJiaBrother commented Nov 23, 2023

rosscreighton commented Jul 10, 2024

Rany-yilian commented Aug 11, 2024

sahilgupta88 commented Oct 13, 2024

HYChou0515 commented Nov 23, 2024 • edited Loading

Orygeunik commented Mar 28, 2022 •

edited

Loading

spowelljr commented Mar 28, 2022 •

edited

Loading

Orygeunik commented Mar 28, 2022 •

edited

Loading

spowelljr commented Apr 20, 2022 •

edited

Loading

Orygeunik commented May 10, 2022 •

edited

Loading

peachestao commented Aug 28, 2022 •

edited

Loading

spowelljr commented Nov 9, 2022 •

edited

Loading

marcoferri72 commented Nov 18, 2022 •

edited

Loading

fozouni commented Aug 5, 2023 •

edited

Loading

HYChou0515 commented Nov 23, 2024 •

edited

Loading