minikube unable to recover from empty IP/name: apiServer.certSANs: Invalid value: "":

[tstromberg]

I somehow ended up with this configuration while doing some permissions testing:

{
    "Name": "minikube",
    "KeepContext": false,
    "EmbedCerts": false,
    "MinikubeISO": "",
    "KicBaseImage": "gcr.io/k8s-minikube/kicbase:v0.0.12-snapshot3@sha256:1d687ba53e19dbe5fafe4cc18aa07f269ecc4b7b622f2251b5bf569ddb474e9b",
    "Memory": 2192,
    "CPUs": 2,
    "DiskSize": 20000,
    "VMDriver": "",
    "Driver": "docker",
    "HyperkitVpnKitSock": "",
    "HyperkitVSockPorts": [],
    "DockerEnv": null,
    "ContainerVolumeMounts": null,
    "InsecureRegistry": null,
    "RegistryMirror": null,
    "HostOnlyCIDR": "192.168.99.1/24",
    "HypervVirtualSwitch": "",
    "HypervUseExternalSwitch": false,
    "HypervExternalAdapter": "",
    "KVMNetwork": "default",
    "KVMQemuURI": "qemu:///system",
    "KVMGPU": false,
    "KVMHidden": false,
    "DockerOpt": null,
    "DisableDriverMounts": false,
    "NFSShare": [],
    "NFSSharesRoot": "/nfsshares",
    "UUID": "",
    "NoVTXCheck": false,
    "DNSProxy": false,
    "HostDNSResolver": true,
    "HostOnlyNicType": "virtio",
    "NatNicType": "virtio",
    "KubernetesConfig": {
        "KubernetesVersion": "v1.19.0",
        "ClusterName": "minikube",
        "APIServerName": "minikubeCA",
        "APIServerNames": null,
        "APIServerIPs": null,
        "DNSDomain": "cluster.local",
        "ContainerRuntime": "docker",
        "CRISocket": "",
        "NetworkPlugin": "",
        "FeatureGates": "",
        "ServiceCIDR": "10.96.0.0/12",
        "ImageRepository": "",
        "LoadBalancerStartIP": "",
        "LoadBalancerEndIP": "",
        "ExtraOptions": null,
        "ShouldLoadCachedImages": true,
        "EnableDefaultCNI": false,
        "CNI": "",
        "NodeIP": "",
        "NodePort": 8443,
        "NodeName": ""
    },
    "Nodes": [
        {
            "Name": "",
            "IP": "",
            "Port": 8443,
            "KubernetesVersion": "v1.19.0",
            "ControlPlane": true,
            "Worker": true
        }
    ],
    "Addons": null,
    "VerifyComponents": {
        "apiserver": true,
        "system_pods": true
    },
    "StartHostTimeout": 360000000000
}⏎    

W0903 08:35:21.821720   82752 logs.go:208] No container was found matching "kube-controller-manager"
I0903 08:35:21.821729   82752 logs.go:120] Gathering logs for describe nodes ...
I0903 08:35:21.821737   82752 ssh_runner.go:148] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.19.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0903 08:35:21.911368   82752 logs.go:127] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.19.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.19.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:

stderr:
E0903 15:35:21.920114    1726 cert_rotation.go:168] key failed with : x509: cannot parse IP address of length 0
E0903 15:35:21.925472    1726 cert_rotation.go:168] key failed with : x509: cannot parse IP address of length 0
The connection to the server localhost:8443 was refused - did you specify the right host or port?
 output: 
** stderr ** 
E0903 15:35:21.920114    1726 cert_rotation.go:168] key failed with : x509: cannot parse IP address of length 0
E0903 15:35:21.925472    1726 cert_rotation.go:168] key failed with : x509: cannot parse IP address of length 0
The connection to the server localhost:8443 was refused - did you specify the right host or port?

** /stderr **
I0903 08:35:21.911406   82752 logs.go:120] Gathering logs for Docker ...
I0903 08:35:21.911415   82752 ssh_runner.go:148] Run: /bin/bash -c "sudo journalctl -u docker -n 400"
I0903 08:35:21.931930   82752 logs.go:120] Gathering logs for container status ...
I0903 08:35:21.931953   82752 ssh_runner.go:148] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
^B[I0903 08:35:24.022685   82752 ssh_runner.go:188] Completed: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a": (2.090744647s)
I0903 08:35:24.022852   82752 logs.go:120] Gathering logs for kubelet ...
I0903 08:35:24.022862   82752 ssh_runner.go:148] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0903 08:35:24.046805   82752 logs.go:120] Gathering logs for dmesg ...
I0903 08:35:24.046832   82752 ssh_runner.go:148] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
W0903 08:35:24.063916   82752 out.go:257] Error starting cluster: run: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.19.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 3
stdout:

stderr:
W0903 15:35:21.395984    1633 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
apiServer.certSANs: Invalid value: "": altname is not a valid IP address, DNS label or a DNS label with subdomain wildcards: a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'); a wildcard DNS-1123 subdomain must start with '*.', followed by a valid DNS subdomain, which must consist of lower case alphanumeric characters, '-' or '.' and end with an alphanumeric character (e.g. '*.example.com', regex used for validation is '\*\.[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
To see the stack trace of this error execute with --v=5 or higher
W0903 08:35:24.064041   82752 out.go:145] 

W0903 08:35:24.064136   82752 out.go:145] 💣  Error starting cluster: run: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.19.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 3
stdout:

stderr:
W0903 15:35:21.395984    1633 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
apiServer.certSANs: Invalid value: "": altname is not a valid IP address, DNS label or a DNS label with subdomain wildcards: a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'); a wildcard DNS-1123 subdomain must start with '*.', followed by a valid DNS subdomain, which must consist of lower case alphanumeric characters, '-' or '.' and end with an alphanumeric character (e.g. '*.example.com', regex used for validation is '\*\.[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
To see the stack trace of this error execute with --v=5 or higher

💣  Error starting cluster: run: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.19.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 3
stdout:

stderr:
W0903 15:35:21.395984    1633 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
apiServer.certSANs: Invalid value: "": altname is not a valid IP address, DNS label or a DNS label with subdomain wildcards: a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'); a wildcard DNS-1123 subdomain must start with '*.', followed by a valid DNS subdomain, which must consist of lower case alphanumeric characters, '-' or '.' and end with an alphanumeric character (e.g. '*.example.com', regex used for validation is '\*\.[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
To see the stack trace of this error execute with --v=5 or higher

W0903 08:35:24.064353   82752 out.go:145] 

W0903 08:35:24.064389   82752 out.go:145] 😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
W0903 08:35:24.064435   82752 out.go:145] 👉  https://github.com/kubernetes/minikube/issues/new/choose
👉  https://github.com/kubernetes/minikube/issues/new/choose
I0903 08:35:24.136894   82752 out.go:109] 

W0903 08:35:24.137021   82752 out.go:145] ❌  Exiting due to GUEST_START: run: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.19.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 3
stdout:

stderr:
W0903 15:35:21.395984    1633 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
apiServer.certSANs: Invalid value: "": altname is not a valid IP address, DNS label or a DNS label with subdomain wildcards: a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'); a wildcard DNS-1123 subdomain must start with '*.', followed by a valid DNS subdomain, which must consist of lower case alphanumeric characters, '-' or '.' and end with an alphanumeric character (e.g. '*.example.com', regex used for validation is '\*\.[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
To see the stack trace of this error execute with --v=5 or higher

❌  Exiting due to GUEST_START: run: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.19.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 3
stdout:

stderr:
W0903 15:35:21.395984    1633 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
apiServer.certSANs: Invalid value: "": altname is not a valid IP address, DNS label or a DNS label with subdomain wildcards: a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'); a wildcard DNS-1123 subdomain must start with '*.', followed by a valid DNS subdomain, which must consist of lower case alphanumeric characters, '-' or '.' and end with an alphanumeric character (e.g. '*.example.com', regex used for validation is '\*\.[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
To see the stack trace of this error execute with --v=5 

minikube was not able to recover from it without delete.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.