Minikube fails to start with docker driver on macOS 11.1. Error: /var/lib/dpkg: no such file or directory

[colli173]

Steps to reproduce the issue:

1. docker system prune --volumes
2. minikube delete
3. docker start --driver=docker

Full output of failed command:

😄 minikube v1.17.0 on Darwin 11.1
✨ Using the docker driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=1990MB) ...
✋ Stopping node "minikube" ...
🔥 Deleting "minikube" in docker ...
🤦 StartHost failed, but will try again: creating host: create: creating: create kic node: container name "minikube": log: 2021-01-28T21:19:13.799746495Z + mount
2021-01-28T21:19:13.806546956Z + df -h /var
2021-01-28T21:19:13.809660555Z + find /var/lib/dpkg
2021-01-28T21:19:13.822425814Z find: '/var/lib/dpkg': No such file or directory: container exited unexpectedly
🔥 Creating docker container (CPUs=2, Memory=1990MB) ...
😿 Failed to start docker container. Running "minikube delete" may fix it: creating host: create: creating: create kic node: container name "minikube": log: 2021-01-28T21:20:01.961530779Z + mount
2021-01-28T21:20:01.964084926Z + df -h /var
2021-01-28T21:20:01.969789288Z + find /var/lib/dpkg
2021-01-28T21:20:01.971640267Z find: '/var/lib/dpkg': No such file or directory: container exited unexpectedly

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

[afbjorklund]

The usual reason for this was that a race condition has caused an empty /var to get mounted...

Using --preload=false might help ? But it would be interesting to see, if this is something new.

What docker version was this ? (sad that we don't log it)

The previous issue was #8100

[colli173]

Thanks! I'll try with --preload=false and update here. It may be worth noting I've reproduced this issue with Minikube version 1.16.0 and 1.17.1 as well

╰─❯ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.7
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:12:42 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:28 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[afbjorklund]

The previous trick was to run a simple container with a /var volume mount, to make sure docker copied it from the image:

docker run --entrypoint /usr/bin/test -v /var ubuntu:20.04 -d /var/lib

That will create a volume with 4.5 MB of various junk that also happens to be in the /var directory of the ubuntu image...

backups/ cache/ lib/ local/ lock@ log/ mail/ opt/ run@ spool/ tmp/

And then we can go ahead and add the minikube preload data, on top of that.

Instead of replacing the files, that it expects to be present (like /var/lib/dpkg)

[colli173]

@afbjorklund issue seems to persist with --preload=false. I'll take another look at the /var dir image trick you mentioned too

❯ minikube start --preload=false
😄  minikube v1.17.0 on Darwin 11.1
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=1990MB) ...
✋  Stopping node "minikube"  ...
🔥  Deleting "minikube" in docker ...
🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: container name "minikube": log: 2021-01-28T23:09:12.620840500Z  + mount
2021-01-28T23:09:12.622663400Z  + df -h /var
2021-01-28T23:09:12.624084000Z  + find /var/lib/dpkg
2021-01-28T23:09:12.625627800Z  find: '/var/lib/dpkg': No such file or directory: container exited unexpectedly
🔥  Creating docker container (CPUs=2, Memory=1990MB) ...
😿  Failed to start docker container. Running "minikube delete" may fix it: creating host: create: creating: create kic node: container name "minikube": log: 2021-01-28T23:09:58.116230900Z  + mount
2021-01-28T23:09:58.118790900Z  + df -h /var
2021-01-28T23:09:58.120561100Z  + find /var/lib/dpkg
2021-01-28T23:09:58.121964700Z  find: '/var/lib/dpkg': No such file or directory: container exited unexpectedly

💢  The minikube docker container exited unexpectedly.

[colli173]

I've been able to get Minikube 1.17.0 running again by selecting Clean/Purge data from the troubleshoot section of the Docker Desktop app and then running minikube start --driver=docker. For whatever reason that worked when docker system prune --volumes wouldn't. Perhaps it's a more comprehensive purge when run from the app

[olivierACRI]

Hi,
I fall into the same issue but without any luck fixing it properly, so here is my 2 cents comment that could help people.
I was using the profile feature in minikube (-p PROFILE_NAME), doing so, I was able to change the profile and the issue vanished.