Minikube virtualbox vm has docker.service masked

[samarjit]

What Happened?

When I logged into vm and check systemctl status docker and it says docker is masked, and I was not able to unmask it.
I was finally be able to run minikube using containerd minikube start --no-vtx-check --container-runtime=containerd. But I am still wondering why docker did not work in my system.

$ minikube start --no-vtx-check
* minikube v1.26.0 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042
* Automatically selected the virtualbox driver
* Downloading VM boot image ...
    > minikube-v1.26.0-amd64.iso....: 65 B / 65 B [----------] 100.00% ? p/s 0s
    > minikube-v1.26.0-amd64.iso: 268.86 MiB / 268.86 MiB  100.00% 6.76 MiB p/s
* Starting control plane node minikube in cluster minikube
* Downloading Kubernetes v1.24.1 preload ...
    > preloaded-images-k8s-v18-v1...: 405.83 MiB / 405.83 MiB  100.00% 6.67 MiB
* Creating virtualbox VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
* Deleting "minikube" in virtualbox ...
! StartHost failed, but will try again: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Failed to enable unit: Unit file /usr/lib/systemd/system/docker.service is masked.

* Failed to start virtualbox VM. Running "minikube delete" may fix it: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\samarjit.samanta\.minikube\machines\minikube\config.json: The system cannot find the file specified.

X Exiting due to GUEST_NOT_FOUND: Failed to start host: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\samarjit.samanta\.minikube\machines\minikube\config.json: The system cannot find the file specified.
* Suggestion: minikube is missing files relating to your guest environment. This can be fixed by running 'minikube delete'
* Related issue: https://github.com/kubernetes/minikube/issues/9130

Attach the log file

$ minikube start --no-vtx-check
* minikube v1.26.0 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042
* Automatically selected the virtualbox driver
* Downloading VM boot image ...
> minikube-v1.26.0-amd64.iso....: 65 B / 65 B [----------] 100.00% ? p/s 0s
> minikube-v1.26.0-amd64.iso: 268.86 MiB / 268.86 MiB 100.00% 6.76 MiB p/s
* Starting control plane node minikube in cluster minikube
* Downloading Kubernetes v1.24.1 preload ...
> preloaded-images-k8s-v18-v1...: 405.83 MiB / 405.83 MiB 100.00% 6.67 MiB
* Creating virtualbox VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
* Deleting "minikube" in virtualbox ...
! StartHost failed, but will try again: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err : Process exited with status 1
output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Failed to enable unit: Unit file /usr/lib/systemd/system/docker.service is masked.

* Failed to start virtualbox VM. Running "minikube delete" may fix it: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\samarjit.samanta\.minikube\machines\minikube\config.json: The system cannot find the file specified.

X Exiting due to GUEST_NOT_FOUND: Failed to start host: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\samarjit.samanta\.minikube\machines\minikube\config.json: The system cannot find the file specified.
* Suggestion: minikube is missing files relating to your guest environment. This can be fixed by running 'minikube delete'
* Related issue: https://github.com/kubernetes/minikube/issues/9130

I went into the VM in virtualbox and ran the below.

$ sudo systemctl unmask docker.service
$ systemctl status docker
Loaded: masked (Reason: Unit docker.service is masked.)
Active: Inactive (dead)
TriggeredBy: docker.socket

$ systemctl list-unit-files docker.service
UNIT FILE STATE VENDOR PRESET
docker.service masked enabled
1 unit file listed.

$ ls -lrt /lib/systemd/system/docker.service
-rw-r--r-- 1 root:root 0 Jul 18 05:57 /lib/systemd/system/docker.service

---

When I ran as root

minikube start --no-vtx-check --vm-driver=virtualbox --container-runtime=docker

• minikube v1.26.0 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042

• Using the virtualbox driver based on user configuration

• Starting control plane node minikube in cluster minikube

• Creating virtualbox VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...

• Deleting "minikube" in virtualbox ...
  ! StartHost failed, but will try again: creating host: create: provisioning: ssh command error:
  command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
  err : Process exited with status 1
  output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
  Failed to enable unit: Unit file /usr/lib/systemd/system/docker.service is masked.

• Creating virtualbox VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...

• Failed to start virtualbox VM. Running "minikube delete" may fix it: creating host: create: provisioning: ssh command error:
  command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
  err : Process exited with status 1
  output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
  Failed to enable unit: Unit file /usr/lib/systemd/system/docker.service is masked.

X Exiting due to RT_DOCKER_RESTART: Failed to start host: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err : Process exited with status 1
output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Failed to enable unit: Unit file /usr/lib/systemd/system/docker.service is masked.

• Suggestion: Remove the invalid --docker-opt or --insecure-registry flag if one was provided
• Related issue: docker fails if --docker-opt storage-driver=overlay2 is set #7070

---

With an old image from coreos I am getting some other error.
$ minikube start --container-runtime=docker --iso-url=https://github.com/coreos/minikube-iso/releases/download/v0.0.5/minikube-v0.0.5.iso --no-vtx-check

• minikube v1.26.0 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042

• Using the virtualbox driver based on existing profile

• Starting control plane node minikube in cluster minikube

• Creating virtualbox VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...

• Deleting "minikube" in virtualbox ...
  ! StartHost failed, but will try again: creating host: create: provisioning: Temporary Error: transferring file to machine &{{C:\Users\samarjit.samanta.minikube\certs\ca.pem /etc/docker ca.pem 0640 } 0xc0018d5410 0xc000166a20}: sudo test -d /etc/docker && sudo scp -t /etc/docker && sudo touch -d "2022-07-17 11:29:36.5606009 -0500" /etc/docker/ca.pem: Process exited with status 1
  output: ���touch: invalid date '2022-07-17 11:29:36.5606009 -0500'

• Failed to start virtualbox VM. Running "minikube delete" may fix it: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\samarjit.samanta.minikube\machines\minikube\config.json: The system cannot find the file specified.

X Exiting due to GUEST_NOT_FOUND: Failed to start host: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\samarjit.samanta.minikube\machines\minikube\config.json: The system cannot find the file specified.

• Suggestion: minikube is missing files relating to your guest environment. This can be fixed by running 'minikube delete'
• Related issue: StartHost failed, but will try again: Error loading existing host. Please try running [minikube delete], then run [minikube start] again.: filestore "minikube": #9130

Operating System

Windows

Driver

VirtualBox

Update - attached verbose log with --alsologtostderr -v=7
minikube-nonadmin.log

[RA489]

/kind support

[klaases]

Hi @samarjit – is this issue still occurring? Were you able to find a solution?

For additional assistance, please consider reaching out to the minikube community:

https://minikube.sigs.k8s.io/community/

We also offer support through Slack, Groups, and office hours.

/triage needs-information

[samarjit]

I believe the issue might have been with virtualbox and something got corrupted in its networking setup. May be I was not able to properly remove minikube and cleanly setup again.

I am no longer using minikube. I had to quickly run something so I went with KinD. Even later I was able to setup cluster with plain kubeadm on virtualbox using vagrant ubuntu box. K8s land changes so fast, I had a project on github for setting up kubeadm on vagrant centos7 but that didn't work so I adapted that to use vagrant ubuntu.

[klaases]

Hi @samarjit, glad it worked out and thanks for the followup!

[Seikilos]

Behind a proxy and having the same issue. The solution was weird though.

I executed the recommended command minikube start --alsologtostderr -v=7

and noticed in the logs that NO_PROXY variable was evaluated in a strange way. The issue was, Windows uses ; as a delimiter in its proxy settings. I've created an env var called NO_PROXY and pasted the no proxy string but kept the semicolon delimiter. Minikube or linux for that matter requires NO_PROXY to use comma. Replacing semicolons with commas fixed this issue.