Offline docs do not work for the Docker driver

[ricosega]

Hi there,

Needed to set up minikube in an "air gapped/offline" environment and we were not able to to do it following the instructions. We tried using many flags at start point like, --cache-images=true, or --download-only, and many combinations, but when we moved the .minikube folder to the offline computers, it wasn't able to start.

We also tried using minikube cache load $image with all of the needed images and same result. Finally I tried downloading the gcr.io/k8s-minikube/kicbase, moving it to the offline and starting it with minikube --base-image="privateregistry/minikubeimage and it worked.

Moreover, when you are in an offline environment most probably you will use a private registry so passing the flag --registry-mirror="https://privateregistry is not enough to make it work, because when you apply any definition yaml it doesn't works because of the error: "x509: certificate signed by unknown authority" which you have documented here, and wanted to confirm that there is no need to create the folders ~/.minikube/files/etc/ssl/certs, if you copy your certificate directly to ~/.minikube/certs/ it also works.

Finally, the last step won't work because minikube does not know the DNS resolution to the private registry, so you have to either, inside of the minikube docker container add your private DNS to the /etc/resolv.conf or directly set the private reigistry resolution at the /etc/hosts which can be solved by for example using docker exec -it minikube bash -c "echo '$IP $REGISTRYNAME' >> /etc/hosts. Is there some way to make this persistent across restarts? Would be a new feature to make the flag --dns-proxy also compatible with docker?

[clamprecht01]

@ricosega Thank you for opening this.

Following for updates.

[priyawadhwa]

Hey @ricosega thanks for opening this issue. A few questions:

• What minikube version are you running?
• What OS and driver are you using?

Running minikube start --download-only with a network connection should (in theory) be enough to run minikube start successfully online.

Could you provide the following output when run offline?

minikube start --alsologtostderr

[ricosega]

Hi @priyawadhwa ,

Ubuntu 18.04 and 20.04.
Minikube 1.15 and 1.16
Docker 19.03.13

All this versions have the same behavior.
About what you're saying:

Running minikube start --download-only with a network connection should (in theory) be enough to run minikube start successfully online.

The problem is not when you're online, the command minikube start --download-only works and starts the minikube, but if you take the ~/.minikube folder and you move it to an offline computer it does not work (as it is written in the docu).
You have to follow all the steps as I mentioned above to properly make it work in an isolated environment.

[priyawadhwa]

Hi @ricosega sorry, I meant to say this:

Running minikube start --download-only with a network connection should (in theory) be enough to run minikube start successfully offline.

When you run minikube start --download-only do you have a docker daemon running? If so, then minikube will download the kicbase image to the docker daemon, which would explain why this doesn't work when you move the .minikube directory to an offline computer.

Could you try:

1. Stopping the docker daemon
2. Running minikube start --download-only
3. Moving the .minikube directory to the offline computer
4. Running minikube start on the offline computer

and let me know if that works? Thanks.

[tstromberg]

I agreed that offline usage should in fact work -- we even have an integration test for it.

Offline support was introduced before the Docker driver was implemented, and we haven't yet updated the documentation to reflect one major quirk about the Docker driver: it does not pull the image from the local filesystem, but instead pulls it from the local Docker daemon.

[afbjorklund]

There was a bit of a twist in this story, it was not only "offline" (on the same host) but also on a different host...

But it is mentioned in the documentation:

For offline use on other hosts, one can copy the contents of ~/.minikube/cache

Which sortof implies that everything is cached there.

[afbjorklund]

In the suggested PR #10918, the image is stored in ~/.minikube/cache/kic folder:

354M	~/.minikube/cache/kic/kicbase_v0.0.18@sha256_ddd0c02d289e3a6fb4bba9a94435840666f4eb81484ff3e707b69c1c484aa45e.tar

This means that it will be copied to the other machine, and available offline as well.

When the first node is started, it will get loaded into Docker daemon (or Podman)

[afbjorklund]

Will be in 1.20