kubeadm w/ corp proxy: x509: certificate signed by unknown authority

[code-with-coffee]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Please provide the following details:

Environment: Windows 10 Pro

Minikube version (use minikube version): v0.33.1

• OS (e.g. from /etc/os-release): Windows 10 Pro
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): Hyper-V
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): "Boot2DockerURL": "file://N:/.minikube/cache/iso/minikube-v0.33.1.iso",
• Install tools: minikube start --vm-driver "hyperv" --hyperv-virtual-switch "SJ Virtual Switch" --docker-env HTTP_PROXY=http://host:port \ --docker-env HTTPS_PROXY=http://host:port --v 9999
• Others:

What happened: Minikube failed to start (Does create the VM though)

What you expected to happen: For minikube to start successfully and completely

How to reproduce it (as minimally and precisely as possible):
minikube start --vm-driver "hyperv" --hyperv-virtual-switch "SJ Virtual Switch" --docker-env HTTP_PROXY=http://host:port \ --docker-env HTTPS_PROXY=http://host:port --v 9999

Output of minikube logs (if applicable):

error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.2: output: v1.13.2: Pulling from kube-apiserver
73e3e9d78c61: Pulling fs layer
503f459b2f97: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:177db4b8e93a6a74ab19435edf17111d3ad18a8a4efef728712ea067ea8047c1: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.2: output: v1.13.2: Pulling from kube-controller-manager
73e3e9d78c61: Pulling fs layer
ef3ba03ba5d4: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:b9027a78d94c15a4aba54d45476c6f295c0db8f9dcb6fca34c8beff67d90a374: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.2: output: v1.13.2: Pulling from kube-scheduler
73e3e9d78c61: Pulling fs layer
9346ad146311: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:3193be46e0b3e215877b122052c0c7d3ef0902cf1dd6efaf3db95f37cf697002: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.2: output: v1.13.2: Pulling from kube-proxy
73e3e9d78c61: Pulling fs layer
0c440f353724: Pulling fs layer
9f11bf6a2d3d: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:01cfa56edcfc350d36cea9c2fc857949b36bc69bf69df6901e0fd9be3c826617: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: 3.1: Pulling from pause
67ddbfb20a22: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: 3.2.24: Pulling from etcd
8c5a7da1afbc: Pulling fs layer
0d363128e48e: Pulling fs layer
1ba5e77f0f6e: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:3cab8e1b9802cbe23a2703c2750ac4baa90b049b65e2a9e0a83e9e2c29f0724f: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: 1.2.6: Pulling from coredns
2796eccf0de2: Pulling fs layer
6ad5128a7d32: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:f59dcacceff45b5474d1385cd5f500d0c019ed9ca50ed5b814ac0c5fcec8699e: x509: certificate signed by unknown authority
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

Anything else do we need to know:
I'm able to pull the same images via docker pull command without an issue.
Except these two:
PS C:> docker pull k8s.gcr.io/kube-scheduler:v1.13.2
Error response from daemon: Get https://k8s.gcr.io/v2/: Proxy Authentication Required

PS C:> docker pull k8s.gcr.io/etcd:3.2.24
Error response from daemon: Get https://k8s.gcr.io/v2/: Proxy Authentication Required

[code-with-coffee]

Weirdly, if I try the docker pull multiple times for the last 2 failing ones, (Proxy Authentication Required), they seem to work, and I'm able to get the images.

[sefroberg]

I think to simplify the issue all you have to do is go onto the vm and issue a docker pull for one of the images. In my case I can reproduce this with this command:
'docker pull k8s.gcr.io/kube-apiserver:v1.13.2'

[sefroberg]

It also happens on Centos 7.3 and 7.6

[tstromberg]

I'm pretty sure there is corporate SSL interception happening here, similar to #2739 - but apparently mostly a problem within the VM.

[sefroberg]

I was able to get minikube up and running yesterday afternoon by following the instructions for first creating a folder structure in the 'files' folder inside the ~/.minikube folder.
.minikube/files/etc/ssl/certs
Then I placed the .pem files I needed for my IT's compliance into this folder. Then rebuilt the vm (need to issue minikube delete and then minikube start). I confirmed the newly created vm has the certs in the correct place.

[tstromberg]

@sefroberg - Excellent. That you needed to run minikube delete is definitely a bug though -- you should just be able to start it again.

[tstromberg]

Thank you @sefroberg for mentioning a solution. I've added it to our official documentation here:

https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md

I think this will really help future minikube users.

[mrdulin]

@tstromberg The link is broken. New link: https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/#x509-certificate-signed-by-unknown-authority

Besides, is there any solution if I can't ask the IT department for the appropriate PEM file?

Some command flag like: --skip-verify-certificate?

[adelca]

@mrdulin hi, you dont need to ask IT department for the appropriate PEM file, you can download it yourself with openssl or via browser (more about this below).

In my case (on Macbook pro), none of the suggested solutions worked. What I did was to hit https://storage.googleapis.com on my browser (Chrome) and download the company CA as cer (it should be easy to spot, the name will not be google), then moved it to KeyChain under System certs and enabled "Trust Always". You can do the same operation with any cert that you might thing needs to be added manually to KeyChain. Restart docker and minikube and things starting to work.

To download cert via browser: https://stackoverflow.com/questions/25940396/how-to-export-certificate-from-chrome-on-a-mac/30177897#comment71501487_30177897

To install it on KyeChain:

• open KeyChain Access
• drag cert to System and Certificates
• GetInfo on the cert and enable "Trust Always" under "Trust" section

Restart docker and minikube:

• stop docker desktop and start it
• minikube stop, minikube delete, minikube start

For Ubuntu users (I did this on a VirutalBx Ubuntu VM):

• download crt (not cer)
• execute on terminal: cp non-trusted-ca-certificate.crt /usr/local/share/ca-certificates/
• execute on terminal: update-ca-certificates
• restart docker and minikube

[ct-santoshmandal]

How can I fix this without contacting IT team in Windows 11 ? Any help highly appreciated.

[giuliohome]

Is that working with virtualbox driver or only with docker?

[giuliohome]

yes (edit -- confirmed, even without --embed-certs because it affects only the kubeconfig, which is not relevant in the connection to the local minikube, my problem being the TSL validition of docker hub from minikube os) https://minikube.sigs.k8s.io/docs/handbook/untrusted_certs / (see also #11771 (comment))

😀 see also https://kubernetes.slack.com/archives/C1F5CT6Q1/p1720896508220059?thread_ts=1720714401.537019&cid=C1F5CT6Q1

[giuliohome]

@sidharthhhh your message has nothing to do with this thread, first of all.