Minikube gcp-auth credentials empty in Google Cloud Shell

[j-windsor]

Steps to reproduce the issue:

1. Open up a Google Cloud Shell. Minikube ships with Cloud Shell.
2. Run minikube addons enable gcp-auth --alsologtostderr. Note that nothing is written to the /var/lib/minikube/google_application_credentials.json file
   From the logs:

I1109 22:00:18.412722   77011 ssh_runner.go:215] scp memory --> /var/lib/minikube/google_application_credentials.json (0 bytes)

1. Run minikube ssh and then stat /var/lib/minikube/google_application_credentials.json to see that the file is in fact empty.
   From the logs:

File: /var/lib/minikube/google_application_credentials.json
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 801h/2049d      Inode: 2159593     Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-11-09 22:00:18.592828615 +0000
Modify: 2020-11-09 22:00:18.592828615 +0000
Change: 2020-11-09 22:00:18.592828615 +0000
 Birth: -

It looks like google.FindDefaultCredentials() returns Credentials.JSON, which "may be nil if authentication is provided by the environment and not with a credentials file, e.g. when code is running on Google Cloud Platform."

Full output of failed command:

jaywindsor@cloudshell:~$ minikube addons enable gcp-auth --alsologtostderr
I1109 22:00:18.352080   77011 addons.go:55] Setting gcp-auth=true in profile "minikube"
I1109 22:00:18.352116   77011 mustload.go:66] Loading cluster: minikube
I1109 22:00:18.352968   77011 cli_runner.go:110] Run: docker container inspect minikube --format={{.State.Status}}
I1109 22:00:18.409982   77011 host.go:66] Checking if "minikube" exists ...
I1109 22:00:18.412722   77011 ssh_runner.go:215] scp memory --> /var/lib/minikube/google_application_credentials.json (0 bytes)
W1109 22:00:18.412787   77011 ssh_runner.go:217] 0 byte asset: &{BaseAsset:{SourcePath:memory TargetDir:/var/lib/minikube TargetName:google_application_credentials.json Permissions:0444 Source:} reader:0xc0004e2f30 length:0}
I1109 22:00:18.413351   77011 cli_runner.go:110] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I1109 22:00:18.476548   77011 sshutil.go:45] new ssh client: &{IP:127.0.0.1 Port:32771 SSHKeyPath:/google/minikube/.minikube/machines/minikube/id_rsa Username:docker}
W1109 22:00:18.580375   77011 ssh_runner.go:246] asked to copy a 0 byte asset: &{BaseAsset:{SourcePath:memory TargetDir:/var/lib/minikube TargetName:google_application_credentials.json Permissions:0444 Source:} reader:0xc0004e2f30 length:0}
W1109 22:00:19.158648   77011 out.go:146] ! Could not determine a Google Cloud project, which might be ok.
! Could not determine a Google Cloud project, which might be ok.
I1109 22:00:19.165163   77011 out.go:110] * To set your Google Cloud project,  run:
                gcloud config set project <project name>
or set the GOOGLE_CLOUD_PROJECT environment variable.
* To set your Google Cloud project,  run:

                gcloud config set project <project name>
or set the GOOGLE_CLOUD_PROJECT environment variable.
I1109 22:00:19.165221   77011 ssh_runner.go:215] scp memory --> /var/lib/minikube/google_cloud_project (0 bytes)
W1109 22:00:19.165239   77011 ssh_runner.go:217] 0 byte asset: &{BaseAsset:{SourcePath:memory TargetDir:/var/lib/minikube TargetName:google_cloud_project Permissions:0444 Source:} reader:0xc0004e3f20 length:0}
W1109 22:00:19.166860   77011 ssh_runner.go:246] asked to copy a 0 byte asset: &{BaseAsset:{SourcePath:memory TargetDir:/var/lib/minikube TargetName:google_cloud_project Permissions:0444 Source:} reader:0xc0004e3f20 length:0}
I1109 22:00:19.182905   77011 addons.go:131] Setting addon gcp-auth=true in "minikube"
I1109 22:00:19.182960   77011 host.go:66] Checking if "minikube" exists ...
I1109 22:00:19.183481   77011 cli_runner.go:110] Run: docker container inspect minikube --format={{.State.Status}}
I1109 22:00:19.238316   77011 addons.go:243] installing /etc/kubernetes/addons/gcp-auth-ns.yaml
I1109 22:00:19.238357   77011 ssh_runner.go:215] scp deploy/addons/gcp-auth/gcp-auth-ns.yaml --> /etc/kubernetes/addons/gcp-auth-ns.yaml (700 bytes)
I1109 22:00:19.238460   77011 cli_runner.go:110] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I1109 22:00:19.294285   77011 sshutil.go:45] new ssh client: &{IP:127.0.0.1 Port:32771 SSHKeyPath:/google/minikube/.minikube/machines/minikube/id_rsa Username:docker}
I1109 22:00:19.400219   77011 addons.go:243] installing /etc/kubernetes/addons/gcp-auth-service.yaml
I1109 22:00:19.400262   77011 ssh_runner.go:215] scp deploy/addons/gcp-auth/gcp-auth-service.yaml --> /etc/kubernetes/addons/gcp-auth-service.yaml (182 bytes)
I1109 22:00:19.417096   77011 addons.go:243] installing /etc/kubernetes/addons/gcp-auth-webhook.yaml
I1109 22:00:19.417136   77011 ssh_runner.go:215] scp deploy/addons/gcp-auth/gcp-auth-webhook.yaml --> /etc/kubernetes/addons/gcp-auth-webhook.yaml (3613 bytes)
I1109 22:00:19.433719   77011 ssh_runner.go:148] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.19.2/kubectl apply -f /etc/kubernetes/addons/gcp-auth-ns.yaml -f /etc/kubernetes/addons/gcp-auth-service.yaml -f /e
tc/kubernetes/addons/gcp-auth-webhook.yaml
I1109 22:00:20.101816   77011 addons.go:342] Verifying addon gcp-auth=true in "minikube"
I1109 22:00:20.110501   77011 out.go:110] * Verifying gcp-auth addon...
* Verifying gcp-auth addon...
I1109 22:00:20.119518   77011 kapi.go:75] Waiting for pod with label "kubernetes.io/minikube-addons=gcp-auth" in ns "gcp-auth" ...
I1109 22:00:20.145085   77011 kapi.go:86] Found 1 Pods for label selector kubernetes.io/minikube-addons=gcp-auth
I1109 22:00:20.145121   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:20.699427   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:21.152341   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:21.708350   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:22.158077   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:22.694305   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:23.154563   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:23.649554   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:24.148969   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:24.679646   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:25.300431   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:26.430428   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:27.139196   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:27.453708   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:27.722313   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:28.221812   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:28.682791   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:29.149287   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:29.650353   77011 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [<nil>]
I1109 22:00:30.318263   77011 kapi.go:108] duration metric: took 10.198739974s to wait for kubernetes.io/minikube-addons=gcp-auth ...
I1109 22:00:30.322624   77011 out.go:110] * Your GCP credentials will now be mounted into every pod created in the minikube cluster.
* Your GCP credentials will now be mounted into every pod created in the minikube cluster.
I1109 22:00:30.325879   77011 out.go:110] * If you don't want your credentials mounted into a specific pod, add a label with the `gcp-auth-skip-secret` key to your pod configuration.
* If you don't want your credentials mounted into a specific pod, add a label with the `gcp-auth-skip-secret` key to your pod configuration.
I1109 22:00:30.327038   77011 addons.go:97] Writing out "minikube" config to set gcp-auth=true...
I1109 22:00:30.330468   77011 out.go:110] * The 'gcp-auth' addon is enabled
* The 'gcp-auth' addon is enabled
jaywindsor@cloudshell:~$ minikube ssh
docker@minikube:~$ stat /var/lib/minikube/google_application_credentials.json
  File: /var/lib/minikube/google_application_credentials.json
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 801h/2049d      Inode: 2159593     Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-11-09 22:00:18.592828615 +0000
Modify: 2020-11-09 22:00:18.592828615 +0000
Change: 2020-11-09 22:00:18.592828615 +0000
 Birth: -

[medyagh]

thanks @j-windsor that does seem like a bug !

@sharifelgamal is the expert on this plugin, it appears we could handle the nil and use the env provided creds instead.

[sharifelgamal]

so yeah we definitely didn't account for the situation where credentials.JSON was empty. we'll need to find a workaround for that.

[sivakku]

We are doing release testing for Secrets Mgr in to Cloud Shell. Can this be addressed in next week gcloud release?

[sharifelgamal]

That is the goal, yes.

[sivakku]

@sharifelgamal - Can you update on plan for enabling this one? i was in today's standup heard it is going to be needing thorough testing of minikube in Cloud Shell VM which we haven't planned for it before.

[medyagh]

@sivakku @j-windsor this issue is assigned for for v1.19.0 and will be included in the end of march release