darwin: oc certificate errors because of golang bug

[cfergeau]

Because of go issue golang/go#52010, oc on macOS no longer deals correctly with 'certificate untrusted' errors.
This causes issues in 2 places in crc:

1. at the end of crc start:

INFO All operators are available. Ensuring stability... 
INFO Operators are stable (2/3)...                
INFO Operators are stable (3/3)...                
DEBU Cluster took 9m30.479741791s to stabilize    
INFO Adding crc-admin and crc-developer contexts to kubeconfig... 
ERRO Cannot update kubeconfig: x509: “ingress-operator@1669088888” certificate is not trusted 
Started the OpenShift cluster.

2. when using oc login:

% ./oc version
Client Version: 4.12.0-ec.5
Kustomize Version: v4.5.7

% ./oc login -u developer https://api.crc.testing:6443
error: x509: “kube-apiserver-lb-signer” certificate is not trusted

% ~/.crc/bin/oc/oc version
Client Version: 4.10.40

% ~/.crc/bin/oc/oc login -u developer https://api.crc.testing:6443

The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://api.crc.testing:6443 (openshift)
Username: developer
Password: 
Login successful.

You don't have any projects. You can try to create a new project, by running

    oc new-project <projectname>

Issue 2. is currently worked around by using an older oc version (see #3375 and crc-org/snc#578), but this soon will no longer be an option, and I don't know of other workarounds which would allow users to still be able to login to the cluster. This is tracked in openshift/oc#1207 and https://bugzilla.redhat.com/show_bug.cgi?id=2097830

Issue 1. is not impacting interactions with the cluster, it's only the addition of the developer and admin users as contexts in the local kubeconfig file which is failing. This is happening in code we vendor from oc.

[cfergeau]

I can reproduce a similar issue also with odo from https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/odo/v3.3.0/odo-darwin-arm64 or from brew :-/

odo from git built with go 1.19 fails with x509: kube-apiserver-lb-signer certificate is not trusted, the same version built with go 1.17 works fine :(

[cfergeau]

I rebuilt odo with the patch, it helps a bit, but is not enough.

• odo.git built with golang 1.19, no oc patches

% ./odo login -u developer https://api.crc.testing:6443   
Connecting to the OpenShift cluster

 �  x509: �kube-apiserver-lb-signer� certificate is not trusted

• odo.git built with golang 1.19 with the patch from this PR

% ./odo login  -u developer https://api.crc.testing:6443        
Connecting to the OpenShift cluster

The server is using an invalid certificate: x509: certificate has expired or is not yet valid: x509: �kube-apiserver-lb-signer� certificate is not trusted
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

 �  x509: �ingress-operator@1669088888� certificate is not trusted

• odo.git built with go 1.17, no patch

% ./odo login  -u developer https://api.crc.testing:6443   
Connecting to the OpenShift cluster

The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://api.crc.testing:6443 (openshift)
Username: developer
Password: 

[cfergeau]

openshift/oc#1207 (comment) has a workaround which fixes both issues (after rebuilding oc and updating crc's go.mod to make use of it).

[cfergeau]

The go bug causing this issue is fixed in 1.18.10, 1.19.5 and 1.20. Hopefully we'll soon get the fixes in RHEL:
https://bugzilla.redhat.com/show_bug.cgi?id=2167409
https://bugzilla.redhat.com/show_bug.cgi?id=2167412

[timroster]

Ran across this issue today attempting to use the binary provided by the OpenShift Console with CRC version 2.14.0+1a1ef27. FWIW, although it does not adhere to the +/- 1 minor version range (after k8s guidance), I am working around this by using the most recent 4.10 oc version. Supposedly after login, I could switch back to 4.12.1 but have not hit anything requiring that.

[cfergeau]

Hopefully this will be fixed 'soon', fixed golang versions are arriving in RHEL, as soon as there are oc binaries built with these fixed versions, the bug will be gone.

[jwhb]

No need to compile yourself if you use the recently built oc binary from Homebrew: brew install openshift-cli

[cfergeau]

Recent builds of the client from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest-4.12/ are also fixed.
crc binaries are built with go versions containing the fix.
Closing this as this should be fixed now.