Move to use okhttp4 for Kubernetes cluster using IPV6

[ppatierno]

If you are running a Kubernetes cluster with IPV6 based infrastructure, the current cluster operator raises the following exception:

2020-07-29 13:59:15 ERROR PlatformFeaturesAvailability:124 - Detection of Kubernetes version failed.
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[io.fabric8.kubernetes-client-4.6.4.redhat-00001.jar:?]
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) ~[io.fabric8.kubernetes-client-4.6.4.redhat-00001.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.ClusterOperationsImpl.fetchVersion(ClusterOperationsImpl.java:51) ~[io.fabric8.kubernetes-client-4.6.4.redhat-00001.jar:?]
	at io.fabric8.kubernetes.client.DefaultKubernetesClient.getVersion(DefaultKubernetesClient.java:293) ~[io.fabric8.kubernetes-client-4.6.4.redhat-00001.jar:?]
	at io.strimzi.operator.PlatformFeaturesAvailability.lambda$getVersionInfoFromKubernetes$5(PlatformFeaturesAvailability.java:122) ~[io.strimzi.operator-common-0.18.0.redhat-00003.jar:0.18.0.redhat-00003]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:316) ~[io.vertx.vertx-core-3.8.5.redhat-00005.jar:3.8.5.redhat-00005]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.8.5.redhat-00005.jar:3.8.5.redhat-00005]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.48.Final-redhat-00001.jar:4.1.48.Final-redhat-00001]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname fd02::1 not verified:
    certificate: sha256/UhRsjCDunYlJodfpfGs6wkQipr9PHbUslnMU+57aehE=
    DN: CN=fd02::1
    subjectAltNames: [fd02:0:0:0:0:0:0:1, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, openshift, openshift.default, openshift.default.svc, openshift.default.svc.cluster.local, fd02::1]

The problem seems to be related to a bug in the okhttp3 library that is used by the fabric8 Kubernetes client used by Strimzi. The okhttp3 library seems not able to handle IPV6 address correctly when verifying hostname during TLS certificate verification.

For that bug, the okhttp project seems to have the following fix square/okhttp#5889. It was already merged into the master but anyway it's planned to be delivered in okhttp4 library (milestone set to 4.5). While we should evaluate the impact of okhttp moving from 3 to 4 (which anyway means to test fabric8 as well), there is a really easy workaround that doesn't need any kind of update.

The fabric8 client reaches the Kubernetes master API through the KUBERNETES_MASTER env var which is, by default, defined as https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT where, in an IPV6 cluster, we have KUBERNETES_SERVICE_HOST: fd02::1. This is causing the issue in okhttp3 library not able to handle that IPV6 address for reaching the master URL.

The workaround I used and verified is setting the KUBERNETES_MASTER to the actual Kubernetes master on the got through the kubectl cluster-info command.
It could even be one of the IPV4 addresses listed in the subjectAltNames of the logged exception above (i.e. kubernetes.default.svc).

The KUBERNETES_MASTER has to be set in the Deployment YAML file of the cluster operator in the spec.template.spec.containers as env for the pod template.

env:
  - name: KUBERNETES_MASTER
    value: https://kubernetes.default.svc

Another way to work around this is just disabling the hostname verification so setting the KUBERNETES_DISABLE_HOSTNAME_VERIFICATION env var to false (in the cluster operator Deployment YAML file)

env:
  - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION
    value: "true"

Even if a workaround exists we should track okhttp4 release together with fabric8 and if it's going to use this new version, in order to avoid the need of the workaround just described.

[ppatierno]

I just opened a corresponding issue in fabric8 as well. It would be great having that library updating to the okhttp4 version directly.

[sknot-rh]

^ fabric8io/kubernetes-client#2632

[scholzj]

This needs to be fixed in Fabric8 Kubernetes client => closing this issue.