[BUG] cannot access Kubernetes dashboard after upgrading to 1.17.7

[przemyslavic]

Describe the bug
From time to time on some cluster you can't get to the dashboard by running kubectl proxy.
The issue was noticed after Kubernetes upgrade to v1.17.7 on AWS RedHat environment.

To Reproduce
Steps to reproduce the behavior:

1. Deploy AWS RHEL cluster
2. Run kubectl proxy on master node
3. Try to run curl -I http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Expected behavior
The dashboard UI is available.
HTTP status code is 200

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: no-cache, private
Cache-Control: no-store
Content-Type: text/html; charset=utf-8
Date: Mon, 29 Jun 2020 13:18:29 GMT
Last-Modified: Fri, 06 Dec 2019 15:14:02 GMT

OS (please complete the following information):

• OS: [RHEL 7.8]

Cloud Environment (please complete the following information):

• Cloud Provider [AWS]

Additional context
Curl command output:

HTTP/1.1 503 Service Unavailable
Cache-Control: no-cache, private
Content-Length: 71
Content-Type: text/plain; charset=utf-8
Date: Mon, 29 Jun 2020 13:47:29 GMT
X-Content-Type-Options: nosniff

curl "http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/"
Error trying to reach service: 'dial tcp 10.244.3.15:8443: i/o timeout'

[ec2-user@ec2 ~]$ kubectl logs -n=kubernetes-dashboard kubernetes-dashboard-5d996f7d46-6tthp
2020/06/29 10:00:20 Starting overwatch
2020/06/29 10:00:20 Using namespace: kubernetes-dashboard
2020/06/29 10:00:20 Using in-cluster config to connect to apiserver
2020/06/29 10:00:20 Using secret token for csrf signing
2020/06/29 10:00:20 Initializing csrf token from kubernetes-dashboard-csrf secret
2020/06/29 10:00:20 Successful initial request to the apiserver, version: v1.17.7
2020/06/29 10:00:20 Generating JWE encryption key
2020/06/29 10:00:20 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting
2020/06/29 10:00:20 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard
2020/06/29 10:00:20 Initializing JWE encryption key from synchronized object
2020/06/29 10:00:20 Creating in-cluster Sidecar client
2020/06/29 10:00:20 Auto-generating certificates
2020/06/29 10:00:20 Successfully created certificates
2020/06/29 10:00:20 Serving securely on HTTPS port: 8443
2020/06/29 10:00:50 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.

[rafzei]

Looks like it related to < v0.6.0 version of epicli. I cannot reproduce the issue by upgrading v0.6.0 to v.0.7.0.
ENV:

• RHEL 7.8
• AWS

[toszo]

Looks like a non-deterministic error. It appears usually on clusters built by pipeline.

[rafzei]

Additional log from kubernetes-metrics-scraper:

I0707 14:37:30.561304    4967 round_trippers.go:420] GET https://10.1.2.197:6443/api/v1/namespaces/kubernetes-dashboard/pods/kubernetes-metrics-scraper-756dd959c8-xz2vt/log
I0707 14:37:30.561316    4967 round_trippers.go:427] Request Headers:
I0707 14:37:30.561321    4967 round_trippers.go:431]     User-Agent: kubectl/v1.17.7 (linux/amd64) kubernetes/b445510
I0707 14:37:30.561325    4967 round_trippers.go:431]     Accept: application/json, */*
I0707 14:37:30.564051    4967 round_trippers.go:446] Response Status: 200 OK in 2 milliseconds
{"level":"info","msg":"Kubernetes host: https://10.96.0.1:443","time":"2020-07-07T14:35:56Z"}
10.244.1.1 - - [07/Jul/2020:14:36:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.17"
10.244.1.1 - - [07/Jul/2020:14:36:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.17"
10.244.1.1 - - [07/Jul/2020:14:36:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.17"
{"level":"error","msg":"Error scraping node metrics: the server could not find the requested resource (get nodes.metrics.k8s.io)","time":"2020-07-07T14:36:56Z"}

What I observe is that issue was gone after VM restart.

[rafzei]

After upgrade K8s to 1.18.6 issue is still present on Flannel & Canal

[rafzei]

I've removed dependency due to new findings on this topic. The Dashboard working well within the Pod. Looks like it's related to manage flannel.1 interface by NetworkManager. The issue seems to be reproducible right now. I'm going to test a fix for that.

[przemyslavic]

The fix seems to have resolved the issue as I am no longer able to reproduce it.