[BUG] kubelet failed to list services through yurthub

[SataQiu]

What happened:
When I add the edge node to a v1.16.9 kubernetes cluster, I found that the kubelet can not list services from cluster correctly.
The kubelet error message is as follows:

Aug 12 15:38:29 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:29.562688   26785 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: unexpected EOF
Aug 12 15:38:30 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:30.568481   26785 request.go:879] Unexpected error when reading response body: unexpected EOF
Aug 12 15:38:30 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:30.568586   26785 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: unexpected EOF
Aug 12 15:38:31 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:31.574361   26785 request.go:879] Unexpected error when reading response body: unexpected EOF
Aug 12 15:38:31 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:31.574476   26785 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: unexpected EOF
Aug 12 15:38:32 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:32.580338   26785 request.go:879] Unexpected error when reading response body: unexpected EOF
Aug 12 15:38:32 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:32.580449   26785 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: unexpected EOF
Aug 12 15:38:33 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:33.587318   26785 request.go:879] Unexpected error when reading response body: unexpected EOF
Aug 12 15:38:33 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:33.587420   26785 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: unexpected EOF
Aug 12 15:38:34 iZhp39lp9t35eul43z3lhvZ kubelet[26785]: E0812 15:38:34.593343   26785 request.go:879] Unexpected error when reading response body: unexpected EOF

The yurthub error message is as follows:

I0812 07:41:58.826515       1 util.go:215] kubelet list services: /api/v1/services?limit=500&resourceVersion=0 with status code 200, spent 8.766926ms
E0812 07:41:58.826598       1 cache_manager.go:350] failed to decode response in saveOneObject unexpected EOF
E0812 07:41:58.826611       1 remote.go:140] kubelet list services: https://apiserver.cluster.local:6443/api/v1/services?limit=500&resourceVersion=0 response cache ended with error, unexpected EOF
I0812 07:41:59.372844       1 util.go:232] start proxying: get /api/v1/namespaces/kube-system/pods/yoda-agent-bzqw8, in flight requests: 27
I0812 07:41:59.377365       1 util.go:215] kubelet get pods: /api/v1/namespaces/kube-system/pods/yoda-agent-bzqw8 with status code 200, spent 4.4546ms
I0812 07:41:59.378131       1 util.go:232] start proxying: patch /api/v1/namespaces/kube-system/pods/yoda-agent-bzqw8/status, in flight requests: 27
I0812 07:41:59.389110       1 util.go:215] kubelet patch pods: /api/v1/namespaces/kube-system/pods/yoda-agent-bzqw8/status with status code 200, spent 10.948942ms
I0812 07:41:59.389383       1 storage.go:464] key(kubelet/pods/kube-system/yoda-agent-bzqw8) storage is pending, just skip it
I0812 07:41:59.389573       1 storage.go:464] key(kubelet/pods/kube-system/yoda-agent-bzqw8) storage is pending, just skip it
I0812 07:41:59.389583       1 cache_manager.go:468] skip to cache object because key(kubelet/pods/kube-system/yoda-agent-bzqw8) is under processing
I0812 07:41:59.390200       1 cache_manager.go:323] pod(kubelet/pods/kube-system/yoda-agent-bzqw8) is MODIFIED
I0812 07:41:59.827531       1 util.go:232] start proxying: get /api/v1/services?limit=500&resourceVersion=0, in flight requests: 27
I0812 07:41:59.832239       1 handler.go:83] mutate master service into ClusterIP:Port=apiserver.cluster.local:6443 for request kubelet list services: https://apiserver.cluster.local:6443/api/v1/services?limit=500&resourceVersion=0
I0812 07:41:59.832682       1 util.go:215] kubelet list services: /api/v1/services?limit=500&resourceVersion=0 with status code 200, spent 5.073414ms
E0812 07:41:59.832750       1 cache_manager.go:350] failed to decode response in saveOneObject unexpected EOF
E0812 07:41:59.832769       1 remote.go:140] kubelet list services: https://apiserver.cluster.local:6443/api/v1/services?limit=500&resourceVersion=0 response cache ended with error, unexpected EOF
I0812 07:42:00.386406       1 util.go:232] start proxying: get /api/v1/namespaces/kube-system/pods/node-local-dns-cx2vg, in flight requests: 27
I0812 07:42:00.390419       1 util.go:215] kubelet get pods: /api/v1/namespaces/kube-system/pods/node-local-dns-cx2vg with status code 200, spent 3.929759ms
I0812 07:42:00.403884       1 util.go:232] start proxying: patch /api/v1/namespaces/kube-system/pods/node-local-dns-cx2vg/status, in flight requests: 27
I0812 07:42:00.413630       1 util.go:215] kubelet patch pods: /api/v1/namespaces/kube-system/pods/node-local-dns-cx2vg/status with status code 200, spent 9.711015ms
I0812 07:42:00.413874       1 storage.go:464] key(kubelet/pods/kube-system/node-local-dns-cx2vg) storage is pending, just skip it
I0812 07:42:00.414015       1 storage.go:464] key(kubelet/pods/kube-system/node-local-dns-cx2vg) storage is pending, just skip it
I0812 07:42:00.414024       1 cache_manager.go:468] skip to cache object because key(kubelet/pods/kube-system/node-local-dns-cx2vg) is under processing
I0812 07:42:00.414520       1 util.go:232] start proxying: get /api/v1/namespaces/kube-system/pods/yoda-agent-bzqw8, in flight requests: 27
I0812 07:42:00.415149       1 cache_manager.go:323] pod(kubelet/pods/kube-system/node-local-dns-cx2vg) is MODIFIED
I0812 07:42:00.418268       1 util.go:215] kubelet get pods: /api/v1/namespaces/kube-system/pods/yoda-agent-bzqw8 with status code 200, spent 3.719903ms
I0812 07:42:00.419072       1 util.go:232] start proxying: patch /api/v1/namespaces/kube-system/pods/yoda-agent-bzqw8/status, in flight requests: 27
I0812 07:42:00.428505       1 util.go:215] kubelet patch pods: /api/v1/namespaces/kube-system/pods/yoda-agent-bzqw8/status with status code 200, spent 9.407982ms
I0812 07:42:00.429538       1 storage.go:464] key(kubelet/pods/kube-system/yoda-agent-bzqw8) storage is pending, just skip it
I0812 07:42:00.429550       1 cache_manager.go:323] pod(kubelet/pods/kube-system/yoda-agent-bzqw8) is MODIFIED
I0812 07:42:00.429556       1 cache_manager.go:327] skip to cache watch event because key(kubelet/pods/kube-system/yoda-agent-bzqw8) is under processing
I0812 07:42:00.833349       1 util.go:232] start proxying: get /api/v1/services?limit=500&resourceVersion=0, in flight requests: 27
I0812 07:42:00.837775       1 handler.go:83] mutate master service into ClusterIP:Port=apiserver.cluster.local:6443 for request kubelet list services: https://apiserver.cluster.local:6443/api/v1/services?limit=500&resourceVersion=0

What you expected to happen:

kubelet can list services through yurthub correctly.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• OpenYurt version: v0.4.1
• Kubernetes version (use kubectl version): v1.16.9
• OS (e.g: cat /etc/os-release): centos 7
• Kernel (e.g. uname -a): Linux iZhp3eu6w356xiwkyll32mZ 3.10.0-957.21.3.el7.x86_64 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

others

It seems that this problem does not appear in the higher version of Kubernetes (such as v1.19.1).

/kind bug

[rambohe-ch]

@SataQiu Thank you for filing the issue.
At first, this issue is known bug, you can add the right rbac settings for system:nodes group to avoid this error. because yurthub has used system:nodes rights to proxy requests to kube-apiserver.

And maybe yurthub can resolve the token from client request Authorization: Bearer xxx header and use the token to reconstruct yurthub client to access kube-apiserver. @SataQiu how about assign you to fix this bug?

[SataQiu]

@rambohe-ch
This looks like a problem with the resource filter.
If I set --enable-resource-filter=false, kubelet then can list services correctly.

[rambohe-ch]

@rambohe-ch
This looks like a problem with the resource filter.
If I set --enable-resource-filter=false, kubelet then can list services correctly.

@SataQiu Thank you for your feedback.
you get the correct reason. and resource filter can be enabled above kubernetes v1.18.