[feature request] Problems for Kubernetes Conformance

[Congrool]

I ran the kubernetes conformance tests and found some problems need to be solved.
Version Info:
OpenYurt v0.5.0 commit: 754f4c2
Kubernetes v1.20.7

OpenYurt Components Involved:
Yurt-Hub
Yurt-Tunnel
Yurt-Controller-Manager

Test Tool:
sonobuoy v0.52.0

The conformance test results is as follows:

Plugin: e2e
Status: failed
Total: 5668
Passed: 0
Failed: 27
Skipped: 5641

Failed tests:
[sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Conformance]
[k8s.io] Pods should support retrieving logs from the container over websockets [NodeConformance] [Conformance]
[sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny attaching pod [Conformance]
[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Burst scaling should run to completion even with unhealthy pods [Slow] [Conformance]
[sig-network] DNS should support configurable pod DNS nameservers [Conformance]
[sig-auth] ServiceAccounts should mount an API token into pods  [Conformance]
[sig-network] Services should have session affinity timeout work for NodePort service [LinuxOnly] [Conformance]
[k8s.io] Variable Expansion should succeed in writing subpaths in container [sig-storage][Slow] [Conformance]
[k8s.io] Pods should support remote command execution over websockets [NodeConformance] [Conformance]
[sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance]
[sig-storage] EmptyDir volumes pod should support shared volumes between containers [Conformance]
[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance]
[sig-network] Networking Granular Checks: Pods should function for intra-pod communication: udp [NodeConformance] [Conformance]
[sig-network] Services should be able to change the type from ExternalName to NodePort [Conformance]
[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Scaling should happen in predictable order and halt if any stateful pod is unhealthy [Slow] [Conformance]
[sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance]
[sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [Conformance]
[sig-network] Networking Granular Checks: Pods should function for node-pod communication: udp [LinuxOnly] [NodeConformance] [Conformance]
[sig-network] Services should have session affinity timeout work for service with type clusterIP [LinuxOnly] [Conformance]
[k8s.io] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file [LinuxOnly] [NodeConformance] [Conformance]
[sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance]
[sig-network] Networking Granular Checks: Pods should function for intra-pod communication: http [NodeConformance] [Conformance]
[sig-network] Services should be able to create a functioning NodePort service [Conformance]
[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance]
[sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance]
[sig-network] Networking Granular Checks: Pods should function for node-pod communication: http [LinuxOnly] [NodeConformance] [Conformance]

Plugin: systemd-logs
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0

After figuring out the fail reasons, there are 3 problems need to be solved to pass the kubernetes conformance tests.

1. Yurthub should have the capability of handling kubectl exec request from one pod to another pod.

Background
sonobuoy will setup a pod sonobuoy-e2e-job to actually run e2e tests. Some e2e tests need to use kubectl exec to run command into another pod.

Logs
First, enter into a pod.
Then try to run kubectl exec.
And we will get logs:

root@sonobuoy-e2e-job-ef4fe437a3fc4b48:/usr/local/bin# kubectl exec -it execpod-affinityc74g7 -ndefault -- bash
Error from server (BadRequest): no accept content type is set.

Details (with -v=10):

I1106 09:30:00.271111    1382 round_trippers.go:425] curl -k -v -XPOST  -H "User-Agent: kubectl/v1.20.7 (linux/amd64) kubernetes/132a687" -H "Authorization: Bearer $TOKEN" -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" 'https://169.254.2.1:10268/api/v1/namespaces/default/pods/execpod-affinityc74g7/exec?command=bash&container=agnhost-container&stdin=true&stdout=true&tty=true'
I1106 09:30:00.273566    1382 round_trippers.go:445] POST https://169.254.2.1:10268/api/v1/namespaces/default/pods/execpod-affinityc74g7/exec?command=bash&container=agnhost-container&stdin=true&stdout=true&tty=true 400 Bad Request in 2 milliseconds
I1106 09:30:00.273581    1382 round_trippers.go:451] Response Headers:
I1106 09:30:00.273596    1382 round_trippers.go:454]     Content-Type: application/json
I1106 09:30:00.273600    1382 round_trippers.go:454]     Date: Sat, 06 Nov 2021 09:30:00 GMT
I1106 09:30:00.273605    1382 round_trippers.go:454]     Content-Length: 145
I1106 09:30:00.274335    1382 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "no accept content type is set.",
  "reason": "BadRequest",
  "code": 400
}]
F1106 09:30:00.274394    1382 helpers.go:115] Error from server (BadRequest): no accept content type is set

Cause
From the log we can find that the pod sonobuoy-e2e-job-ef4fe437a3fc4b48 will send exec POST http request to yurthub. However, yurthub seems to have no capability to handle the exec POST request.

2. Yurthub should have the capability of handling websocket request.

Background
Some e2e tests need to setup websocket connection to fetch logs of pods.

Logs

FAIL: Failed to open websocket to wss://169.254.2.1:10268/api/v1/namespaces/pods-3167/pods/pod-logs-websocket-37afbc79-8b43-4d82-9d83-df89928fe5f3/log?container=main: websocket.Dial wss://169.254.2.1:10268/api/v1/namespaces/pods-3167/pods/pod-logs-websocket-37afbc79-8b43-4d82-9d83-df89928fe5f3/log?container=main: bad status

others

However, openyurt v0.4.0 has passed the conformance test. I tried to restart yurt-hub with options --access-server-through-hub=false --enable-dummy-if=false and rerun the failed cases. Finally, all failed cases passed.

[root@master tmp]# sonobuoy results $(sonobuoy retrieve)
Plugin: e2e
Status: passed
Total: 5668
Passed: 27
Failed: 0
Skipped: 5641

Plugin: systemd-logs
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0

Thus, we should figure out how does both two options affect the test results.

/kind feature

[rambohe-ch]

@Congrool Thank you for raising issue.

1. Yurthub should have the capability of handling kubectl exec request from one pod to another pod.

The problem maybe have solved by this pr: #560

2. Yurthub should have the capability of handling websocket request.

Yes, Yurthub does not support websocket at present, Would you like to make yurthub to support websocket?

I think in later version of OpenYurt, we need to enable these two parameters for yurthub: --access-server-through-hub=true --enable-dummy-if=true. and the function of these parameters as following:

• access-server-through-hub: Pods on edge node access kube-apiserver through yurthub or not, if true, go through yurthub, else do not go through.
• enable-dummy-if: yurthub will create a dummy type network interface on node or not. and network interface named yurt-hub-dummy0. The function is that pods(not HostNetwork) will access yurthub by this dummy network interface.

[Congrool]

@rambohe-ch Thank you for your reply.

2. Yurthub should have the capability of handling websocket request.

I'd like to take over this work.

[Congrool]

/assign