SPDY Connection upgrades with kubernetes API server not working.

[rapatchi]

Describe the bug

I am trying to use YARP as reverse proxy behind a kubernetes cluster. Normal HTTP requests work fine but API's which require connection upgrade are not working(For example: kubectl exec). I tried using IHttpUpgradeFeature as well to make YARP understand if this request is a upgrade request or not. But with that change as well it's not working.

To Reproduce

• You would need to have a kubernetes cluster.

• Get your K8s cluster's CA data and use it as client certificate and point the destination address to k8s api server.

• Deploy nginx on to your k8s cluster by following this doc: https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/#creating-and-exploring-an-nginx-deployment

• Change the api server endpoint in your kubeconfig to yarp proxy server.

• Run kubectl exec command with the pod information form your previously created deployment.

This is the response we get from k8s API server when we use YARP as reverse proxy in between:
error: unable to upgrade connection: unable to upgrade: unable to negotiate protocol: client supports [v4.channel.k8s.io, v3.channel.k8s.io, v2.channel.k8s.io, channel.k8s.io], server accepts [v4.channel.k8s.io v3.channel.k8s.io v2.channel.k8s.io channel.k8s.io]

Further technical details

• Include the version of the packages you are using: Running using ReverseProxy.Code.Sample of latest master as of 10/10/2020
• The platform (Linux/macOS/Windows) Windows

[Tratcher]

Is it using WebSockets and those are the subprotocols? Or is that a completely different protocol?

Can you share the headers as received by YARP/Kestrel? You can capture these in middleware, a network trace, or a kestrel connection trace.

Upgrades are currently limited to WebSocket requests to mitigate #255.

[rapatchi]

Hi Chris,

These are the headers passed:
curl -k -v -XPOST -H "Upgrade: SPDY/3.1" -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "X-Forwarded-For: 127.0.0.1" -H "User-Agent: kubectl.exe/v1.18.3 (windows/amd64) kubernetes/2e7996e" -H "Authorization: Bearer 4462ac733db5548700dbe49652bc30cd00bf92fa66cd771250f88af70907925a672cabe28c8b1324d60870eee4bc9509695e1fa5683f73b17bc90631b8a9320d" -H "Content-Length: 0" -H "Connection: Upgrade" 'https://rapatchi1-rapatchi-1bfbb5-6efbff9d.hcp.eastus.azmk8s.io:443/api/v1/namespaces/default/pods/nginx-deployment-574b87c764-b94vq/exec?command=ls&container=nginx&stderr=true&stdout=true'

[Tratcher]

Ah, Upgrade: SPDY/3.1. Looks like we'll have to do something about #255.

In the meantime the only workaround for you would be to tweak or remove this line:

reverse-proxy/src/ReverseProxy/Service/Proxy/HttpProxy.cs

Lines 101 to 102 in 6311df4

	// Mitigate https://github.com/microsoft/reverse-proxy/issues/255, IIS considers all requests upgradeable.
	&& string.Equals("WebSocket", context.Request.Headers[HeaderNames.Upgrade], StringComparison.OrdinalIgnoreCase);

[rapatchi]

I even tried that as well but after that as well it does not seem to work. I get 403.

[Tratcher]

You'll have to check the logs on your destination server for that, it sounds like an issue with the Bearer token.

[rapatchi]

I used golang's reverse proxy with the same client and token it's working. But with yarp it's not working.

[Tratcher]

The Bearer token is passed through transparently by YARP, there must be another difference then. Logs from the destination server would help. Otherwise, you can capture a network trace of request for both the golang and yarp scenarios for comparison.

[rapatchi]

Update: Issue here seems to be with kubernetes. It is working in latest version(1.19) on kubernetes just by commenting the other line as mentioned above.

[Tratcher]

Note even K8 wants to remove SPDY (kubernetes/kubernetes#7452), they just haven't managed to do it yet. A temporary mitigation like we did for WebSockets should be adequate.

[Tratcher]

Note that since the only thing we're doing special for upgrade requests is to make sure the request version is set to HTTP/1.1, someone could get a similar effect by using a custom request transform (once we have those working more #60).

[rapatchi]

@Tratcher This above is because of kubernetes itself. The fix is now backported to older versions of kubernetes. More details of issue is mentioned here: kubernetes/kubernetes#95733.

But anyways the mitigation which you suggested above needs to be fixed for it to work seamlessly

[Tratcher]

Oh, header merging, that's no good. #155 covers writing some tests for that scenario. We already fixed one issue with cookies.

[wenzj-code]

Describe the bug

I am trying to use YARP as reverse proxy behind a kubernetes cluster. Normal HTTP requests work fine but API's which require connection upgrade are not working(For example: kubectl exec). I tried using IHttpUpgradeFeature as well to make YARP understand if this request is a upgrade request or not. But with that change as well it's not working.

To Reproduce

• You would need to have a kubernetes cluster.
• Get your K8s cluster's CA data and use it as client certificate and point the destination address to k8s api server.
• Deploy nginx on to your k8s cluster by following this doc: https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/#creating-and-exploring-an-nginx-deployment
• Change the api server endpoint in your kubeconfig to yarp proxy server.
• Run kubectl exec command with the pod information form your previously created deployment.

This is the response we get from k8s API server when we use YARP as reverse proxy in between:
error: unable to upgrade connection: unable to upgrade: unable to negotiate protocol: client supports [v4.channel.k8s.io, v3.channel.k8s.io, v2.channel.k8s.io, channel.k8s.io], server accepts [v4.channel.k8s.io v3.channel.k8s.io v2.channel.k8s.io channel.k8s.io]

Further technical details

• Include the version of the packages you are using: Running using ReverseProxy.Code.Sample of latest master as of 10/10/2020
• The platform (Linux/macOS/Windows) Windows

How to write an apiservice-agent with spdy, just like kubectl->apiservice-agent-> k8s-apiservice
I just want to achieve this function like "kubectl exec "

[Tratcher]

Comments on closed issues are not tracked, please open a new issue with the details for your scenario.