Websocket connections rejected

[pl4nty]

Describe the bug
Might be too early in this project's lifecycle, but I'm seeing rejected websocket connections in a basic gateway/HTTPRoute deployment behind cloudflared. Testing with ksdn117/web-socket-test is successful when the gateway is bypassed, but fails behind the gateway.

To Reproduce
Steps to reproduce the behavior:

1. Deploy an HTTPRoute and ksdn117/web-socket-test or similar
2. Attempt to connect via websocket
3. See websocket closed response

Expected behavior
Websocket connection established.

Your environment

• Version of the NGINX Kubernetes Gateway: v0.2.0
• Version of Kubernetes: 1.25.2
• Kubernetes platform (e.g. Mini-kube or GCP): AKS
• Details on how you expose the NGINX Gateway Pod (e.g. Service of type LoadBalancer or port-forward): ClusterIP service with cloudflared tunnel to its DNS name
• Logs of NGINX container: kubectl -n nginx-gateway logs -l app=nginx-gateway -c nginx

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
10.244.0.88 - - [27/Nov/2022:12:14:32 +0000] "GET /app/0!default HTTP/1.1" 500 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.42"

• NGINX Configuration: kubectl -n nginx-gateway exec <gateway-pod> -c nginx -- nginx -T

# configuration file /etc/nginx/nginx.conf:
load_module /usr/lib/nginx/modules/ngx_http_js_module.so; events {}  pid /etc/nginx/nginx.pid; http { include /etc/nginx/conf.d/*.conf; js_import /usr/lib/nginx/modules/njs/httpmatches.js; }

# configuration file /etc/nginx/conf.d/http.conf:
upstream default_tsuro_80 {
    random two least_conn;
    server 10.244.0.110:80;
}
server {
        listen 80 default_server;
        default_type text/html;
        return 404;
}
server {
        server_name tsuro.tplant.au;
        location / {
                proxy_set_header Host $host;
                proxy_pass http://default_tsuro_80$request_uri;
        }
}
server {
    listen unix:/var/lib/nginx/nginx-502-server.sock;
    access_log off;
    return 502;
}
server {
    listen unix:/var/lib/nginx/nginx-500-server.sock;
    access_log off;
    return 500;
}

Additional context
Maybe the Connection or Upgrade headers aren't set correctly, like the fix in ingress-nginx? The application provides this error:

[info] c.jpro - application requested: 0!default
[error] a.a.ActorSystemImpl - Internal server error, sending 500 response
java.lang.RuntimeException: WebSocket returned for non WebSocket request
        at scala.sys.package$.error(package.scala:30)
        at play.core.server.AkkaHttpServer.executeHandler(AkkaHttpServer.scala:383)
        at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:305)
        at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:174)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1376)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:480)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:376)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)

Aha! Link: https://nginx.aha.io/features/NKG-19

[pleshakov]

Hi @pl4nty

We haven't added the required NGINX configuration to support Websocket.

Let's keep this issue open to track the development of this feature.

[evanjarrett]

Hi, I'm also interested in a proper way to support websockets through the gateway.
In the meantime I was able to patch NKG with a hack that allows me to get the functionality I need. Essentially, when I set the header to Upgrade, regardless of value, NKG adds in the correct $http_upgrade.

+++ b/internal/mode/static/nginx/config/servers_template.go
@@ -48,7 +48,12 @@ server {
 
         {{- if $l.ProxyPass -}}
             {{ range $h := $l.ProxySetHeaders }}
+                {{- if eq $h.Name "Upgrade" -}}
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+                {{- else }}
         proxy_set_header {{ $h.Name }} "{{ $h.Value }}";
+                {{- end }}
             {{- end }}
         proxy_set_header Host $gw_api_compliant_host;
         proxy_pass {{ $l.ProxyPass }}$request_uri;

My thought for a more permanent solution would be to add an annotation to the HTTPRoute for enabling websockets.
The downside would be all rules on that route would apply the websocket headers.

[kate-osborn]

@evanjarrett @pl4nty thanks for your interest in the project. We will be working on a fix for this shortly and it should be available in our next release (v0.6.0).