proposal: x/net/http2: support for WebSockets over HTTP/2

[mitar]

What version of Go are you using (go version)?

go version go1.17.3 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What did you do?

I wanted to use Websockets over HTTP2.

What did you expect to see?

Support for it in Go standard library.

What did you see instead?

It does not support it yet.

See #46319 for background.

WebSockets over HTTP2 have been standardized as RFC 8441 and Firefox and Chromium supports that. Thus, I suggest that support is added for Websockets over HTTP2. The server should also be able to specify using HTTP/2 SETTINGS parameter that it supports Websockets over HTTP2.

[aojea]

also related #27244

[aojea]

Just for reference https://datatracker.ietf.org/doc/html/rfc8441 it seems this requires (please correct me if I'm wrong, not very familiar with the stdlib implemententation):

1. Implement the SETTINGS_ENABLE_CONNECT_PROTOCOL SETTINGS

Requires adding an HTTP2 server option to enable the setting

2. Implement the The Extended CONNECT Method

This requires adding a new field to the requests

   o  A new pseudo-header field :protocol MAY be included on request
      HEADERS indicating the desired protocol to be spoken on the tunnel
      created by CONNECT.  The pseudo-header field is single valued and
      contains a value from the "Hypertext Transfer Protocol (HTTP)
      Upgrade Token Registry" located at
      <https://www.iana.org/assignments/http-upgrade-tokens/>

and adding the logic to use the field if SETTINGS_ENABLE_CONNECT_PROTOCOL has been received.

EDIT1 :/

go/src/net/http/request.go

Lines 107 to 109 in 90fb5a4

	// Go's HTTP client does not support sending a request with
	// the CONNECT method. See the documentation on Transport for
	// details.

EDIT2, so this seems to be on purpose

#22554 (comment)

So most users of CONNECT (including Go's build infrastructure in: https://github.com/golang/build/blob/ce623a5/cmd/buildlet/reverse.go#L213) either fmt.Fprintf the CONNECT request themselves, or do *http.Request.Write it to a self-dialed net.Conn, rather than using *http.Transport.

3. Bootstrap the websocket protocol

HTTP/2 WebSockets need a way to read from and write to the HTTP/2 stream for a request, not a way to hijack the connection.

[aojea]

I think that this will only require adding a new option on the http2 server to enable the SETTINGS_ENABLE_CONNECT_PROTOCOL option and another one in the client to let the user know if that setting is enable.

If I understand correctly how CONNECT works in golang, the rest of the logic can be done at a higher level in a similar way that is being done with https://github.com/golang/net/tree/master/websocket

[JeremyLoy]

Considering the gorilla/websocket library is currently unmaintained, which is causing a lot of concern in the community, I think this is a great opportunity to solidify and complete websocket support in the standard library

[aojea]

I did a prototype https://github.com/aojea/net/pull/1/files to get a better idea on what will be required to implement this.

My main concern is that golang seems to avoid implementing the CONNECT semantics in the stdlib #22554 (comment), leaving the implementation to the users.
The requirement of the Extended CONNECT Method will require to add new CONNECT semantics (including a new pseudoheader) to the stdlib, I don't know what the golang team think about this 😄

[hexfusion]

How can we move this forward? I think many folks are interested in the implementation of rfc8441 it has been a few years[1].

@neild could you possibly help direct us on next steps?

[1] #32763

[ethanpailes]

I've started poking at this a bit and it seems that a resolution will require two patches: one to the x/net/http2 package to allow stream oriented interaction with an http2 stream, and then a followup change to x/net/websocket to add the actual websocket support. I've talked about this a bit on the go-nuts mailing list, but it really should be a public discussion, so I'm moving it here. Here's the most recent message I posted about a strawman patch I've put together:

I've got something working for exposing HTTP/2 streams as Stream struct that is spiritually an io.ReadWriteCloser on the client side, and using existing interfaces on the server side. I'm posting about it here to give people a chance to share initial thoughts before I write up something more formal. I've posted my changes on gerrit in case anyone wants to look at the details.

On the server side of things, it seems that the current interface is basically already adequate except for the fact that the server currently rejects any requests using non https/http schemes. The http.ResponseWriter that gets passed down into HTTP/2 handlers implements http.Flusher, allowing you to force writes out on to the wire, and you can just use the http.Request.Body as the reader for the stream. The right thing to do here seems relatively straightforward. We can do one of a few things:

1. Convert the scheme check to a regex check based on RFC3986 3.1. The spec does say in 8.1.2.3 that "All HTTP/2 requests MUST include exactly one valid value for the ":method", ":scheme", and ":path" pseudo-header fields, ... An HTTP request that omits mandatory pseudo-header fields is malformed (Section 8.1.2.6)." (emphasis mine), but in the same section it also says '":scheme" is not restricted to "http" and "https" schemed URIs.'. I think this should be read to mean that the only check we should be performing on scheme is that it parses correctly according to RFC3986 3.1. This is the option I think makes the most sense.
2. Just expand the allowlist in the check so that it includes wss and ws as well as http and https. This is a more conservative option and I think can be defended on the basis that the http library isn't generally expected to serve as a proxy, so a fixed list of allowed schemes is fine.
3. Drop the check entirely as I've done in my strawman patch. This seems wrong.

In terms of the client side of things, I'm relatively happy with the interface for the Stream struct I added. It re-uses the http.Request and http.Response types to let people set and read headers in a familiar way, and is basically just an unpacked version of the RoundTrip routine that keeps the request open after the stream gets initially created and only closes it when the user explicitly asks for a close. There are some things that seem clunky about it to me though:

1. It's hanging off the http2.Transport struct. As a user I would expect something like this on the http.Client, so maybe it is better to put it there somehow. That does run into the issue of this routine not being available for HTTP/1.1 though.
2. It might be better not to re-use the http.Request/http.Response types since we aren't really making requests and responses. Maybe a more unpacked API that directly references headers and trailers makes more sense.
3. I always feel weird about performing IO operations without passing a context down. Currently Stream uses the context attached to the request passed in to open the stream, which seems fine, but maybe a more explicit API would be better.

cc: @neild @ianlancetaylor

[ethanpailes]

I just noticed https://datatracker.ietf.org/doc/html/rfc8441#section-5 saying that for HTTP/2 wss:// URIs should be translated to https:// and ws:// URIs should be translated to http:// URIs, so it seems that we may not need to make any changes to the server side of the HTTP/2 stack after all.

[neild]

Commented on https://go.dev/cl/408835: Do we need a new stream-oriented Transport API? RoundTrip already keeps the request stream open so long as the request or response body are still being written.

We'd need to add support for SETTINGS_ENABLE_CONNECT_PROTOCOL. That should be straightforward.

We also need a way to set/get the :protocol pseudo-header. Maybe just put it in the Header map?

Unless we put some level of websocket bootstrap support directly into net/http, a websocket client is going to need to know whether it's going to be speaking HTTP/1 or HTTP/2, since the initial protocol bootstrap differs between the two. This is probably okay.

[ethanpailes]

Yeah, I think those are all good points. The suggestion you made in the CL works with no changes to http2 for a GET request, but runs into SETTINGS_ENABLE_CONNECT_PROTOCOL trouble when I try to use CONNECT with it.

I've started poking at SETTINGS_ENABLE_CONNECT_PROTOCOL support a bit. It seems like it should be easy enough to just advertise it in the initial settings frame on the server side and then set it as a flag on the ClientConn on the server side. We probably don't even need to expose it to users on the client side, we could just check the flag and return an error if they try to send a CONNECT message and the setting is not enabled.

Currently, the client seems to clobber the scheme and path when you try sending a CONNECT request, which the server doesn't much like, so I'm trying to figure that out. I'm a little worried about the compatibility of no longer clobbering those parts of the URI though, since users might have come to rely on that behavior. Do you know what the reason for clobbering it like this is?

[neild]

HTTP/2 CONNECT explicitly doesn't allow the :scheme and :path pseudo-headers:
https://datatracker.ietf.org/doc/html/rfc7540#section-8.3

The SETTINGS_ENABLE_CONNECT_PROTOCOL parameter changes the semantics of CONNECT.

ClientConn.encodeHeaders should handle CONNECT requests with a :protocol pseudo-header differently from ones without--in particular, it should return an error if the server didn't set SETTINGS_ENABLE_CONNECT_PROTOCOL, and it should include the :path and :scheme if the server did.

[ethanpailes]

Ah, that makes sense. I had thought it was just an advertisement that the server knew how to handle the connect protocol, but a mode switch makes more sense. Then it probably sense to just add an option to http2.Server to tell it to send the ENABLE_CONNECT_PROTOCOL setting and have the client check the setting to determine if it ought to clobber the scheme and path.

[neild]

I'm not sure we even need an option; maybe just always send SETTINGS_ENABLE_CONNECT_PROTOCOL.

[ethanpailes]

I was thinking that someone might be using the http package to implement a proxy, though that seems like a pretty strange usecase. If you think we don't have to worry about that, then having no public API changes would be awesome. I'll make my reworking of the patch I posted just send it all the time.

[neild]

SETTINGS_ENABLE_CONNECT_PROTOCOL doesn't prevent the old form of CONNECT, it just advertises support for the extended one with a :protocol pseudo-header as well.

It is true that if we advertise it unconditionally, then existing CONNECT handlers will mishandle a CONNECT with a :protocol pseudo-header. I'm not sure if that's a problem.

[ethanpailes]

I've had the time to rework that straw-man patch to just use the request and response body on the client side as suggested and it results in a much cleaner implementation with basically no public API impact.

The one issue is that it is not totally clear from the existing interface how to ask the http client to set the :protocol psudoheader. My understanding is that users are not really supposed to set psudoheaders directly in the headers struct or even know they exist for the most part. For the moment, I've added a magic HACK-HTTP2-Protocol header you can inject into the request headers table, but this is obviously terrible and unworkable. If you can think of a way to set :protocol with the existing interface, I would love to hear about it. If not, I'm going to work on a strawman patch for x/net/websocket just to prove to myself that this http2 SETTINGS_ENABLE_CONNECT_PROTOCOL support is actually sufficient to implement websocket support and then make a series of more formal proposals.

1. Propose to add a new Protocol field to http.Request so that users can get the client to set :protocol = websocket
2. Propose extending the x/net/http2 to support SETTINGS_ENABLE_CONNECT_PROTOCOL using much the same code I just posted
3. Propose extending x/net/websocket, details are still a bit hazy here since I don't really have code I'm happy with just yet.

If that all sounds reasonable, no need to look at any of this too closely until I get to those proposals.

[ethanpailes]

Fortunately, it seems like implementing websocket support in terms of that SETTINGS_ENABLE_CONNECT_PROTOCOL patch is fairly straightforward. I put together a strawman patch here: ethanpailes/golang-org-x-net@df38e5d

[ethanpailes]

#53208 is a more formal proposal for the SETTINGS_ENABLE_CONNECT_PROTOCOL support. I'll work on one for the websocket support now.

[ethanpailes]

#53209 is a more formal proposal for the actual HTTP/2 websocket support. It depends on #53208.

[let4be]

Any update?

[masx200]

Any update?

[ethanpailes]

@masx200 the main update is #53208 (comment), the key proposal blocking this proposal has entered the "likely accept" state which means that if nothing else comes up work can begin to implement that. That proposal is probably more important because without it not even third party crates can do a reasonable WebSocket implementation for http2 (technically they could, but they would need to make an entire http stack themselves).

[WeidiDeng]

@ethanpailes Now that the proposal is accepted, will you continue work on this or perhaps I can give it a go? As a member of caddy I want to work on this issue.

[gwitsch]

+1 for status update and integration.
Stumbled accross this issue by using Traefik as a proxy for Java backend services.

[ethanpailes]

@WeidiDeng my original reason for wanting this feature is no longer relevant to me, so I'm not doing it as part of my day job anymore. I would like to see it go over the finish line though so I do intend to pick it up eventually. If you're feeling modivated to work on it, I won't stand in your way though.

[gopherbot]

Change https://go.dev/cl/610977 mentions this issue: http2: server advertises SETTINGS_ENABLE_CONNECT_PROTOCOL support

[gopherbot]

Change https://go.dev/cl/630875 mentions this issue: http2: add SETTINGS_ENABLE_CONNECT_PROTOCOL support

[luavixen]

+1 cause I stumbled upon this when trying to set up Caddy as a reverse proxy

[mchandler-plato]

Testing out his change and im hitting a panic with echo and processing the websocket:

cho: http2: panic serving 127.0.0.1:38990: interface conversion: *http2.responseWriter is not http.Hijacker: missing method Hijack
goroutine 487 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1()
	/home/lodle/go/pkg/mod/github.com/golang/[email protected]/http2/server.go:2468 +0x253
panic({0x11a9da0?, 0xc00155e300?})
	/snap/go/10743/src/runtime/panic.go:785 +0x136
github.com/labstack/echo/v4.(*Response).Hijack(0xc0005a4120)
	/home/lodle/go/pkg/mod/github.com/labstack/echo/[email protected]/response.go:94 +0x85
golang.org/x/net/websocket.Server.serveWebSocket({{0x0, 0x0, {0x0, 0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, 0x0}, ...}, ...)
	/home/lodle/go/pkg/mod/github.com/golang/[email protected]/websocket/server.go:74 +0x117
golang.org/x/net/websocket.Server.ServeHTTP({{0x0, 0x0, {0x0, 0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, 0x0}, ...}, ...)
	/home/lodle/go/pkg/mod/github.com/golang/[email protected]/websocket/server.go:70 +0x58
git.playchat.net/playchat/gameproxyservice/internal/handler.(*WebHandler).StreamPSession(0xc000838660, {0x1403a80, 0xc000a141e0})

usage:

func HandleWebSocket(c echo.Context) error {
	wsh := websocket.Handler(func(ws *websocket.Conn) {
		defer ws.Close()
		runStream(c.Request().Context(), ws)
	})

	wss := websocket.Server{Handler: wsh}
	wss.ServeHTTP(c.Response(), c.Request())
	return nil
}

My guess is if its a http2 stream not a http1 connection the net package shouldnt need to hijack the connection when processing the websocket

[mchandler-plato]

Hacking up the net package i did make it work with chrome -> envoy -> go over http.
See https://gist.github.com/mchandler-plato/ad89f4e398e9232c4370ec45604101f0 for what i changed.

[aojea]

@mchandler-plato thanks for the snippet, I was not able to find your email to add you as co-author but if you share it I will add you to https://go-review.googlesource.com/c/net/+/632755

It seems is now easy to support websockets over http2

[mchandler-plato]

@mchandler-plato thanks for the snippet, I was not able to find your email to add you as co-author but if you share it I will add you to https://go-review.googlesource.com/c/net/+/632755

Sent you an email with details. Thanks!

[gopherbot]

Change https://go.dev/cl/632899 mentions this issue: websocket: add http2 support using extended connect