net/http: document how to create client CONNECT requests

[Audrey-Randall]

What version of Go are you using (go version)?

Go1.9

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

windows/amd64

What did you do?

This error is reproducible at https://play.golang.org/p/9m7rlnqsTm

I attempted to create a new HTTP request with http.NewRequest() using a URI that starts with a number, with no protocol, that also contains a port:
_, err = http.NewRequest(http.MethodConnect, "0.docs.google.com:443", nil)

What did you expect to see?

The request should be created with no error.

What did you see instead?

I get the error "parse : first path segment in URL cannot contain colon"

[bradfitz]

This is actually working as intended. "host:port" is not a valid URL, and http.NewRequest takes a URL, not a host:port.

For a CONNECT request (which is a weirdo type of request), you'll probably just want to construct the *http.Request by hand. It's specially recognized by some but not all of net/http helpers. In particular, http.NewRequest does not go out of its way to support CONNECT weirdness.

[bradfitz]

Actually, I suppose we could also make NewRequest special-case CONNECT and also support host:port there.

@tombergan, thoughts?

[sergeyfrolov]

Thanks for reopening this!

I would like to note that it's not necessarily about CONNECT. If user wants to specify port for other types of requests, say, http.MethodGet, then behavior is the same:

• "0google.com:443" fails
• "google.com:443" doesn't fail

[bradfitz]

@sergeyfrolov, for GET requests or anything other than CONNECT, we will not change behavior. "foo.com:443" is not a valid URL. You need a scheme (http://, https://, etc).

[tombergan]

I would like to note that it's not necessarily about CONNECT. If user wants to specify port for other types of requests, say, http.MethodGet, then behavior is the same:

"0google.com:443" fails
"google.com:443" doesn't fail

That's because "google.com:443" doesn't parse how you think it does ... it parses "google.com" into the scheme, not the host. If "google.com:443" is working for you, that's accidental rather than intentional.
https://play.golang.org/p/omOEwB02Me

Actually, I suppose we could also make NewRequest special-case CONNECT and also support host:port there.

I don't think this would work, for two reasons. First, what scheme does the following example use when connecting to foo.com?

// Assume that NewRequest special-cases CONNECT to support host:port.
// req.URL = {Scheme: ??? , Host = "google.com:443"}
req, _ := http.NewRequest("CONNECT", "google.com:443", nil)

// Connect over TCP to www.proxy.com, which will forward the TCP connection
// to google.com:443. Then, do we TLS to google.com:443 or use raw TCP?
// The :443 suggests that the caller wants TLS, but in general we cannot know
// since req.URL does not have a scheme.
tr := http.Transport{Proxy: http.ProxyURL("http://www.proxy.com/")}
tr.RoundTrip(req)

Second, the following would be confusing:

// Assume that NewRequest special-cases CONNECT to support host:port.
// Note that r1.Host != r2.Host
// r1.URL = {Scheme: ??? , Host = "google.com:443"}
// r2.URL = {Scheme: "google.com", Opaque: "443", Host = ""}
r1, _ := http.NewRequest("CONNECT", "google.com:443", nil)
r2, _ := http.NewRequest("GET", "google.com:443", nil)

@sergeyfrolov's example is a good one. It is confusing that "0google.com:443" fails but "google.com:443" does not. More precisely:

// This is what happens currently.
// r1.URL = {Scheme: "google.com", Opaque: "443", Host = ""}
// r1.URL = nil
r1, _ := http.NewRequest("CONNECT", "google.com:443", nil)
r2, _ := http.NewRequest("CONNECT", "0google.com:443", nil)

Maybe it would be sufficient to fail both of the above http.NewRequest calls? We could do this by adding a requirement to http.NewRequest: when the method is "CONNECT", the URL must contain a non-empty host. In theory, this could break callers ... I can't imagine there are any callers that legitimately want to create a CONNECT request with an empty host, but the possibility is there.

[bradfitz]

I don't think this would work, for two reasons. First, what scheme does the following example use when connecting to foo.com?

Oh, right.

Okay, there's nothing we can do here except documentation.

I'll repurpose this as a documentation bug, to make it clear how to construct CONNECT requests.

[Audrey-Randall]

@tombergan Thanks for the clarification!

[ewust]

I don't think this would work, for two reasons. First, what scheme does the following example use when connecting to foo.com?

CONNECT isn't supposed to have a scheme; see RFC 7231, Section 4.3.6: https://tools.ietf.org/html/rfc7231#section-4.3.6

You only send a host:port, and then the proxy you sent the CONNECT method to blindly forwards arbitrary TCP data bidirectionally. It's up to the client that requested the CONNECT method to know what scheme it should talk to the remote host.

For example, if I send
CONNECT google.com:443 HTTP/1.1\r\nHost: google.com:443\r\n\r\n

The next thing I send into that is a raw TLS client hello message. So it doesn't make sense to have the CONNECT method require a scheme. In fact, if you send CONNECT https://google.com:443 HTTP/1.1 You'll get an error from anything that supports CONNECT (e.g. Squid sends a 503 with ERR_DNS_FAIL).

[bradfitz]

@ewust, the scheme isn't for the CONNECT argument, but for the HTTP (or HTTPS) server handling the CONNECT request. What we need to clarify in the docs is how users are expected to do CONNECT requests with Go's net/http package.

In general, Go's Transport doesn't permit sending a CONNECT request in any useful way (for HTTP/1 proxies), since you can't hijack the connection from the CONNECT response body. That is #17227.

Currently you can only do useful CONNECTS to http2 servers with Go.

So most users of CONNECT (including Go's build infrastructure in: https://github.com/golang/build/blob/ce623a5/cmd/buildlet/reverse.go#L213) either fmt.Fprintf the CONNECT request themselves, or do *http.Request.Write it to a self-dialed net.Conn, rather than using *http.Transport.

[tombergan]

Thanks Brad. I was drafting a more poorly-written version of that response.

or do *http.Request.Write it to a self-dialed net.Conn, rather than using *http.Transport

Yep, I think this is what most people do.

[gopherbot]

Change https://golang.org/cl/86277 mentions this issue: net/http: document CONNECT more