net/http: transport caches permanently broken persistent connections if write error happens during h2 handshake

[cbf123]

The fix for #34978 is not correct, the issue is still present.

What version of Go are you using (go version)?

$ go version
go version go1.13.12 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/cfriesen/.cache/go-build"
GOENV="/home/cfriesen/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/cfriesen/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/cfriesen/devel/h2bug/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build843802376=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Test-case is here: https://github.com/mikedanese/h2bug

What did you expect to see?

The persistent connection is not cached by http.Transport.

What did you see instead?

The persistent connection is cached by http.Transport.

More

Commit https://go.googlesource.com/go/+/88186e5e232625f9c91d639e0cb90a88c6cf1172 was submitted in order to fix #34978, but it appears to be not quite correct due to how http2/transport.go gets included into http via h2_bundle.go with an "http2" prefix added.

If I run the above testcase using "dlv test", then add a breakpoint in src/net/http/transport.go right after the assignment to isH2DialError, and then run "print pconn.alt", it gives me

(dlv) print pconn.alt
net/http.RoundTripper(golang.org/x/net/http2.erringRoundTripper) {
        err: error(*errors.errorString) *{
                s: "broken pipe",},}

At the same time, "isH2DialError" is "false" when it clearly should be "true" for the bugfix to be correct.

Trying to explicitly assert the types gave the following interesting results:

(dlv) print pconn.alt.(http2.erringRoundTripper)
golang.org/x/net/http2.erringRoundTripper {
        err: error(*errors.errorString) *{
                s: "broken pipe",},}
(dlv) print pconn.alt.(http.http2erringRoundTripper)
Command failed: interface conversion: net/http.RoundTripper is golang.org/x/net/http2.erringRoundTripper, not net/http.http2erringRoundTripper

[mikedanese]

Without rearchitecting the fix for #34978, one option is to change this assert:

go/src/net/http/transport.go

Line 1688 in fa98f46

if e, ok := alt.(http2erringRoundTripper); ok {

To check an interface that both http2erringRoundTripper and http2.erringRoundTripper implement.

[networkimprov]

cc @fraenkel

[fraenkel]

I can't see a way of fixing this without changing the http2 side to something we can detect on both sides.

[gopherbot]

Change https://golang.org/cl/243257 mentions this issue: net/http2: erringRoundTripper can be type asserted

[gopherbot]

Change https://golang.org/cl/243258 mentions this issue: net/http: erringRoundTripper is an interface

[gopherbot]

Change https://golang.org/cl/249842 mentions this issue: net/http: update bundled x/net/http2

[gopherbot]

Change https://golang.org/cl/249937 mentions this issue: net/http2: add test for erringRoundTripper

[gopherbot]

Change https://golang.org/cl/261919 mentions this issue: [release-branch.go1.15] src, net/http: update vendor, h2_bundle.go regeneration

[gopherbot]

Change https://golang.org/cl/258540 mentions this issue: [release-branch.go1.15] src, net/http: update vendor, regenerate h2_bundle.go

[gopherbot]

Change https://golang.org/cl/258538 mentions this issue: [release-branch.go1.14] src, net/http: update vendor, regenerate h2_bundle.go

[KevinWang15]

This bug is fixed, isn't it?

https://go-review.googlesource.com/c/net/+/243257
https://go-review.googlesource.com/c/go/+/243258

[KevinWang15]

I don't see it in release branches other than go 1.16, is it possible to backport it to go 1.15, go 1.14 by any chance?

Thanks, cc @fraenkel

(This one actually could affect Kubernetes: kubernetes/kubernetes#87615 (comment) Kubernetes 1.20 is still using golang 1.15)

[networkimprov]

@gopherbot please backport to 1.15, as Kubernetes is affected.

[gopherbot]

Backport issue(s) opened: #45076 (for 1.15).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[KevinWang15]

Any chance for 1.14 as well? 😂 cc @networkimprov

[seankhliao]

Only the latest 2 releases (1.15 and 1.16) are supported

[liggitt]

@networkimprov thanks for opening #45076, would be good to get this fixed in 1.15 since it can't easily be worked around by callers

[networkimprov]

Jordan, you may want to note that on #45076, as I have no influence on backport decisions.

[gopherbot]

Change https://golang.org/cl/304309 mentions this issue: [release-branch.go1.15-bundle] net/http2: fix erringRoundTripper

[gopherbot]

Change https://golang.org/cl/304210 mentions this issue: [release-branch.go1.15] net/http: fix detection of Roundtrippers that always error

[gopherbot]

Change https://golang.org/cl/305489 mentions this issue: [release-branch.go1.15] net/http: update bundled x/net/http2