net/http: auth.tiki-dsp.io sent ENHANCE_YOUR_CALM on >4KB header

[Freshwood]

This issue could be similar with this:
#20979

What version of Go are you using (go version)?

$ go version
Tested with GO 1.12 and GO 1.13.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Tobias Frischholz\AppData\Local\go-build
set GOENV=C:\Users\Tobias Frischholz\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\code\work\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\code\work\ai-provisioner\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\TOBIAS~1\AppData\Local\Temp\go-build857823394=/tmp/go-build -gno-record-gcc-switches

What did you do?

We want to create a new realm programmatically in our idenity provider Keycloak with the go http client library.
We are using the keycloak:6.0.1 version.
The problem only arises if the HTTP authorization header is greater than 4KB. With headers under 4KB we get a success response.

To fix this problem we are testing the following:

• Try a different client with big headers -> Request was successful
• Try the new go version 1.13.1 -> Error; Enhance your calm
• Keycloak uses Jboss. Disabled http2 connector -> Error; Enhance your calm
• Set http2client=0 to disable http2 go client -> Request was successful
• Try different http client from other languages -> Request was successful

The actual workaround is to set "http2client=0" off. Unfortunately there is no clean way to give the go http client a property which disable the http2 client support.
Currently we have no idea what the problem could be.
Maybe there is a problem between go http client and the Keycloak JBoss server implementation...

What did you expect to see?

A success response from the identity provider keycloak without to set the flag "http2client=0"

What did you see instead?

Http Code 420 enhance your calm.
Here is the http2 detailed error message. (GODEBUG=http2debug=2)

2019/10/10 10:19:22 http2: Transport failed to get client conn for auth.tiki-dsp.io:443: http2: no cached connection was available
2019/10/10 10:19:23 http2: Transport creating client conn 0xc000052900 to 213.95.153.164:443
2019/10/10 10:19:23 http2: Framer 0xc00015a540: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2019/10/10 10:19:23 http2: Framer 0xc00015a540: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2019/10/10 10:19:23 http2: Transport encoding header ":authority" = "auth.tiki-dsp.io"
2019/10/10 10:19:23 http2: Transport encoding header ":method" = "GET"
2019/10/10 10:19:23 http2: Transport encoding header ":path" = "/auth/admin/realms/zol"
2019/10/10 10:19:23 http2: Transport encoding header ":scheme" = "https"
2019/10/10 10:19:23 http2: Transport encoding header "authorization" = "Bearer "
2019/10/10 10:19:23 http2: Transport encoding header "accept-encoding" = "gzip"
2019/10/10 10:19:23 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2019/10/10 10:19:23 http2: Framer 0xc00015a540: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=6884
2019/10/10 10:19:23 http2: Framer 0xc00015a540: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=128, INITIAL_WINDOW_SIZE=65536, MAX_FRAME_SIZE=16777215
2019/10/10 10:19:23 http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=128, INITIAL_WINDOW_SIZE=65536, MAX_FRAME_SIZE=16777215
2019/10/10 10:19:23 http2: Framer 0xc00015a540: wrote SETTINGS flags=ACK len=0
2019/10/10 10:19:23 http2: Framer 0xc00015a540: read WINDOW_UPDATE len=4 (conn) incr=2147418112
2019/10/10 10:19:23 http2: Transport received WINDOW_UPDATE len=4 (conn) incr=2147418112
2019/10/10 10:19:23 http2: Framer 0xc00015a540: read SETTINGS flags=ACK len=0
2019/10/10 10:19:23 http2: Transport received SETTINGS flags=ACK len=0
2019/10/10 10:19:23 http2: Framer 0xc00015a540: read GOAWAY len=8 LastStreamID=1 ErrCode=ENHANCE_YOUR_CALM Debug=""
2019/10/10 10:19:23 http2: Transport received GOAWAY len=8 LastStreamID=1 ErrCode=ENHANCE_YOUR_CALM Debug=""
2019/10/10 10:19:23 transport got GOAWAY with error code = ENHANCE_YOUR_CALM
2019/10/10 10:19:23 http2: Transport readFrame error on conn 0xc000052900: (*errors.errorString) EOF
2019/10/10 10:19:23 RoundTrip failure: http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=ENHANCE_YOUR_CALM, debug=""
Get https://auth.tiki-dsp.io/auth/admin/realms/zol: http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=ENHANCE_YOUR_CALM, debug=""

Here is also a code snippet which was causing the above mentioned issue:

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {

	url := "https://"

	req, _ := http.NewRequest("GET", url, nil)

	req.Header.Add("authorization", "Bearer ")

	res, err := http.DefaultClient.Do(req)

	if err != nil {
		println(err.Error())
		return
	}

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))
}

[andybons]

@bradfitz

[bradfitz]

Try a different client with big headers -> Request was successful

Which client? Using HTTP/1 or HTTP/2?

This sounds like their HTTP/2 server is misconfigured.

I don't see what heuristic we could use to fall back to HTTP/1 in this case. On any ErrCode=ENHANCE_YOUR_CALM? Seems severe.

I'd rather they fix their server.

[tiki-sk]

We fixed the issue after proper server configuration. The header field size for http2 was limited on our nginx server. You can close the issue.

[ALTree]

Closing as requested.