CAPD does not work when compiled with Go 1.20.6

[sbueringer]

I had a Docker update on 13.07 which updated to Docker 24.0.4. I realized today that CAPD doesn't work anymore with that

Basically the exec is broken:

{"ts":1689947751414.039,"caller":"controller/controller.go:324","msg":"Reconciler error","controller":"dockermachine","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"DockerMachine","DockerMachine":{"name":"capi-quickstart-control-plane-5w5mn","namespace":"default"},"namespace":"default","name":"capi-quickstart-control-plane-5w5mn","reconcileID":"dfd6131e-27c0-4533-b92f-010233980f40","err":"failed to update DockerCluster.loadbalancer configuration: failed to create directory /usr/local/etc/haproxy/haproxy.cfg: error attaching to container exec: http: invalid Host header"}

I found some similar reports:

• go 1.20.6: failed to attach to exec: http: invalid Host header nektos/act#1908
• Golang client fails to attach to streams with "http: invalid Host header" with go1.20.6, go1.19.11 moby/moby#45935

I would suggest like described there to bump to github.com/docker/docker v24.0.5-0.20230714235725-36e9e796c6fc+incompatible to fix the issue.

I think it would be also good to backport this to all release branches. Otherwise I would expect that CAPD will just stop working as soon as folks update Docker.

As soon as Docker v24.0.5 is released we should bump to the release

[sbueringer]

/triage accepted

[killianmuldoon]

/reopen

Let's use this to track the upcoming release of v24.0.5 and to ensure it gets cherry-picked to relevant branches.

[k8s-ci-robot]

@killianmuldoon: Reopened this issue.

In response to this:

/reopen

Let's use this to track the upcoming release of v24.0.5 and to ensure it gets cherry-picked to relevant branches.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[killianmuldoon]

Certainly release-1.4 and release-1.5 - not sure what the status of release-1.3 as the last release on the 1.3 branch will likely be before the docker release. @sbueringer thoughts?

[sbueringer]

I would cherry-pick as long as we still can. Better have a v1.3 version out that still works for a bit (until something else breaks :D)

[cpuguy83]

This is a client-side issue.
So if your client is built with a newer version of Go it will get this invalid host header stuff.

[cpuguy83]

Only commenting to clarify, it doesn't matter if someone updates docker on the host or not. It is entirely an issue with Go and the docker client lib. Bumping the client lib to what's in the (currently) unreleased branch should take care of it.

[sbueringer]

Thank you very much. Then I misinterpreted what I saw locally. I just checked and apparently Go was updated to 1.20.6 on my machine on 14th July.

I assume the rebuild in Tilt then built the CAPD manager with my locally Go version and that's how this issue suddenly surfaced

Running cmd: cd test/infrastructure/docker;mkdir -p .tiltbuild/bin;CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build  -gcflags 'all=-N -l' -ldflags '-extldflags "-static"' -o .tiltbuild/bin/manager main.go

When I pin to Go 1.20.5 (without the Docker bump) everthing still works.

[killianmuldoon]

/assign