Not able to start kind cluster in Mac(M2) using podman [podman exec broken upstream due to mismatched client/server version]

[kamalkraj]

podman version 4.8.0
kind version 0.20.0

kind create cluster -v 1

using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Creating cluster "kind" ...
DEBUG: podman/images.go:67] Pulling image: docker.io/kindest/node@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72 ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✗ Writing configuration 📜
Deleted nodes: ["kind-control-plane"]
ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "podman exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 255
Command Output: v1.27.3Error: unmarshalling error into &errorhandling.ErrorModel{Because:"", Message:"", ResponseCode:0}, data "Not Found\n": invalid character 'N' looking for beginning of value
Stack Trace:
sigs.k8s.io/kind/pkg/errors.WithStack
	sigs.k8s.io/kind/pkg/errors/errors.go:59
sigs.k8s.io/kind/pkg/exec.(*LocalCmd).Run
	sigs.k8s.io/kind/pkg/exec/local.go:124
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman.(*nodeCmd).Run
	sigs.k8s.io/kind/pkg/cluster/internal/providers/podman/node.go:146
sigs.k8s.io/kind/pkg/exec.OutputLines
	sigs.k8s.io/kind/pkg/exec/helpers.go:81
sigs.k8s.io/kind/pkg/cluster/nodeutils.KubeVersion
	sigs.k8s.io/kind/pkg/cluster/nodeutils/util.go:37
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.getKubeadmConfig
	sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config/config.go:158
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.(*Action).Execute.func1.1
	sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config/config.go:90
sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1
	sigs.k8s.io/kind/pkg/errors/concurrent.go:30
runtime.goexit
	runtime/asm_arm64.s:1172

[kamalkraj]

with podman 4.7.2 working properly

[BenTheElder]

failed to get file: command "podman exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 255
Command Output: v1.27.3Error: unmarshalling error into &errorhandling.ErrorModel{Because:"", Message:"", ResponseCode:0}, data "Not Found\n": invalid character 'N' looking for beginning of value

this looks like a podman bug. we're not doing any unmarshalling ourselves, just asking podman to exec cat of a plain text file ...

[BenTheElder]

podman failing exec entirely should probably be debugged with the podman maintainers.

[gnomefin]

I also use podman 4.7.2 and it works: https://github.com/containers/podman/releases/tag/v4.7.2 there is something wrong with 4.8.0

[vietk]

Hello,
I got the same issue but with a Mac with an Intel Chip

[BenTheElder]

It looks like it's confirmed that downgrading podman version will fix it? containers/podman#20821

[BenTheElder]

This appears to be containers/podman#20821 (comment)

The bug is podman exec doesn't work due to server/client version mismatch, so I'd say this is 100% a podman bug and currently the recommendation is to downgrade your podman installation to an older version until a bugfix is out in podman OR ensure that you have the latest podman client binary to go with the podman VM.

[BenTheElder]

Should be fixed upstream containers/podman#20831

[containerscrew]

Same error using mac M1 PRO arm64 and podman 4.8.0.

Workaround, as other people mentioned:

1. Uninstall v4.8.0 (in my case, using brew): brew uninstall podman podman-desktop podman-compose && brew cleanup
2. Download pkg package from https://github.com/containers/podman/releases (v4.7.2)

Now works fine :)

[BenTheElder]

containers/podman#20821 (comment) may also be an option, aside from upgrading to the next release with CLI skew fix.

[BenTheElder]

I'm going to close this as the situation was resolved in podman, but pin it for some time for visibility.