unable to create/run a podman machine using libkrun provider with podman 5.3.0

[benoitf]

Issue Description

Creating/starting a podman machine using libkrun on macOS I have the following error
Error: krunkit exited unexpectedly with exit code 1

Steps to reproduce the issue

Steps to reproduce the issue

download and install podman 5.3.0 from the releases page on macOS (sequoia 15.1)

1. CONTAINERS_MACHINE_PROVIDER=libkrun podman machine init --now my-machine

Looking up Podman Machine image at quay.io/podman/machine-os:5.3 to create VM
Extracting compressed file: my-machine-arm64.raw: done
Machine init complete
Starting machine "my-machine"
Error: krunkit exited unexpectedly with exit code 1

Describe the results you received

Error: krunkit exited unexpectedly with exit code 1

Describe the results you expected

machine starts

podman info output

podman info                                                                                                                                  OS: darwin/arm64
provider: libkrun
version: 5.3.0

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:63626: connect: connection refused

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Tried with --log-leve=DEBUG and got this terminal window (and the command is hanging)

CONTAINERS_MACHINE_PROVIDER=libkrun podman machine --log-level=DEBUG init --now my-machine3                                                
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with `libkrun` virtualization provider
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0000] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun.json is 66
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun2.json is 67
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun3.json is 67
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine.json is 72
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine2.json is 73
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0000] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json is 73
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-arm64.raw is 83
Looking up Podman Machine image at quay.io/podman/machine-os:5.3 to create VM
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying to access "quay.io/podman/machine-os:5.3"
DEBU[0000] Found credentials for quay.io/podman/machine-os in credential helper containers-auth.json in file /Users/benoitf/.config/containers/auth.json
DEBU[0000]  No signature storage configuration found for quay.io/podman/machine-os:5.3, using built-in default file:///Users/benoitf/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io
DEBU[0000] GET https://quay.io/v2/
DEBU[0000] Ping https://quay.io/v2/ status 401
DEBU[0000] GET https://quay.io/v2/auth?account=fbenoit&scope=repository%3Apodman%2Fmachine-os%3Apull&service=quay.io
DEBU[0001] Increasing token expiration to: 60 seconds
DEBU[0001] GET https://quay.io/v2/podman/machine-os/manifests/5.3
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json"
DEBU[0001] found image in digest: "sha256:414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7"
DEBU[0001] GET https://quay.io/v2/podman/machine-os/manifests/sha256:414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7
DEBU[0004] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0004] original artifact file name: podman-machine.aarch64.applehv.raw.zst
DEBU[0004] GET https://quay.io/v2/podman/machine-os/manifests/sha256:414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7
DEBU[0004] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0004] original artifact file name: podman-machine.aarch64.applehv.raw.zst
DEBU[0004] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache/414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7.raw.zst is 140
DEBU[0004] cached image exists and is latest: /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache/414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7.raw.zst
DEBU[0004] Detected compression format zstd
Extracting compressed file: my-machine3-arm64.raw: done
DEBU[0012] --> imagePath is "/Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-arm64.raw"
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.ign is 72
DEBU[0012] resizing /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-arm64.raw to 107374182400 bytes
DEBU[0012] writing ignition file to "/Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.ign"
DEBU[0012] writing configuration file "/Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json"
Machine init complete
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json is 73
Starting machine "my-machine3"
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun.json is 66
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun2.json is 67
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun3.json is 67
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine.json is 72
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine2.json is 73
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json is 73
DEBU[0012] connection refused: http://localhost:63860/vm/state
DEBU[0012] connection refused: http://localhost:64052/vm/state
DEBU[0012] connection refused: http://localhost:64269/vm/state
DEBU[0012] connection refused: http://localhost:49277/vm/state
DEBU[0012] connection refused: http://localhost:65067/vm/state
DEBU[0012] connection refused: http://localhost:49345/vm/state
DEBU[0012] writing configuration file "/Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json"
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.pid is 67
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-api.sock is 76
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/podman.sock is 65
DEBU[0012] {true 1500 map[forward-dest:[/run/user/501/podman/podman.sock] forward-identity:[/Users/benoitf/.local/share/containers/podman/machine/machine] forward-sock:[/var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-api.sock] forward-user:[core]] [] map[] /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.log /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.pid 49327}
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-gvproxy.sock is 80
DEBU[0012] gvproxy command-line: /opt/podman/bin/gvproxy -debug -mtu 1500 -ssh-port 49327 -listen-vfkit unixgram:///var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-gvproxy.sock -forward-user core -forward-identity /Users/benoitf/.local/share/containers/podman/machine/machine -forward-sock /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-api.sock -forward-dest /run/user/501/podman/podman.sock -pid-file /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.pid -log-file /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.log
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-gvproxy.sock is 80
DEBU[0012] checking that "gvproxy" socket is ready
DEBU[0013] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3.log is 71
DEBU[0013] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3.sock is 72
DEBU[0013] helper binary path is: /opt/podman/bin/krunkit
DEBU[0013] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-ignition.sock is 87
DEBU[0013] first boot detected
DEBU[0013] serving ignition file over /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-ignition.sock
DEBU[0013] listening for ready on: /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3.sock
DEBU[0013] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.ign is 72
DEBU[0013] reading ignition file: /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.ign
DEBU[0013] waiting for ready notification
DEBU[0013] helper command-line: [/opt/podman/bin/krunkit --cpus 6 --memory 2048 --bootloader efi,variable-store=/Users/benoitf/.local/share/containers/podman/machine/libkrun/efi-bl-my-machine3,create --device virtio-blk,path=/Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-arm64.raw --device virtio-rng --device virtio-vsock,port=1025,socketURL=/var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3.sock,listen --device virtio-net,unixSocketPath=/var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-gvproxy.sock,mac=5a:94:ef:e4:0c:ee --device virtio-fs,sharedDir=/Users,mountTag=a2a0ee2c717462feb1de2f5afd59de5fd2d8 --device virtio-fs,sharedDir=/private,mountTag=71708eb255bc230cd7c91dd26f7667a7b938 --device virtio-fs,sharedDir=/var/folders,mountTag=a0bb3a2c8b0b02ba5958b0576f0d6530e104 --restful-uri tcp://localhost:49345 --device virtio-gpu,width=800,height=600 --device virtio-input,pointing --device virtio-input,keyboard --gui --device virtio-vsock,port=1024,socketURL=/Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-ignition.sock,listen]
DEBU[0013] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/krunkit-debug.sh is 72
DEBU[0013] ignition socket device: /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-ignition.sock

If I switch to podman v5.2.5 I'm able to create and start podman machines

[benoitf]

it seems related to containers/krunkit#17 or any other fixes made in this repository as I'm able to start using a version compiled from the main branch

[Luap99]

Starts fine on my system but I don't have macos 15 installed.

cc @jakecorrenti @slp @baude

[benoitf]

@Luap99 yes it does not seem to affect all the person using macOS

But with 5.2.5 it works on my computer and not with 5.3.0 using krunkit

[jakecorrenti]

Taking a look

[jakecorrenti]

I'm unable to reproduce by installing 5.3 from the installer on macOS 15.1. I'm still able to start a machine with krunkit v0.1.3 and podman 5.3

[jakecorrenti]

@benoitf have you tried using the upstream main branch of krunkit to see if that solved your issue? There's been a few commits to deal with memory limits and it will provide a --krun-log-level flag to get some more information

[benoitf]

it seems the issue is with machine with a lot of memory (so not all computers are affected)

and yes I'm able to start with the main branch krunkit

without the latest patch (fixing some memory issues), but the patch with krun-log-level

I have

[2024-11-14T13:25:58Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x102674000 guest_addr=0 len=8000000
[2024-11-14T13:25:58Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x300000000 guest_addr=40000000 len=80000000
[2024-11-14T13:25:58Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x7000000000 guest_addr=100000000 len=1800000000
[2024-11-14T13:25:58Z ERROR krun] Building the microVM failed: Internal(Vm(SetUserMemoryRegion(MemoryMap)))
Error: unable to begin running krun workload

[benoitf]

@jakecorrenti this is what I was referencing there: #24559 (comment)

[jakecorrenti]

I missed that comment. Seems like we might need to make a krunkit 0.1.4

[step21]

Thanks for fixing. Do you know if this is already available? Should podman-desktop update to it automatically soon or can/should I update manually?

[Luap99]

We will create a new 5.3.1 release with the fix included likely on Thursday, then you and/or podman-desktop need to update to that version.

[benoitf]

5.3.1 release will be released as part of Podman Desktop 1.15 (but you'll be to find nightly releases of Podman Desktop including it as soon as 5.3.1 is published. Or you will be able to use the installer from Podman directly)

For now either you need to rollout to podman v5.2.5 if you're facing the issue or upgrade manually to krunkit 0.1.4

workaround:

grab krunkit 0.1.4 from https://github.com/containers/krunkit/releases/tag/v0.1.4

copy the file to /opt/podman/bin/krunkit

at your own risk, reset the entitlements/signature to make macOS happy

(clone first https://github.com/containers/krunkit to ~/git/containers/krunkit/ and then)

sudo codesign --deep  --entitlements ~/git/containers/krunkit/krunkit.entitlements --sign - /opt/podman/bin/krunkit
sudo xattr -r -d com.apple.quarantine /opt/podman/bin/krunkit

then the command CONTAINERS_MACHINE_PROVIDER=libkrun podman machine init --now my-machine will work

[step21]

thanks @benoitf @Luap99

[step21]

Thanks, though I think I'm gonna wait for now. Is it possible that the krunkit from releases was compiled for macos sdk 15 or so? I am still on sonoma and it gave an error like that it required a newer os. I managed to compile krunkit manually with libkrun-efi from homebrew, but on starting it, it doesn't find libvirgl and its other libs. But no worries I can wait.

[benoitf]

@step21 you could try now https://github.com/containers/podman/releases/tag/v5.3.1