Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-x86_64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable #16054

arcusfelis · 2022-10-05T09:19:28Z

podman fails to start

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-x86_64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable

Steps to reproduce the issue:

Stop machine.
Start machine.
Machine starts or not starts very unpredictably.

Describe the results you received:

Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-x86_64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable

Describe the results you expected:
Stuff to work? More descriptive error messages?

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: connection to bastion host (ssh://core@localhost:50422/run/user/502/podman/podman.sock) failed: dial tcp [::1]:50422: connect: connection refused

Output of podman info:

podman info
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: connection to bastion host (ssh://core@localhost:50422/run/user/502/podman/podman.sock) failed: dial tcp [::1]:50422: connect: connection refused

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

==> podman: stable 4.2.1 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/4.2.1 (178 files, 48.5MB) *
  Poured from bottle on 2022-09-15 at 22:39:56
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go-md2man ✘, [email protected] ✘
Required: qemu ✔
==> Options
--HEAD
        Install HEAD version
==> Caveats
zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Analytics
install: 24,813 (30 days), 64,356 (90 days), 211,186 (365 days)
install-on-request: 24,012 (30 days), 62,808 (90 days), 209,311 (365 days)
build-error: 1 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.): mac

The text was updated successfully, but these errors were encountered:

arcusfelis · 2022-10-05T09:27:38Z

I've removed old machine and created a new one. For a NEW working machine I have:

podman version
Client:       Podman Engine
Version:      4.2.1
API Version:  4.2.1
Go Version:   go1.18.6
Built:        Tue Sep  6 21:16:02 2022
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      4.2.1
API Version:  4.2.1
Go Version:   go1.18.5
Built:        Wed Sep  7 21:58:19 2022
OS/Arch:      linux/amd64

info

host:
  arch: amd64
  buildahVersion: 1.27.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: '
  cpuUtilization:
    idlePercent: 89.07
    systemPercent: 6.84
    userPercent: 4.08
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "36"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 502
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.19.12-200.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1683369984
  memTotal: 2066890752
  networkBackend: netavark
  
  ociRuntime:
    name: crun
    package: crun-1.6-2.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.6
      commit: 18cf2efbb8feb2b2f20e316520e0fd0b6c41ef4d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/502/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 1m 28.00s
  
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106825756672
  graphRootUsed: 2288799744
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/502/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.1
  Built: 1662580699
  BuiltTime: Wed Sep  7 21:58:19 2022
  GitCommit: ""
  GoVersion: go1.18.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.1

Luap99 · 2022-10-05T10:26:04Z

This error is coming from qemu, likely a bug in qemu and not podman.

DarcySail · 2022-10-28T05:00:35Z

I have met the same issue, but I didn't use podman, only qemu.

github-actions · 2022-11-28T00:07:30Z

A friendly reminder that this issue had no activity for 30 days.

Thomasdezeeuw · 2022-11-28T09:56:34Z

I recently also hit this issue, so it's still relevant (not stale).

rhatdan · 2022-11-28T21:48:58Z

Since this is a qemu bug there is little Podman can do.

yusufcakmakk · 2022-11-30T15:37:15Z

It was solved when I restarted the computer. You can restart your computer. I have no idea about the problem.

Thomasdezeeuw · 2022-11-30T16:05:59Z

It was solved when I restarted the computer. You can restart your computer. I have no idea about the problem.

For me it's also solved by waiting a little bit. It seems some process (podman or QEMU I don't know) is slower in shutdown than the Podman Desktop UI is indicating.

archiewx · 2022-12-06T15:22:25Z

This is very strange~

JavierValtierra · 2022-12-15T20:02:32Z

I had this same issue today, what I did to "solve" it is to kill the qemu process with a kill -9 PID command. Then, I run the podman machine start command and it started successfully.

barnysanchez · 2023-01-12T14:29:08Z

I had this same issue today, what I did to "solve" it is to kill the qemu process with a kill -9 PID command. Then, I run the podman machine start command and it started successfully.

this saved my day. I have experienced the problem multiple times, ending on having to reboot my Mac, but this did the trick.

oherrala · 2023-01-17T09:20:34Z

Today I also got into this problem. I tried to start podman machine and it gave me exit status 255. Next try was unsuccessfully and the podman machine start process just hanged and I killed it with ctrl-c. Third time I got the error mentioned in this issue (Error: qemu exited unexpectedly with exit code 1). After killing the /opt/homebrew/bin/qemu-system-aarch64 process from my machine, I could once again start podman machine.

The session with my shell:

% podman machine start                                                                                                                                            
Starting machine "podman-machine-default"                                                                                                                                                     
Waiting for VM ...                                                                             
Mounting volume... /Users/oherrala:/Users/oherrala                                             
Error: exit status 255

% podman machine start                                             
Starting machine "podman-machine-default"                                                      
Waiting for VM ... 

<I kept waiting and waiting and waiting, then hit CTRL-C to end the waiting>
^C

% podman machine start
Starting machine "podman-machine-default"                                                      
Waiting for VM ...                                                                             
Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-aarch64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable

% ps auxwww|grep podman
% kill 63230

% podman machine start 
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users/oherrala:/Users/oherrala

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command: 

        podman machine set --rootful

API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully

sirmspencer · 2023-03-01T22:24:31Z

It was a failed shutdown for me. Thanks @oherrala! More simply

% ps auxwww|grep podman
% kill 63230
% podman machine start

thmang82 · 2023-03-10T15:28:33Z

Just also run into this issue on an M1 Macbook. Started with podman not being able to do anything. The virtual machine did not react. Stopping the machine with "podman machine stop" did hang forever. Afterwards i killed the qemu process and started the machine again. Not very user friendly.

Any chance podman can detect a non-responsive vm? This would help a lot. Happens every x days after resuming from suspend unfortunately.

ctrought · 2023-03-11T07:45:59Z

Just also run into this issue on an M1 Macbook. Started with podman not being able to do anything. The virtual machine did not react. Stopping the machine with "podman machine stop" did hang forever. Afterwards i killed the qemu process and started the machine again. Not very user friendly.

Any chance podman can detect a non-responsive vm? This would help a lot. Happens every x days after resuming from suspend unfortunately.

Same here, also M1 macbook. Needs to restart podman machine several times a day due to frequent hanging.

heyvito · 2023-03-14T20:34:31Z

I don't know up to which level this is relevant (perhaps virtualisation-wise?) but I have noticed Docker for macOS also behaving like that after resuming from suspend on an M1, but not on Intel. Killing the process and starting the VM again (both on Docker and Podman) fixes the issue.

Edit: running macOS 13.2.1 (22D68)

hundehausen · 2023-03-16T19:40:52Z

I have the same problem on latest macOS

hawktang · 2023-04-24T04:28:36Z

Same problem M1

lihuanshuai · 2023-05-05T08:23:03Z

if your qemu is upgraded, try remove podman machine and initialize again

podman machine rm
podman machine init

tscuite · 2023-05-11T06:19:23Z

Occasionally this situation is required and the process needs to be manually killed

flomar77 · 2023-06-26T06:52:46Z

I get the same error on Intel mac

sneko · 2023-06-28T13:52:21Z

Except if I'm wrong your solution @lihuanshuai (#16054 (comment)) removes all images locally.

Any workaround to keep them?

(The initial issue is pretty annoying, it's been 3 times in a few weeks... wondering if it's due to my macos going inactive/sleep)

vrothberg · 2023-07-13T12:07:57Z

@deboer-tim, with #19210 I am unable to see any flake in stop+start. As reported in #18648, when doing the restart in the UI, are stop and start overlapping/running concurrently or do they run in sequence?

deboer-tim · 2023-07-13T12:15:27Z

They are in sequence. Initially I thought it was when I stopped and quickly restarted (because that's what I was doing at the time, and I couldn't always reproduce), but it doesn't appear to be timing related.

When I opened #18648 I thought there were times when I couldn't restart in the UI but it worked from CLI, but my memory is a bit foggy. Today I have a machine that's failing in both cases:

% podman machine start podman-machine-demo
Starting machine "podman-machine-demo"
Waiting for VM ...
Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-aarch64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable

If there's anything you want me to check feel free to ping.

vrothberg · 2023-07-13T12:16:41Z

Thanks, @deboer-tim !

Can you try running #19210?

deboer-tim · 2023-07-13T19:02:57Z

As per DMs, I get the same behaviour with one of my machines even with #19210. I did try to start it without first, but fails consistently now.

vrothberg · 2023-07-31T12:35:52Z

@deboer-tim, can you try again with the latest main branch?

After a failed start, we can run into (somehow inconsistent) states where the machine won't start because a previous QEMU process is still running and the PID file is being used. Stop didn't resolve the issue as this state wasn't detected. Allow to recover from this state by a) detecting it during start and error out with a more helpful message than the error QEMU would otherwise spit out, and b) by enabling stop to kill the dangling QEMU process - even after a failed stop. With the changes, a recovery may look as follows: ``` _ podman git:(main) _ ./bin/darwin/podman machine start Starting machine "podman-machine-default" Error: cannot start VM "podman-machine-default": another instance of "/opt/homebrew/bin/qemu-system-aarch64" is already running with process ID 970: please stop and restart the VM _ podman git:(main) _ ./bin/darwin/podman machine stop Machine "podman-machine-default" stopped successfully _ podman git:(main) _ ./bin/darwin/podman machine start Starting machine "podman-machine-default" Waiting for VM ... ``` Please note that this change does not prevent us from running into such inconsistent states but only allows for recovering from them. [NO NEW TESTS NEEDED] - there is no reliable reproducer. Fixes: containers#16054 Signed-off-by: Valentin Rothberg <[email protected]>

chevdor · 2023-08-09T08:32:02Z

I ran into this issue as well today with:

podman version 4.6.0
QEMU emulator version 8.0.3 (qemu-system-x86_64 )

Here is a oneliner to solve the issue:

ps -edf | grep qemu-system | grep -v grep | awk '{print $2}' | xargs -I{} kill -9 {}; podman machine stop

You may then run podman machine start and keep rolling

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 5, 2022

Luap99 added the machine label Oct 5, 2022

github-actions bot added the stale-issue label Nov 28, 2022

github-actions bot removed the stale-issue label Nov 29, 2022

benoitf added the podman-desktop label May 31, 2023

This was referenced May 31, 2023

Mac error: qemu exited unexpectedly with exit code 1 podman-desktop/podman-desktop#2696

Closed

Error restarting podman machine too quickly #18648

Closed

slemeur mentioned this issue Jul 6, 2023

Impossible to restart a podman machine podman-desktop/podman-desktop#3127

Closed

vrothberg mentioned this issue Jul 31, 2023

Error: dial unix /var/folders/.../T/podman/qmp_podman-machine-default.sock: connect: no such file or directory #18410

Closed

vrothberg mentioned this issue Aug 1, 2023

machine: QEMU: recover from failed start #19469

Merged

openshift-merge-robot closed this as completed in #19469 Aug 2, 2023

benoitf mentioned this issue Aug 14, 2023

Rootless Podman machine can't restart when multiple machines exist podman-desktop/podman-desktop#3526

Closed

themr0c mentioned this issue Aug 17, 2023

docs: added _Recovering from a failed start_ to troubleshooting podman-desktop/podman-desktop#3579

Merged

benoitf mentioned this issue Sep 7, 2023

Podman machine restart action results to Restart Failed status with error: Cannot lock pid file when there is pod running podman-desktop/podman-desktop#3816

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Nov 8, 2023

github-actions bot locked as resolved and limited conversation to collaborators Nov 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-x86_64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable #16054

Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-x86_64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable #16054

arcusfelis commented Oct 5, 2022

arcusfelis commented Oct 5, 2022

Luap99 commented Oct 5, 2022

DarcySail commented Oct 28, 2022

github-actions bot commented Nov 28, 2022

Thomasdezeeuw commented Nov 28, 2022

rhatdan commented Nov 28, 2022

yusufcakmakk commented Nov 30, 2022

Thomasdezeeuw commented Nov 30, 2022

archiewx commented Dec 6, 2022

JavierValtierra commented Dec 15, 2022

barnysanchez commented Jan 12, 2023

oherrala commented Jan 17, 2023 •

edited

Loading

sirmspencer commented Mar 1, 2023

thmang82 commented Mar 10, 2023

ctrought commented Mar 11, 2023

heyvito commented Mar 14, 2023 •

edited

Loading

hundehausen commented Mar 16, 2023

hawktang commented Apr 24, 2023

lihuanshuai commented May 5, 2023

tscuite commented May 11, 2023

flomar77 commented Jun 26, 2023

sneko commented Jun 28, 2023

vrothberg commented Jul 13, 2023

deboer-tim commented Jul 13, 2023

vrothberg commented Jul 13, 2023

deboer-tim commented Jul 13, 2023

vrothberg commented Jul 31, 2023

chevdor commented Aug 9, 2023 •

edited

Loading

Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-x86_64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable #16054

Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-x86_64: cannot create PID file: Cannot lock pid file: Resource temporarily unavailable #16054

Comments

arcusfelis commented Oct 5, 2022

arcusfelis commented Oct 5, 2022

Luap99 commented Oct 5, 2022

DarcySail commented Oct 28, 2022

github-actions bot commented Nov 28, 2022

Thomasdezeeuw commented Nov 28, 2022

rhatdan commented Nov 28, 2022

yusufcakmakk commented Nov 30, 2022

Thomasdezeeuw commented Nov 30, 2022

archiewx commented Dec 6, 2022

JavierValtierra commented Dec 15, 2022

barnysanchez commented Jan 12, 2023

oherrala commented Jan 17, 2023 • edited Loading

sirmspencer commented Mar 1, 2023

thmang82 commented Mar 10, 2023

ctrought commented Mar 11, 2023

heyvito commented Mar 14, 2023 • edited Loading

hundehausen commented Mar 16, 2023

hawktang commented Apr 24, 2023

lihuanshuai commented May 5, 2023

tscuite commented May 11, 2023

flomar77 commented Jun 26, 2023

sneko commented Jun 28, 2023

vrothberg commented Jul 13, 2023

deboer-tim commented Jul 13, 2023

vrothberg commented Jul 13, 2023

deboer-tim commented Jul 13, 2023

vrothberg commented Jul 31, 2023

chevdor commented Aug 9, 2023 • edited Loading

oherrala commented Jan 17, 2023 •

edited

Loading

heyvito commented Mar 14, 2023 •

edited

Loading

chevdor commented Aug 9, 2023 •

edited

Loading