Podman Machine VM should not auto update and reboot without user interaction

[tmoschou]

Issue Description

We have made the switch from Docker Desktop for MacOS to Podman / Podman Desktop for nearly a year now. Unfortunately, on a too frequent basis, we experience issues with the stability of the Podman Machine VM (MacOS). Most I think can be attribute to breaking changes from pushed auto updates (via Zincati) in particular compatibility with docker / docker-compose.

Users are reporting they are having to either kill the VM if it gets into a bricked state (seems to be gvproxy socket unresponsive / can't ssh into machine) or destroy and recreate the VM, loosing any data/state.

Can I please suggest at a minimum Podman, via likely the ignition file, disables Zincati from auto updating the VM and rebooting while containers are in use and without prompt. https://coreos.github.io/zincati/usage/auto-updates/#disabling-auto-updates.

Examples of bugs impacted by Zincati auto updating and rebooting the local VM.

• Compose not working after upgrade to 4.6.2 #19930
• systemctl reboot inside podman machine clears volume mounts #15976
• Podman fails to start containers due to RLIMIT_NPROC being too high #18555
• (Seemingly?) undocumented default capabilities change in 4.4.0+ #17504

#19930 is particularly bad, I am advising users to rollback and/or use an older CoreOS ISO.

podman machine ssh
sudo rpm-ostree rollback
sudo systemctl disable --now zincati.service
sudo systemctl reboot

❯  podman version
Client:       Podman Engine
Version:      4.6.1
API Version:  4.6.1
Go Version:   go1.20.7
Git Commit:   f3069b3ff48e30373c33b3f5976f15abf8cfee20
Built:        Fri Aug 11 03:43:43 2023
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.6.2
API Version:  4.6.2
Go Version:   go1.20.7
Built:        Wed Sep 13 05:37:26 2023
OS/Arch:      linux/arm64

Steps to reproduce the issue

Steps to reproduce the issue

1. podman machine init
2. run some containers and workloads
3. wait for VM to reboot or get into a bad state

Describe the results you received

• local VM reboots
• gvproxy can be unresponsible
• podman machine ssh doesn't work
• containers are re-started
• host file mounts are lost

Describe the results you expected

local VM is not autoupdated / rebooted without user interaction

podman info output

host:
  arch: arm64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.98
    systemPercent: 0.01
    userPercent: 0.01
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 2033
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 503
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.4.15-200.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 7738359808
  memTotal: 8299999232
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/503/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.aarch64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/503/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.aarch64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 20h 7m 18.00s (Approximately 0.83 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 0
    stopped: 9
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 53082042368
  graphRootUsed: 23652081664
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 33
  runRoot: /run/user/503/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1694549246
  BuiltTime: Wed Sep 13 05:37:26 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/arm64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[afbjorklund]

• Update notification for the update of the podman machine image #14514
• Add some documentation on how podman machine is updated #11680

[baude]

@tmoschou thanks for the excellent write up ... very appreciated.

[cgwalters]

Persistently disabling auto updates is documented at https://coreos.github.io/zincati/usage/auto-updates/#disabling-auto-updates

Alternatively, one can keep zincati.service disabled by default, but only systemctl start it when desired.