Running "podman system reset" deleted my entire $HOME directory #18295

gg7 · 2023-04-20T14:41:50Z

gg7
Apr 20, 2023

Issue Description

podman info was reporting that the vfs storage driver was in use, even though my kernel is new enough (6.1) and I have /usr/bin/fuse-overlayfs present.

From what I read that's because I already had vfs layers, so I ran podman system reset as my own (non-root) user, which worked mostly okay (I couldn't get the default to be overlay so I had to create ~/.config/containers/storage.conf, but that's for another ticket I guess).

Then I decided to also run podman system reset as root because I remember running a few containers as root years ago. As a result, podman proceeded to delete the entire /root/ folder.

Steps to reproduce the issue

I have been able to reproduce the issue. For example, let's create /root/.ssh/ and run podman system reset:

gentoo ~ # ls -alh
total 16K
drwxr-xr-x  3 root root 4.0K 2023-04-20 12:39:49 ./
drwxr-xr-x 21 root root 4.0K 2023-04-20 12:35:32 ../
drwxr-xr-x  2 root root 4.0K 2023-04-20 12:39:49 .ssh/
-rw-------  1 root root 1.2K 2023-04-20 12:39:49 .viminfo
hostname ~ # podman system reset                                                                           
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
Are you sure you want to continue? [y/N] y
gentoo ~ # ls -alh
total 0
gentoo ~ # pwd
/root
gentoo ~ # ls -al /root                                                                                  
total 48K
drwxr-xr-x 10 root root 4.0K 2023-04-20 12:43:10 ./
drwxr-xr-x 21 root root 4.0K 2023-04-20 12:40:12 ../
drwxr-xr-x  3 root root 4.0K 2023-04-20 12:40:12 .config/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 mounts/
drwx------  3 root root 4.0K 2023-04-20 12:43:10 overlay/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 overlay-containers/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 overlay-images/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 overlay-layers/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 overlay-locks/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 tmp/
-rw-r--r--  1 root root    3 2023-04-20 12:43:10 defaultNetworkBackend
-rw-r--r--  1 root root   64 2023-04-20 12:43:10 storage.lock
-rw-r--r--  1 root root    0 2023-04-20 12:43:10 userns.lock
gentoo ~ #

Maybe /root/ is intact, but podman mounted something on top of it? Unfortunately that's not the case:

gentoo ~ # mount | grep root
gentoo ~ #

Describe the results you received

podman deleted /root/.

Describe the results you expected

I expected podman to delete all containers/pods/images/networks/build cache/machines/volumes, but not my /root/ folder.

podman info output

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: app-containers/conmon-2.0.30
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: v2.0.30'
  cpuUtilization:
    idlePercent: 98.95
    systemPercent: 0.66
    userPercent: 0.38
  cpus: 6
  distribution:
    distribution: gentoo
    version: "2.13"
  eventLogger: journald
  hostname: gentoo
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.22-gentoo-dist
  linkmode: dynamic
  logDriver: journald
  memFree: 45350371328
  memTotal: 67280412672
  networkBackend: cni
  ociRuntime:
    name: crun
    package: app-containers/crun-1.4.5
    path: /usr/bin/crun
    version: |-
      crun version 1.4.5
      commit: c381048530aa750495cf502ddb7181f2ded5b400
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.2.0
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.4
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 4h 9m 25.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /root
  graphRootAllocated: 498425044992
  graphRootUsed: 203847122944
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /root
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1679773711
  BuiltTime: Sat Mar 25 19:48:31 2023
  GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915
  GoVersion: go1.20.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Bare metal.

Additional information

AFAIK I am using the defaults as far as storage is concerned:

gentoo ~ # ls -alh /etc/containers/
total 44K
drwxr-xr-x   3 root root 4.0K 2023-04-20 12:34:10 ./
drwxr-xr-x 115 root root  12K 2023-04-20 14:33:22 ../
drwxr-xr-x   2 root root 4.0K 2023-03-25 20:00:01 registries.d/
-rw-r--r--   1 root root 4.6K 2023-04-20 11:17:15 policy.json
-rw-r--r--   1 root root 4.7K 2023-03-25 19:49:34 policy.json.example
-rw-r--r--   1 root root  959 2023-04-20 11:17:58 registries.conf
-rw-r--r--   1 root root  883 2023-03-25 19:49:34 registries.conf.example

In my opinion it's highly surprising for podman to delete /root using out-of-the-box configuration.

Luap99 · 2023-04-20T15:04:16Z

Luap99
Apr 20, 2023
Maintainer

graphRoot: /root

Podman was configured somehow to use /root. A podman system reset will delete the graph root so this eplains why /root is deleted.

Do you have a /usr/share/containers/storage.conf file?

0 replies

gg7 · 2023-04-20T16:57:01Z

gg7
Apr 20, 2023
Author

Do you have a /usr/share/containers/storage.conf file?

I do not:

gentoo ~ # stat /usr/share/containers/storage.conf
stat: cannot statx '/usr/share/containers/storage.conf': No such file or directory
gentoo ~ #

The only storage.conf I have is for my own user (the one I created earlier today):

gentoo ~ # find / -name storage.conf
/home/gg7/.config/containers/storage.conf
gentoo ~ #

0 replies

Luap99 · 2023-04-20T17:00:14Z

Luap99
Apr 20, 2023
Maintainer

Do you have something under /var/lib/containers/, if you do please delete it and try to run podman info again it should default to this directly as storage location normally.

0 replies

gg7 · 2023-04-20T17:54:33Z

gg7
Apr 20, 2023
Author

Do you have something under /var/lib/containers/?

Yes, I do:

gentoo ~ # find /var/lib/containers/
/var/lib/containers/
/var/lib/containers/cache
/var/lib/containers/.keep_app-containers_podman-0
/var/lib/containers/storage
/var/lib/containers/storage/tmp
/var/lib/containers/storage/libpod
/var/lib/containers/storage/libpod/bolt_state.db

if you do please delete it and try to run podman info again it should default to this directly as storage location normally.

That works, thank you:

gentoo ~ # rm -r /var/lib/containers/cache /var/lib/containers/storage
gentoo ~ # find /var/lib/containers/
/var/lib/containers/
/var/lib/containers/.keep_app-containers_podman-0
gentoo ~ # podman info | grep graphRoot:
  graphRoot: /var/lib/containers/storage
gentoo ~ #

Looks like podman info has recreated /var/lib/containers/storage:

gentoo ~ # find /var/lib/containers/
/var/lib/containers/
/var/lib/containers/.keep_app-containers_podman-0
/var/lib/containers/storage
/var/lib/containers/storage/mounts
/var/lib/containers/storage/tmp
/var/lib/containers/storage/overlay
/var/lib/containers/storage/overlay/l
/var/lib/containers/storage/overlay/.has-mount-program
/var/lib/containers/storage/defaultNetworkBackend
/var/lib/containers/storage/libpod
/var/lib/containers/storage/libpod/bolt_state.db
/var/lib/containers/storage/storage.lock
/var/lib/containers/storage/userns.lock
/var/lib/containers/storage/overlay-layers
/var/lib/containers/storage/overlay-layers/layers.lock
/var/lib/containers/storage/overlay-images
/var/lib/containers/storage/overlay-images/images.lock
/var/lib/containers/storage/overlay-containers
/var/lib/containers/storage/overlay-containers/containers.lock

Even though I have fixed this particular computer, in my opinion this is still a bug which should be fixed. I have not created anything under /var/lib/containers/ by hand -- it must have been an earlier podman run. podman deleted /root through no fault of my own and this can happen to others too.

I would break down the issue as:

bug 1 (root cause): podman picks a suboptimal default graphRoot (at the very least polluting root's home directory) just because /var/lib/containers/ is partially initialized (because /var/lib/containers/cache exists?)
bug 2: podman system reset did not delete /var/lib/containers/cache and /var/lib/containers/storage for me. That command is also meant to delete $HOME/.config/containers/storage.conf , right? If it deletes config files, why not clean up the default location as well?
bug 3: podman system reset deleted the whole /root folder, instead of /root/{storage,cache,...} (say, if someone had manually set graphRoot to /root). Why isn't the deletion code more defensive? If someone sets graphRoot: /, this will nuke their whole system. At the very least the warning message can print which directory will be deleted.

Just my two cents, I am not a podman expert. And thanks again for the help!

1 reply

Luap99 Apr 21, 2023
Maintainer

What I do not understand it is how it is possible to have the libpod directory under /var/lib/containers but the graphroot from the storage was set to /root.

How old is your system, it is possible that there was a bug in some ancient podman version which caused this.

I think what we can do is to also display the list of directories that will be deleted on reset.

rhatdan · 2023-04-20T20:57:35Z

rhatdan
Apr 20, 2023
Maintainer

We have no idea how this happened or why it would delete /root. If you created a reproducer we would certainly fix it. We have never heard of something like this happening. and have no idea how it happened if the root directory in storage.conf did not point at that directory.

0 replies

gg7 · 2023-08-14T23:55:45Z

gg7
Aug 14, 2023
Author

Mystery solved: #18349 (comment)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running "podman system reset" deleted my entire $HOME directory #18295

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Running "podman system reset" deleted my entire $HOME directory #18295

gg7 Apr 20, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Replies: 6 comments · 1 reply

Luap99 Apr 20, 2023 Maintainer

gg7 Apr 20, 2023 Author

Luap99 Apr 20, 2023 Maintainer

gg7 Apr 20, 2023 Author

Luap99 Apr 21, 2023 Maintainer

rhatdan Apr 20, 2023 Maintainer

gg7 Aug 14, 2023 Author

gg7
Apr 20, 2023

Replies: 6 comments 1 reply

Luap99
Apr 20, 2023
Maintainer

gg7
Apr 20, 2023
Author

Luap99
Apr 20, 2023
Maintainer

gg7
Apr 20, 2023
Author

Luap99 Apr 21, 2023
Maintainer

rhatdan
Apr 20, 2023
Maintainer

gg7
Aug 14, 2023
Author