Need podman to mount symbolic links for docker compatibility

[wnm3]

Issue Description

I have used /etc/synthetic.conf to create a symbolic link off root to a directory in my home directory. The net effect is to have:

ls -l / |grep store
lrwxr-xr-x   1 root  wheel    16 Oct  9 12:15 store -> Users/wnm3/store

as specified by the following (note: there is a tab separating the data in /etc/synthetic.conf:

cat /etc/synthetic.conf 
store	Users/wnm3/store

If I attempt to mount /store/pipeline using -v in the podman run command as follows:

docker run -e DPLAPIKEY=${DPLAPIKEY} -v ~/store/pipeline:/opt/ol/wlp/output/defaultServer/data --publish 9078:9078 --publish 9440:9440 --detach --name pipelinesvcs pipelinesvcs

It fails with Error: statfs /store/pipeline: no such file or directory even though the data are there (/store just is a symlink directory).

Steps to reproduce the issue

Steps to reproduce the issue

1. create a ~/store/pipeline directory on a MacOS machine (mine is M1 but this also fails for Intel) using commands: cd ~ ; mkdir -p store/pipeline
2. create a synlinked directory using /etc/synthetic.conf as shown above and reboot
3. confirm the directory exists using ls -l / |grep store
4. attempt to run a container using -v /store/pipeline:/store/pipeline (this should fail)
5. attempt to run the same container using -v ~/store/pipeline:/store/pipeline (this should work)

Describe the results you received

Error: statfs /store/pipeline: no such file or directory
And the container does not run

Describe the results you expected

The container runs without issue and the contents of /store/pipeline are visible in the container.

podman info output

Note: docker aliased to podman 

docker info
host:
  arch: arm64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 98.35
    systemPercent: 0.7
    userPercent: 0.95
  cpus: 1
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 2046
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.5.5-200.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 793878528
  memTotal: 1980387328
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9.2-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.aarch64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.aarch64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 72h 33m 53.00s (Approximately 3.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 3546521600
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.0
  Built: 1695839065
  BuiltTime: Wed Sep 27 14:24:25 2023
  GitCommit: ""
  GoVersion: go1.20.8
  Os: linux
  OsArch: linux/arm64
  Version: 4.7.0


docker version  
Client:       Podman Engine
Version:      4.7.0
API Version:  4.7.0
Go Version:   go1.21.1
Git Commit:   073183fe1723d7bda826b574437891976a958c65
Built:        Wed Sep 27 11:35:55 2023
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.7.0
API Version:  4.7.0
Go Version:   go1.20.8
Built:        Wed Sep 27 14:24:25 2023
OS/Arch:      linux/arm64

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

See details for setting up /store -> ~/store above

Additional information

This happens without fail. It is an incompatibility with docker. We need this to handle shell scripts used to start containers on various systems where additional disk space has been mounted off the root directory.

[rhatdan]

@ashley-cui @baude I think this is specific to Mac.

[baude]

if you ssh into the machine, does it look correct? I'm wondering if 9P does not honor it?

[wnm3]

No, when ssh'ed into the podman machine, there is no /store directory listed, however other symbolic links are shown:

ls -l /
total 17
drwxr-xr-x.   5 root   80  160 Oct  2 10:59 Users
lrwxrwxrwx.   3 root root    7 Oct  4 19:13 bin -> usr/bin
drwxr-xr-x.   8 root root 1024 Oct  9 12:05 boot
drwxr-xr-x.  16 root root 3560 Oct  9 12:51 dev
drwxr-xr-x.  85 root root 8192 Oct  9 12:05 etc
lrwxrwxrwx.   3 root root    8 Oct  4 19:13 home -> var/home
lrwxrwxrwx.   3 root root    7 Oct  4 19:13 lib -> usr/lib
lrwxrwxrwx.   3 root root    9 Oct  4 19:13 lib64 -> usr/lib64
lrwxrwxrwx.   3 root root    9 Oct  4 19:13 media -> run/media
lrwxrwxrwx.   3 root root    7 Oct  4 19:13 mnt -> var/mnt
lrwxrwxrwx.   3 root root    7 Oct  4 19:13 opt -> var/opt
lrwxrwxrwx.   3 root root   14 Oct  4 19:13 ostree -> sysroot/ostree
drwxr-xr-x.   6 root root  192 Oct  9 12:15 private
dr-xr-xr-x. 172 root root    0 Oct  9 12:51 proc
lrwxrwxrwx.   3 root root   12 Oct  4 19:13 root -> var/roothome
drwxr-xr-x.  38 root root 1000 Oct  9 12:51 run
lrwxrwxrwx.   3 root root    8 Oct  4 19:13 sbin -> usr/sbin
lrwxrwxrwx.   3 root root    7 Oct  4 19:13 srv -> var/srv
dr-xr-xr-x.  12 root root    0 Oct  9 12:51 sys
drwxr-xr-x.   4 root root   66 Oct  4 19:13 sysroot
drwxrwxrwt.  11 root root  220 Oct 13 10:19 tmp
drwxr-xr-x.  12 root root  155 Dec 31  1969 usr
drwxr-xr-x.  25 root root 4096 Oct  9 12:05 var

What I see from the command line:

ls -l /
total 12
drwxrwxr-x  81 root  admin  2592 Oct 10 20:01 Applications
drwxr-xr-x  68 root  wheel  2176 Oct  2 11:00 Library
drwxr-xr-x@ 10 root  wheel   320 Sep 16 09:28 System
drwxr-xr-x   5 root  admin   160 Oct  2 10:59 Users
drwxr-xr-x   3 root  wheel    96 Oct 12 09:47 Volumes
drwxr-xr-x@ 39 root  wheel  1248 Sep 16 09:28 bin
drwxr-xr-x   2 root  wheel    64 Feb 26  2022 cores
dr-xr-xr-x   4 root  wheel  5840 Oct  9 12:14 dev
lrwxr-xr-x@  1 root  wheel    11 Sep 16 09:28 etc -> private/etc
lrwxr-xr-x   1 root  wheel    25 Oct  9 12:15 home -> /System/Volumes/Data/home
drwxr-xr-x   7 root  wheel   224 May 24 15:57 opt
drwxr-xr-x   6 root  wheel   192 Oct  9 12:15 private
drwxr-xr-x@ 64 root  wheel  2048 Sep 16 09:28 sbin
lrwxr-xr-x   1 root  wheel    16 Oct  9 12:15 store -> Users/wnm3/store
lrwxr-xr-x@  1 root  wheel    11 Sep 16 09:28 tmp -> private/tmp
drwxr-xr-x@ 11 root  wheel   352 Sep 16 09:28 usr
lrwxr-xr-x@  1 root  wheel    11 Sep 16 09:28 var -> private/var

[baude]

inside the machine, does /Users/wnm3/store exist?

[wnm3]

Yes, of course. As stated in the initial problem statement, I can start the container mounting ~/store/pipeline but not using /store/pipeline because /store is created using /etc/synthetic.conf and while it appears as a symbolic link in ls I gather it must be different since podman doesn't see it.

[baude]

the problem here is how the mount between host and virtual machine is working. by default, we mount /Users/ to /Users/name on the virtual machine. In your case, you have a symlink at the root (/) level. But the host's / is not being shared into the virtual machine so it does not resolve.

does it work if you do something like:

podman machine init -v /Users:/Users -v /private:/private -v /var/folders:/var/folders -v /store:/store

[wnm3]

Yes, that worked. Thank you. Is there a way to make the machine initialize this way all the time (e.g., if I reboot)?

[rhatdan]

You can add the mounts to containers.conf file.

man containers.conf
...
MACHINE TABLE
...
       volumes=["$HOME:$HOME"]

       Host directories to be mounted as volumes into the VM by default.  Environment variables like $HOME as well as complete paths are supported for the source and destination. An optional third field :ro can be used to  tell  the
       container engines to mount the volume readonly.

       On Mac, the default volumes are:

       [ "/Users:/Users", "/private:/private", "/var/folders:/var/folders" ]