Support native source folder for volume mount in remote model (#8016)

[fogfish]

The discussion about an issue #8016

[aslilac]

Can we lock that issue, now that we have a discuss to talk in instead?

[baude]

done

[vanyasvl]

I have an issue with inline sed on 9p virtfs mount under MacOS with latest podman:

sed -i 's/aaa/zzz/g' /Users/user/q/123
sed: preserving permissions for ‘/Users/user/q/sedPtcM87’: Protocol not supported

Mount on podman machine:

vol0 on /Users/user type 9p (rw,relatime,sync,dirsync,access=client,trans=virtio)

part of strace:

read(3, "qqq\nzzz\nsss\n", 8192)        = 12
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "", 8192)                       = 0
fchown(4, 501, 1000)                    = 0
fgetxattr(3, "system.posix_acl_access", 0x7ffd6dbd18b0, 132) = -1 ENODATA (No data available)
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=12, ...}, AT_EMPTY_PATH) = 0
fsetxattr(4, "system.posix_acl_access", "\2\0\0\0\1\0\6\0\377\377\377\377\4\0\4\0\377\377\377\377 \0\4\0\377\377\377\377", 28, 0) = -1 EPROTONOSUPPORT (Protocol not supported)
fsetxattr(4, "system.posix_acl_access", "\2\0\0\0\1\0\6\0\377\377\377\377\4\0\4\0\377\377\377\377 \0\4\0\377\377\377\377", 28, 0) = -1 EPROTONOSUPPORT (Protocol not supported)

Issue is in fsetxattr syscall

[fogfish]

Thank you for reporting it! I can reproduce it as well. It seems to be well know issue at QEMU https://lists.gnu.org/archive/html/qemu-devel/2020-12/msg02655.html

Let's re-open an issue, otherwise it would not be resolved.

[shanesmith]

What is the expected ownership of bind mounted volumes in a container?

I would expect it would that that of the container's USER (or --user flag in podman run), but I see that the macOS user is kept instead in a rootful container.

In the example below, the file is owned by shane (501) on macOS and that is reflected in the Podman container, unlike Docker which sets the owner to root.

$ ls -la adir
total 0
drwxr-xr-x   3 shane  staff   96  6 Apr 13:21 .
drwxr-xr-x  16 shane  staff  512  6 Apr 13:14 ..
-rw-r--r--   1 shane  staff    0  6 Apr 13:20 afile

$ podman machine init -v $HOME:$HOME --now

$ podman -c podman-machine-default-root run -v $PWD/adir:/adir alpine ls -la /adir
total 0
drwxr-xr-x    3 501      dialout         96 Apr  6  2022 .
dr-xr-xr-x    1 root     root            40 Apr  5 12:02 ..
-rw-r--r--    1 501      dialout          0 Apr  6  2022 afile

$ docker run -v $PWD/adir:/adir alpine ls -la /adir
total 4
drwxr-xr-x    3 root     root            96 Apr  6 17:21 .
drwxr-xr-x    1 root     root          4096 Apr  6 17:25 ..
-rw-r--r--    1 root     root             0 Apr  6 17:20 afile

[shanesmith]

@baude sorry to be pinging you directly, but I wonder if you have any thoughts on the above? =)

[baude]

@shanesmith well docker has a significant advantage here in that they can just assume root for everything. we do not have the luxury. we def fiddle with permissions to make them usable by the core fcos user int he vm, and like you observed, we try to set the uid (number) to the same as the macos uid, or you run into neverending permissions errors. Perhaps it might be best if you were to say what is wrong with our approach and we see about making you happy?

[ssbarnea]

Apparently sudo setenforce 0 is not enough as I still seen the error.

I am starting to believe that this might be a bug in podman where it does attempt to use lsetxattr on mounts that do not support it. But at the same time, I did not see the same type of error when I mounted $HOME.

[rhatdan]

setenforce 0, would not change the behaviour. If Podman thinks it needs to do SELinux actions of setting labels, it will do it in enforcing mode and permissive mode, and still fail if it is not allowed. I just need to see where it is reporting failure.

[ssbarnea]

You are right, it was not selinux. It was containers/podman-compose#509 -- which for some reason seems closed, while I do not see it as resolved.

The presence of :Z would produce this. Once I removed the :Z, it started to work just fine.

[ssbarnea]

I proposed a workaround for ansible-runner at ansible/ansible-runner#1146 but you can easily see how this would not scale well. What about other tools that might try to do the same.

I am wondering if we could neuter the :Z on macos, so we would accept the label without doing anything about it. Arguably we could print a debugging message saying that we are ignoring the label as not being supported. WDYT?

[rhatdan]

I think the best could do is drop it to logrus.Warnf when ENOSUP.

[gaba-xyz]

I'm seeing very poor I/O performance on the mounted volume inside of containers. I've posted benchmarks from my MBP i7 below. Is this expected?

host

$ dd if=/dev/zero of=$HOME/Documents/zeroes_host.bak count=1M
1048576+0 records in
1048576+0 records out
536870912 bytes (512.0MB) copied, 5.220928 seconds, 103 MB/s

container

$ dd if=/dev/zero of=/app/zeroes_container.bak count=1M
1048576+0 records in
1048576+0 records out
536870912 bytes (512.0MB) copied, 280.546129 seconds, 1.8MB/s

[samskiter]

Is there a solution to this issue identified yet? I'm seeing it on macos. is it a qemu issue? do i need to change my volume mount setup?

[bernhardriegler]

In the issue it is discussed that volumes need to be setup on init of the podman machine.

see #8016

also https://medium.com/@butkovic/favoring-podman-over-docker-desktop-33368e031ba0

[samskiter]

Thanks so much for the reply, i missed that! Is there a resolution for this bug coming?

[samskiter]

(not that I was seeing the mounting of volumes partially working but for some files was getting either permission denied or something like this error from above: preserving permissions for ‘/Users/user/q/sedPtcM87’: Protocol not supported

[samskiter]

in my case it happens during a mv command. The file is moved but the permissions presumably aren't correctly set.

[vanyasvl]

sed -i also doesn't work due to Protocol not supported error

[samskiter]

It seems this is related back to qemu and 9pfs https://gitlab.com/qemu-project/qemu/-/issues/944

[arduino-man]

I am being affected by this issue as well. Was very happy to move to Podman from Docker but this issues might just be what makes me go back :(

[ssbarnea]

Here is something to think about, please refrain from making comments that do not add any value to the discussion, they only spam and alienate others at best, making less likely to want to engage in finding solutions for the identified problems. If you just like/agree or disagree/dislike something, use the buttons, not comments.

We could try to help others by hiding some useless comments but is draining. Yes, I am frustrated from time to time about the experience I get while using podman but I also know that podman team was one of the most open to suggestions and welcoming pull requests, very different than the other solution we are still competing with. Remember that podman is as free as it gets, you don't need a license or subscription to use it. But your time is too valuable for debugging these issue, pay someone to help you but don't troll others.

Here is my current guide on podman tricks on macos: https://github.com/ansible/vscode-ansible/wiki/macos - remoting is not covered yet because I know is not working, also I know that is not really working even with docker. If you need mounting from local, using remoting is not suitable, performance wise anyway.. to easy to end-up wanting to mount gigabytes of data, even by mistake.

[pykenny]

@ssbarnea I saw the comment here, which says:

Is likely that most MacOS users would only want to mount from within their home directory, ...

Does the solution work on external drive, or any volume outside of home directory? Working on a MacBook with limited storage left, so ideally I want both the virtual machine and containers to use external drive for storage.

[pykenny]

Just saw the notes in the tutorial:

You will not be able to mount directories outside your MacOS home directory

So does that mean running Podman on MacOS with external drive is still (entirely?) not doable?

[rhatdan]

This is by default, you can add additional directories via the podman machine init command.

man podman-machine-init
...
   --volume, -v=source:target[:options]
       Mounts a volume from source to target.

       Create a mount. If /host-dir:/machine-dir is specified as the *source:target*, Podman mounts host-
       dir in the host to machine-dir in the Podman machine.

       Additional options may be specified as a comma-separated string. Recognized  options  are:  *  ro:
       mount  volume  read-only * rw: mount volume read/write (default) * security_model=[model]: specify
       9p security model (see below)

       The   9p   security    model    [determines]    https://wiki.qemu.org/Documentation/9psetup#Start‐
       ing_the_Guest_directly  if  and how the 9p filesystem translates some filesystem operations before
       actual storage on the host.

       In order to allow symlinks to work, on MacOS the default security model is
        none.

       The value of mapped-xattr specifies that 9p store symlinks and some file  attributes  as  extended
       attributes  on  the host. This is suitable when the host and the guest do not need to interoperate
       on the shared filesystem, but has caveats for actual shared access; notably, symlinks on the  host
       are  not usable on the guest and vice versa. If interoperability is required, then choose none in‐
       stead, but keep in mind that the guest will not be able to do things that  the  user  running  the
       virtual machine cannot do, e.g. create files owned by another user. Using none is almost certainly
       the best choice for read-only volumes.

       Example: -v "$HOME/git:$HOME/git:ro,security_model=none"

       Default volume mounts are defined in containers.conf.   Unless  changed,  the  default  values  is
       $HOME:$HOME.

[pykenny]

Tried mapping my external drive to VM's system root:

podman machine init -v="/Volumes/my_extern_drive_root:/"

➡️

Extracting compressed file
Image resized.
Machine init complete
To start your machine run:

	podman machine start

---

podman machine start

➡️

Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Volumes/my_extern_drive_root:/
WARN[0039] API socket failed ping test

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

	podman machine set --rootful

API forwarding listening on: /Users/pykenny/.local/share/containers/podman/machine/podman-machine-default/podman.sock

The system helper service is not installed; the default Docker API socket
address can't be used by podman. If you would like to install it run the
following commands:

	sudo /usr/local/Cellar/podman/4.3.1/bin/podman-mac-helper install
	podman machine stop; podman machine start

---

sudo /usr/local/Cellar/podman/4.3.1/bin/podman-mac-helper install

podman machine stop; podman machine start

➡️

Waiting for VM to exit...
Machine "podman-machine-default" stopped successfully
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Volumes/my_extern_drive_root:/
Error: exit status 255

🤷

• The VM is still in running status, even though getting error message from machine-start command (which means doing podman machine stop right after still works).
• Rebuild the VM (podman machine rm; podman machine init; podman machine start) seems to avoid the error instead of restart existing VM.
  But then you got this error on podman machine ssh:

  Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
  Fedora CoreOS 37.20230122.2.0
  Tracker: https://github.com/coreos/fedora-coreos-tracker
  Discuss: https://discussion.fedoraproject.org/tag/coreos
  
  cannot chdir to /var/home/core: No such file or directory
  

[pykenny]

Seems that --volume option only handles volume mapping, but doesn't deal with location to install and run the VM.

[pykenny]

Okay, so let's forget about the plan to move everything anything (including the VM) to remote drive, and try following the instructions in "Volumn" chapter of Podman in Action, with few modifications to fit Podman's remote mode (Podman 4.3.1, MacOS 13.2.1):

1️⃣ Map VM's $HOME/sandbox/podman directory to mines in the host system:

podman machine init -v "$HOME/sandbox/podman:$HOME/sandbox/podman"

(Reference: man podman-machine-init:)

   --volume, -v=source:target[:options]
       Mounts a volume from source to target.


       Create a mount. If /host-dir:/machine-dir is specified as the *source:target*, Podman
       mounts host-dir in the host to machine-dir in the Podman machine.

➡️

Extracting compressed file
Image resized.
Machine init complete
To start your machine run:

	podman machine start

---

2️⃣ Start VM:
Run podman machine start ➡️ 🆗

---

3️⃣ Run example image and map /var/www/html directory in the container, to VM's $HOME/sandbox/podman/html, which is supposed to be $HOME/sandbox/podman/html in my host system beneath.

podman run -d -v "$HOME/sandbox/podman/html:/var/www/html:ro,z" -p 8080:8080 quay.io/rhatdan/myimage

(Reference: man podman-run:)

   --volume, -v=[[SOURCE-VOLUME|HOST-DIR:]CONTAINER-DIR[:OPTIONS]]
       Create a bind mount. If -v /HOST-DIR:/CONTAINER-DIR is specified, Podman bind mounts
       /HOST-DIR from the host into /CONTAINER-DIR in the Podman container. Similarly, -v
       SOURCE-VOLUME:/CONTAINER-DIR will mount the named volume from the host into the
       container. If no such named volume exists, Podman will create one. If no source is
       given, the volume will be created as an anonymously named volume with a randomly
       generated name, and will be removed when the container is removed via the --rm flag
       or the podman rm --volumes command.


       (Note when using the remote client, including Mac and Windows (excluding WSL2)
       machines, the volumes will be mounted from the remote server, not necessarily the
       client machine.)

➡️ ⚡⚡⚡

Trying to pull quay.io/rhatdan/myimage:latest...
Getting image source signatures
Copying blob sha256:e3460238f8a1f4698e1ec867ff96682f5d45debdd10e0503742fd15124d8bf5b
Copying blob sha256:c7765172d3ce59f229d53f5c2a60346ad3922c29baa7ae19d31ef9866117d743
Copying blob sha256:dfd8c625d0226c52da48ce402e79bc6e60a360d732bb7f6523c62cb714ec0a0d
Copying blob sha256:2b782a9ad894d15e65ee92d0e294b8358cfc69d94bfd5b2cf8d5d286376a0f4a
Copying blob sha256:a1eadb69adf1f7b62f76fc7bc2d7f8c28e6c03dc1f6024a4f9fd1329412efc89
Copying config sha256:2c7e43d880382561ebae3fa06c7a1442d0da2912786d09ea9baaef87f73c29ae
Writing manifest to image destination
Storing signatures
Error: lsetxattr /(MY_$HOME)/sandbox/podman/html: operation not supported

[pykenny]

Reason for forgetting about all the stuff about using external drive:

Just saw one another message from @ssbarnea in the same document above:

You will not be able to mount directories outside your MacOS home directory, so don't try to open workspaces hosted on other disks or network drivers, as podman will not be able to mount them.

😭

[pykenny]

By default, podman-machine-start maps host system's $HOME to VM's $HOME unless we overwrite the configuration. However and unfortunately, the same error will still happen for the example above.

[pykenny]

Also, changing mount type before starting VM does not work as well with the same error message emitted

• Update: Also tried adding /private/tmp:/private/tmp and /var/folders/ in mount settings. Still out of luck.

[pykenny]

Tried upgrading Podman to 4.4.1 (brew upgrade podman), podman-machine-rm the VM, and finally redo all steps. Doesn't help.

[pykenny]

Finally, removing z flag (which MacOS doesn't support) in mount option works:

• [macOS] Error: for attach: setxattr #13631

[ssbarnea]

Based on my tests, installing 4.4.1 removing old podman machine and reinstalling it using default settings did add the expected 3 folders to the machine mount, the two macos temp ones and /Users, which should be fine for most users.

[pykenny]

Not my case, unfortunately. But seems that podman-run's documentation has changed a lot compared to its 4.3 companion. So maybe the configuration syntax has changed and I'm reading podman-run's manpage again now. (By the way, Podman in Action was written during the 4.1 era)

[pykenny]

Also the new manpage's broken on my console (compared to rendered HTML)

[pykenny]

Solved. Documentation should mention about that.

[awidjaja]

In my case, podman machine is initialized using podman desktop on Windows, so this workaround don't work. I use a separate WSL distribution for my local development environment. By adding podman connection setup, I can run podman remotely using that podman machine. But how can I mount my local development directory (not the host home dir) into the podman machine? I think the mounting should be done upon connection add, rather than machine init. If possible, the podman client should open a tunnel which expose the local directory contents for the podman machine to access remotely.

[ecerulm]

I was experiencing this problem today using podman 4.5.1 on macOS Ventura 13.4.

Restarting the podman machine should generally work , I can see that the default podman machine already mounts /Users, /private and /var/folders

podman machine stop
podman machine start 
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders
podman run -ti --rm -v "$HOME:/myhome" ubi8 
[root@ba2054e82401 /]# ls /myhome

If the podman machine it's not mounting those you can recreate and try again, but that deletes all your containers, images, etc.

podman machine stop
podman machine rm
podman machine init # optionally put init options here like --memory 8192
podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders
podman run -ti --rm -v "$HOME:/myhome" ubi8 
[root@ba2054e82401 /]# ls /myhome