chown/chmod on mounted directory: Permission denied

[solarkraft]

Hey, I really like lima so far and wish I could use it for development, but unfortunately I've come across a workflow-breaking issue.

My docker image copies files into a mounted directory and changes their ownership to the custom user 1000. The copying works just fine, but the chown fails with Permission denied.

This issue can be confirmed in a shell; chmod fails in the same way.

root@wiki:/# chown 1000 /app/
root@wiki:/# chown 1000 /usr/
root@wiki:/# chmod 777 /app/
root@wiki:/# chmod 777 /usr/
root@wiki:/# chown 1000 /config/
chown: changing ownership of '/config/': Permission denied
root@wiki:/# chmod 777 /config/
chmod: changing permissions of '/config/': Permission denied

It's the same for a custom mount inside of $HOME and a directory in /tmp/lima.

The files end up with
drwxrwxrwx 1 nobody nobody 128 Sep 9 13:17 php

Manually touched files get
-rw-r--r-- 1 nobody nobody 0 Sep 9 13:22 test

I'm on an M1 Macbook Pro with macOS 12 and installed QEMU using the provided shell file. The same setup just works with Docker Desktop.

[AkihiroSuda]

This is a known limitation of the current sshfs server, which works without the root privilege.

[solarkraft]

Thanks for the info! Is it going to be resolved when VirtFS is built in? Is running rootfully a workaround and is it documented somewhere? Starting a rootful VM works, but sudo lima nerdctl compose up fails with FATA[0000] must not run as the root.

[abiosoft]

I am actually able to modify mounted directories as long as the directory is writeable by the host's macOS user.
I'll give your docker image a try.

Sorry, I didn't see your most recent comment #231 (comment)

[abiosoft]

but sudo lima nerdctl compose up fails with FATA[0000] must not run as the root.

It should actually be lima sudo nerdctl compose up?

[solarkraft]

Thanks for the hint. Unfortunately doesn't resolve the permission issue.

[diepfote]

This is a known limitation of the current sshfs server, which works without the root privilege.
@AkihiroSuda

How could we resolve this with a quick fix (less security but more interop)? If I disable rootless containerd and enable rootful on the guest I can't use nerdctl:

FATA[0000] rootless containerd not running? (hint: use `containerd-rootless-setuptool.sh install` to start rootless containerd): stat /run/user/501/containerd-rootless: no such file or directory

OR

why are you using allow_root instead of allow_other

lima/pkg/hostagent/mount.go

Line 51 in 8092199

SSHFSAdditionalArgs: []string{"-o", "allow_root"},

?

Thanks for taking the time :)

[AkihiroSuda]

If we could get https://github.com/afbjorklund/qemu/commits/9p-darwin merged into the upstream, we will be able to store the file owner info into xattr (or a separate file). https://wiki.qemu.org/Documentation/9psetup (mapped-xattr, mapped-file)

I'm not sure we will use 9p by default, though, due to a performance issue.

[diepfote]

Thanks.

Still not quite sure why the sshfs option allow_other would not be an option.
From the manpage:

   FUSE options:
       -d -o debug
              enable debug output (implies -f)

       -f     foreground operation

       -s     disable multi-threaded operation

       -o allow_other
              allow access to other users

       -o allow_root
              allow access to root

[diepfote]

or maybe a hacky way of doing it:

       -o uid=N
              set file owner

       -o gid=N
              set file group

[AkihiroSuda]

These flags have nothing to do with allowing chown/chmod.

[diepfote]

ahh this is how I confused you. I found this issue after looking for issues regarding "incorrect" host mount permissions.
My suggestion was merely about fixing permissions for host mounted dirs for non-root users in a container.

Would that solve it?

Sorry for the confusion.

[diepfote]

If I build lima myself to test it. Is there anything I need to take into account? Do I need to re-create the lima vm if I change these options? Or is a re-compile sufficient?

[abiosoft]

If I build lima myself to test it. Is there anything I need to take into account? Do I need to re-create the lima vm if I change these options? Or is a re-compile sufficient?

There is nothing special to take into account and you do not need to re-create the vm, it should just work like the binary you downloaded on github.

Volumes are mounted on startup, all you need to do is shutdown, and start the vm (using your custom build).

[fxposter]

Is there any way forward here? I get similar "permission denied" errors and it's not clear whether it can be fixed in current setup at all.

I am using lima with docker and regardless on whether I use rootless docker or rootful docker + sudo - I still get permission denied errors. Is it an sshfs limitation? or can we expect it to work in the future (ignoring 9p work).

[fxposter]

Clarification: I didn't get permission denied for chown itself for rootful docker installation. Instead I could not connect to docker via sock file created by this:

ssh -f -N -p 60006 -i ~/.lima/_config/user -o NoHostAuthenticationForLocalhost=yes -L $HOME/docker.sock:/run/user/$(id -u)/docker.sock 127.0.0.1

I needed to run docker open via 127.0.0.1:2375 instead of using a socket file. and now it works fine except for some problems related to docker image and systemd integration itself, but this is out of scope of this ticket, so you can ignore my last comment.

[chrisdostert]

Any updates or ideas on a plan to solve this? chown / chmod on mounted files is pretty commonly needed and likely a blocker for a fair number of folks.

[AkihiroSuda]

Any updates or ideas on a plan to solve this? chown / chmod on mounted files is pretty commonly needed and likely a blocker for a fair number of folks.

The plan is to use mapped-xattr or mapped-file of virtio 9P, but the patch is not merged for macOS hosts yet, and seems to need more testers: NixOS/nixpkgs#122420

[willcohen]

As a followup, the latest version of the 9p patch is https://gitlab.com/wwcohen/qemu/-/tree/9p-darwin and that's where the in-progress work will go as it progresses towards resubmission upstream. Any comments on how to improve would be GREATLY welcomed before I submit again.

[mritd]

Sorry, after reading the discussion above I still don't quite understand...

How can I solve this problem when I use sshfs? I installed the latest HEAD branch using brew, but I still get this error... How do I need to adjust the source code to fix this? (I'm just on my own For use on laptops, you don't need to consider too much security).

Sorry, it was my test error. Tips for others who need help:

• Use brew to install the master branch on mac: brew install lima --HEAD
• Install 7.0 version of qemu: brew install qemu(or brew upgrade qemu)
• Set 9p as mount driver in config file: mountType: 9p
• Set directory writable permission to true: writable: true
• Set 9p cache type to mmap: cache: "mmap"

Finally it may be necessary to remove the previous mounted directory, eg ./data:/var/lib/mysql --> rm -rf ./data

Now recreate the lima vm, there is no more permission problem.

[mritd]

Latest test:

When using 9p for directory mounting, if the executable file in the docker image is copied to the host directory and executed in the container, the following error message may be generated:

bash exec... Connection reset by network

This may be a bug of qemu, you need to install qemu from master branch to fix it:

brew uninstall qemu --ignore-dependencies
brew install qemu --HEAD

[nulllpoint]

Sorry, after reading the discussion above I still don't quite understand...

How can I solve this problem when I use sshfs? I installed the latest HEAD branch using brew, but I still get this error... How do I need to adjust the source code to fix this? (I'm just on my own For use on laptops, you don't need to consider too much security).

Sorry, it was my test error. Tips for others who need help:

• Use brew to install the master branch on mac: brew install lima --HEAD
• Install 7.0 version of qemu: brew install qemu(or brew upgrade qemu)
• Set 9p as mount driver in config file: mountType: 9p
• Set directory writable permission to true: writable: true
• Set 9p cache type to mmap: cache: "mmap"

Finally it may be necessary to remove the previous mounted directory, eg ./data:/var/lib/mysql --> rm -rf ./data

Now recreate the lima vm, there is no more permission problem.

I use podman by lima-vm, container is mysql/mysql-server, it need chown mysql:mysql xxx, but when use -v mapping host's dir , propmpt chown: changing ownership of 'xxx': Operation not permitted
macos version: 12.6
lima version: 0.13.0
qemu version: 7.1.0
lima's yaml config:

mountType: 9p
mounts:
- location: "/Users/xxx/lima_workspace/podman"
  writable: true
  9p:
    cache: mmap

[mritd]

@nulllpoint In a recent release, the securityModel option changed its default value; you may need to specify it manually for mapped-xattr.

[nulllpoint]

@nulllpoint In a recent release, the securityModel option changed its default value; you may need to specify it manually for mapped-xattr.

The above is useful, the problem was solved.
This problem has bothered me for almost two days, hahhhh....
Thank you for your busy reply.

[swalkinshaw]

For another data point, I'm experiencing the same issue with the new vz driver and the virtiofs mount type.

[maoxuner]

@nulllpoint In a recent release, the securityModel option changed its default value; you may need to specify it manually for mapped-xattr.

I'm using colima 0.4.6 with lima 0.13.0 qemu 7.1.0. It seems 9p options are not supported in colima 😢

---

@mritd By the way, happy to see you here. We have exchanged friend links before.

[mritd]

@pawmaster 😆😆😆

off topic, in fact you can directly replace colima with lima:

# This example requires Lima v0.8.0 or later
images:
# Hint: run `limactl prune` to invalidate the cache
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"

arch: "aarch64"
cpus: 8
memory: "16G"
disk: "100G"

mountType: "9p"
mounts:
- location: "~"
  writable: true
  9p:
    securityModel: mapped-xattr
    cache: "mmap"
- location: "/opt/composes"
  writable: true
  9p:
    securityModel: mapped-xattr
    cache: "mmap"
- location: "/tmp/lima"
  writable: true
  9p:
    securityModel: mapped-xattr
    cache: "mmap"
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false
provision:
- mode: system
  # This script defines the host.docker.internal hostname when hostResolver is disabled.
  # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
  # Names defined in /etc/hosts inside the VM are not resolved inside containers when
  # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
  script: |
    #!/bin/sh
    sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    if command -v docker >/dev/null 2>&1; then
      docker run --platform=linux/arm64 --privileged --rm tonistiigi/binfmt --install all
      exit 0
    else
      if [ ! -e /etc/systemd/system/docker.socket.d/override.conf ]; then
        mkdir -p /etc/systemd/system/docker.socket.d
        # Alternatively we could just add the user to the "docker" group, but that requires restarting the user session
        cat <<-EOF >/etc/systemd/system/docker.socket.d/override.conf
    [Socket]
    SocketUser=${LIMA_CIDATA_USER}
    EOF
      fi
      export DEBIAN_FRONTEND=noninteractive
      curl -fsSL https://get.docker.com | sh
    fi
probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
      echo >&2 "docker is not installed yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until pgrep dockerd; do sleep 3; done"; then
      echo >&2 "dockerd is not running"
      exit 1
    fi
  hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
  # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
  # resolve inside containers, and not just inside the VM itself.
  hosts:
    host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
  To run `docker` on the host (assumes docker-cli is installed), run the following commands:
  ------
  docker context create aarch64 --docker "host=unix://{{.Dir}}/sock/docker.sock"
  docker context use aarch64
  ------
networks:
# The instance can get routable IP addresses from the vmnet framework using
# https://github.com/lima-vm/vde_vmnet. Available networks are defined in
# $LIMA_HOME/_config/networks.yaml. Supported network types are "host",
# "shared", or "bridged".
- lima: shared

[maoxuner]

@mritd
Thank you for your advice. Currently, I use lima .lima/_config/override.yaml file to override volume settings. I'm not familiar with lima nor colima. Since colima is a out of box plan, I prefer using colima now. Maybe try native lima later.

[qbx2]

Try posixovl with -F.