Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running Singularity in a user namespace created by an unprivileged user doesn't work #2698

Closed
stxue1 opened this issue Feb 29, 2024 · 5 comments · Fixed by #2699
Closed

Running Singularity in a user namespace created by an unprivileged user doesn't work #2698

stxue1 opened this issue Feb 29, 2024 · 5 comments · Fixed by #2699
Labels
bug Something isn't working

Comments

@stxue1
Copy link

stxue1 commented Feb 29, 2024

Version of Singularity
What version of Singularity are you using? Run:

singularity-ce version 4.1.1-jammy

Describe the bug
When making a namespace as an unprivileged user, either the UID in the namespace is mapped to 0 or something else. When the UID is mapped to 0 singularity thinks it is root, and tries to perform operations that it may not be permitted to do even if it's not technically root. If the UID is mapped to something else, Singularity says that it's not installed correctly as setuid means it expects to be root but isn't getting UID 0.

To Reproduce
Steps to reproduce the behavior:
To reproduce when UID is mapped to 0, the container can't be built:

unshare -r singularity -v -d run docker://ubuntu:latest
...
...
...
FATAL   [U=0,P=165134]     replaceURIWithImage()         Unable to handle docker://ubuntu:latest uri: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: etc/gshadow: apply hdr metadata: restore chown metadata: /tmp/build-temp-1529656932/rootfs/etc/gshadow: lchown /tmp/build-temp-1529656932/rootfs/etc/gshadow: invalid argument

Full logs here

To reproduce when UID is mapped to something else. The container is able to be built/unpacked but it can't execute:

unshare -c singularity -v -d run docker://ubuntu:latest
...
...
...
ERROR   [U=1000,P=162828]  init()                        No setuid installation found, for unprivileged installation use: ./mconfig --without-suid

Full logs here

Expected behavior
Singularity should be able to run the container.

OS / Linux Distribution
Which Linux distribution are you using?

NAME="Pop!_OS"
VERSION="22.04 LTS"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 22.04 LTS"
VERSION_ID="22.04"
HOME_URL="https://pop.system76.com"
SUPPORT_URL="https://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=jammy
UBUNTU_CODENAME=jammy
LOGO=distributor-logo-pop-os

Installation Method
Installed through the .deb file on the Github releases page.

Additional context
I'm trying get Singularity with FUSE mounts to run inside an unprivileged Docker container. In order to use FUSE inside an unprivileged docker container, I need to be inside a user namespace according to this issue, but Singularity doesn't work inside an unprivileged user namespace.

@stxue1 stxue1 added the bug Something isn't working label Feb 29, 2024
@dtrudg
Copy link
Member

dtrudg commented Mar 1, 2024

Please could you give a bit more context:

  • Are the failing commands run inside a Docker container, or are they run on the host at this point? If in a Docker container, do they work instead on the host?
  • What is the overall goal here... i.e. why do you need to use Singularity with FUSE mounts insides an unprivileged Docker container? Can you explain why singularity must be run inside an unprivileged Docker container, and what FUSE mounts are needed... do you mean Singularity's FUSE mounts or external FUSE mounts?

@dtrudg
Copy link
Member

dtrudg commented Mar 1, 2024

Did some quick digging, and may be able to solve this without additional context.

For the unshare -r case the following patch should solve the issue, and will be included in the next release:

diff --git a/internal/pkg/ociimage/unpack.go b/internal/pkg/ociimage/unpack.go
index fdbc132d7..55cbd0736 100644
--- a/internal/pkg/ociimage/unpack.go
+++ b/internal/pkg/ociimage/unpack.go
@@ -16,7 +16,9 @@ import (
        umocilayer "github.com/opencontainers/umoci/oci/layer"
        "github.com/opencontainers/umoci/pkg/idtools"
        "github.com/sylabs/singularity/v4/internal/pkg/util/fs"
+       "github.com/sylabs/singularity/v4/pkg/util/namespaces"
        "github.com/sylabs/singularity/v4/pkg/sylog"
+
 )
 
 // isExtractable checks if we have extractable layers in the image. Shouldn't be
@@ -69,7 +71,8 @@ func UnpackRootfs(_ context.Context, srcImage v1.Image, destDir string) (err err
        }
 
        // Allow unpacking as non-root
-       if os.Geteuid() != 0 {
+       insideUserNs, _ := namespaces.IsInsideUserNamespace(os.Getpid())
+       if os.Geteuid() != 0 || insideUserNs {
                mapOptions.Rootless = true
 
                uidMap, err := idtools.ParseMapping(fmt.Sprintf("0:%d:1", os.Geteuid()))

@dtrudg
Copy link
Member

dtrudg commented Mar 1, 2024

The unshare -c case is a separate issue.

With unshare -c you will need to use the -u flag for singularity - this is required to tell a setuid installation of singularity to run in usernamespace mode, instead of setuid mode.

unshare -c singularity run -u alpine_latest.sif

At present this is not working, as a wrapped extraction of an image is failing to honour the -u flag. I'll track this as a separate issue.

You can workaround this issue, at present, via creating a sandbox dir and then running with -u, e.g.:

$ singularity build -s test-sandbox docker://ubuntu:latest

$ unshare -c singularity run -u test-sandbox/
Singularity> echo "HELLO"
HELLO
Singularity> 

dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 1, 2024
If we are running from within a user namespace, then use rootless OCI
layer with umoci.

This permits the extraction to complete when singularity is run under
`unshare -r`.

Part of sylabs#2698
dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 1, 2024
If singularity is executed with `--userns/-u` then it should also use a
user namespace where it executes `unsquashfs` in a wrapped manner.

Previously the `unsquashfs` wrapping was without `--userns/-u` in a
setuid installation. This caused extraction to fail from within a
non-root-mapped user namespace (e.g. `unshare -c`).

Part of sylabs#2698
dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 1, 2024
If we are running from within a user namespace, then use rootless OCI
layer with umoci.

This permits the extraction to complete when singularity is run under
`unshare -r`.

Part of sylabs#2698
dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 1, 2024
If singularity is executed with `--userns/-u` then it should also use a
user namespace where it executes `unsquashfs` in a wrapped manner.

Previously the `unsquashfs` wrapping was without `--userns/-u` in a
setuid installation. This caused extraction to fail from within a
non-root-mapped user namespace (e.g. `unshare -c`).

Part of sylabs#2698
dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 1, 2024
If singularity is executed with `--userns/-u` then it should also use a
user namespace where it executes `unsquashfs` in a wrapped manner.

Previously the `unsquashfs` wrapping was without `--userns/-u` in a
setuid installation. This caused extraction to fail from within a
non-root-mapped user namespace (e.g. `unshare -c`).

Part of sylabs#2698
dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 1, 2024
If singularity is executed with `--userns/-u` then it should also use a
user namespace where it executes `unsquashfs` in a wrapped manner.

Previously the `unsquashfs` wrapping was without `--userns/-u` in a
setuid installation. This caused extraction to fail from within a
non-root-mapped user namespace (e.g. `unshare -c`).

Part of sylabs#2698
dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 1, 2024
If singularity is executed with `--userns/-u` then it should also use a
user namespace where it executes `unsquashfs` in a wrapped manner.

Previously the `unsquashfs` wrapping was without `--userns/-u` in a
setuid installation. This caused extraction to fail from within a
non-root-mapped user namespace (e.g. `unshare -c`).

Part of sylabs#2698
dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 1, 2024
If singularity is executed with `--userns/-u` then where possible it
should also use a user namespace where it executes `unsquashfs` in a
wrapped manner.

Previously the `unsquashfs` wrapping was without `--userns/-u` in a
setuid installation. This caused extraction to fail from within a
non-root-mapped user namespace (e.g. `unshare -c`).

Part of sylabs#2698
@stxue1
Copy link
Author

stxue1 commented Mar 2, 2024

Hi, thanks for the quick response!

* Are the failing commands run inside a Docker container, or are they run on the host at this point? If in a Docker container, do they work instead on the host?

I first encountered this issue when running the commands inside a Docker container, but was able to reproduce the exact same issue on a host machine. Neither the container or the host work, but the host seemed more easily reproducible so I included that as the bug report instead.

* What is the overall goal here... i.e. why do you need to use Singularity with FUSE mounts insides an unprivileged Docker container? Can you explain why singularity must be run inside an unprivileged Docker container, and what FUSE mounts are needed... do you mean Singularity's FUSE mounts or external FUSE mounts?

I'm trying to create a Docker container with an application preinstalled. The app is a workflow engine and uses Singularity as one of it's container backends to run tasks. I noticed that sandboxing images could take up a small amount of time for each task, so I wanted to see how much of a difference having Singularity mount with FUSE would be. In theory the container could be set up as privileged, which would allow it to work, but it would be better if that weren't the case.

I applied the patch and unshare -r and unshare -c work properly on my host machine and within the Docker container. Making Singularity use FUSE with unshare -c --keep-caps -m singularity -v -d run -u docker://hello-world does return an error on cleanup:

DEBUG   [U=1000,P=508312]  UnmountWithFuseLazy()         Executing FUSE unmount command: /usr/bin/fusermount3 -z -u /tmp/rootfs-3641030937/root
/usr/bin/fusermount3: failed to unmount /tmp/rootfs-3641030937/root: Invalid argument
ERROR   [U=1000,P=508312]  CleanupHost()                 While running host cleanup tasks: while unmounting fuse directory: /tmp/rootfs-3641030937/root: exit status 1
DEBUG   [U=1000,P=508312]  CleanupHost()                 Exiting CleanupHost - Cleanup failure
ERROR   [U=1000,P=508199]  Master()                      Unprivileged host cleanup failed: host cleanup failed

My guess is because Singularity tries to cleanup twice as the same unmount command appears above:

DEBUG   [U=1000,P=508311]  UnmountWithFuseLazy()         Executing FUSE unmount command: /usr/bin/fusermount3 -z -u /tmp/rootfs-3641030937/root

Log

This is probably a separate issue, but it doesn't break Singularity's functionality since the /tmp/rootfs-* folder is properly cleaned up. But everything looks to be working properly now, thanks!

@dtrudg
Copy link
Member

dtrudg commented Mar 4, 2024

Thanks for the feedback. The fusermount error is indeed something that will need to be addressed differently / separately. I'll open a separate issue for that, and we can tackle it a bit later on.

dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 4, 2024
If we are running from within a user namespace, then use rootless OCI
layer with umoci.

This permits the extraction to complete when singularity is run under
`unshare -r`.

Part of sylabs#2698
dtrudg added a commit to dtrudg/singularity that referenced this issue Mar 4, 2024
If singularity is executed with `--userns/-u` then where possible it
should also use a user namespace where it executes `unsquashfs` in a
wrapped manner.

Previously the `unsquashfs` wrapping was without `--userns/-u` in a
setuid installation. This caused extraction to fail from within a
non-root-mapped user namespace (e.g. `unshare -c`).

Part of sylabs#2698
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants