Bind mounting nscd socket in the sandbox causes getpwuid to return the wrong user #4991

hmenke · 2021-07-07T11:57:18Z

Describe the bug

When building a fixed-output derivation querying the user with getpwuid (3) results in the user with the UID 1000 outside the sandbox.

Steps To Reproduce

The nscd socket is bind mounted into the sandbox when building a fixed output derivation. This is probably because nscd handles DNS requests and since the build has access to the network this functionality is desirable.

nix/src/libstore/build/local-derivation-goal.cc

Lines 1747 to 1760 in 6182ae6

    
                       if (derivationIsImpure(derivationType)) { 
        
                           // Only use nss functions to resolve hosts and 
        
                           // services. Don’t use it for anything else that may 
        
                           // be configured for this system. This limits the 
        
                           // potential impurities introduced in fixed-outputs. 
        
                           writeFile(chrootRootDir + "/etc/nsswitch.conf", "hosts: files dns\nservices: files\n"); 
        
                           /* N.B. it is realistic that these paths might not exist. It 
        
                              happens when testing Nix building fixed-output derivations 
        
                              within a pure derivation. */ 
        
                           for (auto & path : { "/etc/resolv.conf", "/etc/services", "/etc/hosts", "/var/run/nscd/socket" }) 
        
                               if (pathExists(path)) 
        
                                   ss.push_back(path); 
        
                       }

However, when we look at the manpage of nscd we find

Nscd provides caching for accesses of the passwd (5) , group (5) , and hosts (5) databases through standard libc interfaces, such as getpwnam (3) , getpwuid (3) , getgrnam (3) , getgrgid (3) , gethostbyname (3) , and others.

So the problem is that inside the sandbox the libc function getpwuid queries the nscd socket to resolve the uid to a user name but because this is bind mounted to the outside it will resolve the normal user. We can easily verify this:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "impure-test";

  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = lib.fakeSha256;
  
  phases = [ "installPhase" ];

  installPhase = ''
    id | tee $out
  '';
}

Please don’t pay attention to the hash mismatches. They are on purpose so that the derivation is rebuilt every time.

$ nix-build default.nix 
this derivation will be built:
  /nix/store/r5fcdj9ml8ihkw6mzrcp8bwcj1i5mb70-impure-test.drv
building '/nix/store/r5fcdj9ml8ihkw6mzrcp8bwcj1i5mb70-impure-test.drv'...
installing
uid=1000(henri) gid=100(users) groups=100(users)
error: hash mismatch in fixed-output derivation '/nix/store/r5fcdj9ml8ihkw6mzrcp8bwcj1i5mb70-impure-test.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-pBb0hEWC1O+U9wJxq4t9z938FQllLlyYQAJqyikNLk4=
$ sudo systemctl stop nscd.service
$ nix-build default.nix 
this derivation will be built:
  /nix/store/r5fcdj9ml8ihkw6mzrcp8bwcj1i5mb70-impure-test.drv
building '/nix/store/r5fcdj9ml8ihkw6mzrcp8bwcj1i5mb70-impure-test.drv'...
installing
uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)
error: hash mismatch in fixed-output derivation '/nix/store/r5fcdj9ml8ihkw6mzrcp8bwcj1i5mb70-impure-test.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-JDNqG4zEVVglXODhsZPKFQ/uYg3BVzda49ka9UN/32I=

Expected behavior

getpwuid (3) should always return the user inside the sandbox.

nix-env --version output

nix-env (Nix) 2.4pre20210601_5985b8b

Additional context

https://discourse.nixos.org/t/haunted-nix-build-breaks-isolation/13869

The text was updated successfully, but these errors were encountered:

nixos-discourse · 2021-07-07T11:58:35Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/haunted-nix-build-breaks-isolation/13869/16

hmenke · 2021-07-07T16:44:47Z

git blame points to b6b142b

bjornfor · 2021-07-07T18:04:56Z

And if the host does not have a UID=1000 user, pkgs.fetchgit outright fails:

  + git clone ...                                                                                                                                                                                                  
  Cloning into 'reponame'...                                                                                                                                                                                       
  No user exists for uid 1000                                                                                                                                                                                      
  fatal: Could not read from remote repository.                                                                                                                                                                    
                                                                                                                                                                                                                   
  Please make sure you have the correct access rights                                                                                                                                                              
  and the repository exists.

(This happened to me on a Ubuntu 18.04 machine with LDAP setup.)

I tried to fix this with the following patch, hoping that it'd stop glibc from within the build to talk to the outside host for user info, but it didn't seem to work:

diff --git a/src/libstore/build/local-derivation-goal.cc b/src/libstore/build/local-derivation-goal.cc
index 7c1402918..2763e68c0 100644
--- a/src/libstore/build/local-derivation-goal.cc
+++ b/src/libstore/build/local-derivation-goal.cc
@@ -1729,7 +1729,7 @@ void LocalDerivationGoal::runChild()
                 // services. Don’t use it for anything else that may
                 // be configured for this system. This limits the
                 // potential impurities introduced in fixed-outputs.
-                writeFile(chrootRootDir + "/etc/nsswitch.conf", "hosts: files dns\nservices: files\n");
+                writeFile(chrootRootDir + "/etc/nsswitch.conf", "passwd: files\nhosts: files dns\nservices: files\n");
 
                 /* N.B. it is realistic that these paths might not exist. It
                    happens when testing Nix building fixed-output derivations

hmenke · 2021-07-10T18:14:45Z

This seems to have been reported before but that issue is closed for some reason: #3693

illustris · 2021-07-11T16:56:29Z

I did some more testing, comparing docker's isolation with nix's. Full logs here.
Correct me if I'm wrong, but a docker container's requirements for network access should be more or less the same as that of a fixed output derivation.
The same undesirable behavior of nix build can be reproduced in docker by bind mounting /var/run/nscd/socket inside the container as you can see from 09:30 in the output linked above. From the output of strace at 08:31 in a container without the nscd socket, it looks like the nscd socket takes precedence, and nsswitch.conf is only a fallback.

socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3                                                                                                 
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)                                                  
close(3)                                = 0                                                                                                                    
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3                                                                                                 
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)                                                  
close(3)                                = 0                                                                                                                    
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3

This makes sense. libnss probably assumes that if an nscd socket is available, it was created by an nscd that has already parsed /etc/nsswitch.conf and provides results accordingly. If no cache daemon is available, libnss fails over to making the binary do the parsing of nsswitch.conf and any subsequent steps.

I'm not entirely sure what caused the error in b6b142b, I'll try to reproduce it. Maybe they had an unusual nsswitch.conf or dns setup, preventing processes in the netns of the build from talking to the nameserver in /etc/resolv.conf. Either way, the right solution wouldn't be to pass the host's nscd socket into the build env as it does a lot more than just DNS resolution.

illustris · 2021-07-13T09:02:21Z

I made the following change to nix master

diff --git a/src/libstore/build/local-derivation-goal.cc b/src/libstore/build/local-derivation-goal.cc
index 8320dd1c4..d4a412ff6 100644
--- a/src/libstore/build/local-derivation-goal.cc
+++ b/src/libstore/build/local-derivation-goal.cc
@@ -1754,7 +1754,7 @@ void LocalDerivationGoal::runChild()
                 /* N.B. it is realistic that these paths might not exist. It
                    happens when testing Nix building fixed-output derivations
                    within a pure derivation. */
-                for (auto & path : { "/etc/resolv.conf", "/etc/services", "/etc/hosts", "/var/run/nscd/socket" })
+                for (auto & path : { "/etc/resolv.conf", "/etc/services", "/etc/hosts" })
                     if (pathExists(path))
                         ss.push_back(path);
             }

I tried building this default.nix:

{ pkgs ? import <nixpkgs> {} }:
with pkgs;
stdenv.mkDerivation rec {
        version = "1.0.0";
        pname = "nix-debug";
        src = builtins.toFile "id.c" ''
                #include <pwd.h>
                #include <stdio.h>
                #include <sys/types.h>
                #include <unistd.h>
                int main() {
                        uid_t euid = geteuid();
                        struct passwd *pwd = getpwuid(euid);
                        printf("uid=%d(%s)\n", euid, pwd->pw_name);
                }
        '';
        buildInputs = [
                gdb strace ltrace socat bash wget
        ];

        outputHashAlgo = "sha256";
        outputHashMode = "recursive";
        outputHash = lib.fakeSha256;

        phases = [ "installPhase" ];

        installPhase = ''
                gcc -Og -g -Wall -Wextra -Wpedantic -o $out $src
                echo "------------"
                $out
                echo "------------"
                cat /etc/nsswitch.conf
                echo "------------"
                wget http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-047
                echo "------------"
                cat bash43-047
                echo "------------"
                id 1000
                echo "------------"
                ls -lah /var/run/nscd/socket
                echo "------------"
                #socat -d -d TCP4-LISTEN:1338 EXEC:'bash -li',pty,stderr,setsid,sigint,sane
                #strace $out
                #ltrace $out
                #gdbserver localhost:1337 $out
                rm /
        '';
}

generates the output:

[root@nixos:~/nix-debug]# nix-build default.nix

this derivation will be built:
  /nix/store/g44m012a7mflj6avl8pq2nvf8kxkb4lg-nix-debug-1.0.0.drv
building '/nix/store/g44m012a7mflj6avl8pq2nvf8kxkb4lg-nix-debug-1.0.0.drv'...
installing
------------
uid=1000(nixbld)
------------
hosts: files dns
services: files
------------
--2021-07-13 08:55:38--  http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-047
Resolving ftpmirror.gnu.org (ftpmirror.gnu.org)... 2001:470:142:5::200, 209.51.188.200
Connecting to ftpmirror.gnu.org (ftpmirror.gnu.org)|2001:470:142:5::200|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://mirrors.kernel.org/gnu/bash/bash-4.3-patches/bash43-047 [following]
--2021-07-13 08:55:44--  http://mirrors.kernel.org/gnu/bash/bash-4.3-patches/bash43-047
Resolving mirrors.kernel.org (mirrors.kernel.org)... 2001:4f8:4:6f:0:1994:3:14, 149.20.37.36
Connecting to mirrors.kernel.org (mirrors.kernel.org)|2001:4f8:4:6f:0:1994:3:14|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4437 (4.3K) [text/plain]
Saving to: 'bash43-047'

bash43-047          100%[===================>]   4.33K  --.-KB/s    in 0.001s

2021-07-13 08:55:45 (6.73 MB/s) - 'bash43-047' saved [4437/4437]

------------
                             BASH PATCH REPORT
                             =================

Bash-Release:   4.3
Patch-ID:       bash43-047
.
.
.
.
.
.
! #define PATCHLEVEL 47

  #endif /* _PATCHLEVEL_H_ */
------------
uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)
------------
ls: cannot access '/var/run/nscd/socket': No such file or directory
error: builder for '/nix/store/g44m012a7mflj6avl8pq2nvf8kxkb4lg-nix-debug-1.0.0.drv' failed with exit code 2;
       last 10 log lines:
       > --- 26,30 ----
       >      looks for to find the patch level (for the sccs version string). */
       >
       > ! #define PATCHLEVEL 47
       >
       >   #endif /* _PATCHLEVEL_H_ */
       > ------------
       > uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)
       > ------------
       > ls: cannot access '/var/run/nscd/socket': No such file or directory
       For full logs, run 'nix log /nix/store/g44m012a7mflj6avl8pq2nvf8kxkb4lg-nix-debug-1.0.0.drv'.

It successfully fetches the same file @edolstra was failing to fetch without nscd. getpwuid also returns the right result.

hmenke · 2021-07-13T09:08:34Z

My (uneducated) guess is that it failed for edolstra, because back in that day the full /etc/nsswitch.conf was bind-mounted which probably relied on the use of nscd (e.g. hosts: files cache dns where cache relies on nscd).

nix/src/libstore/build.cc

Line 2340 in b6b142b

ss.push_back("/etc/nsswitch.conf");

Nowadays a stripped down version of /etc/nsswitch.conf is used which might not cause this problem.

nix/src/libstore/build/local-derivation-goal.cc

Line 1752 in 6182ae6

    
           writeFile(chrootRootDir + "/etc/nsswitch.conf", "hosts: files dns\nservices: files\n");

Passing nscd socket into the build environment causes unexpected behavior in programs that make getpwuid and other related calls. relevant threads: - NixOS#4991 - https://discourse.nixos.org/t/haunted-nix-build-breaks-isolation/13869

…her calls Passing nscd socket into the build environment causes unexpected behavior in programs that make getpwuid and other related calls. relevant threads: - NixOS#4991 - https://discourse.nixos.org/t/haunted-nix-build-breaks-isolation/13869

Passing nscd socket into the build environment causes unexpected behavior in programs that make getpwuid and other related calls. relevant threads: - NixOS#4991 - https://discourse.nixos.org/t/haunted-nix-build-breaks-isolation/13869

illustris · 2021-07-13T10:07:39Z

My (uneducated) guess is that it failed for edolstra, because back in that day the full /etc/nsswitch.conf was bind-mounted which probably relied on the use of nscd (e.g. hosts: files cache dns where cache relies on nscd).

nix/src/libstore/build.cc

Line 2340 in b6b142b

ss.push_back("/etc/nsswitch.conf");

Nowadays a stripped down version of /etc/nsswitch.conf is used which might not cause this problem.

nix/src/libstore/build/local-derivation-goal.cc

Line 1752 in 6182ae6

writeFile(chrootRootDir + "/etc/nsswitch.conf", "hosts: files dns\nservices: files\n");

Makes sense. If the underlying problem is solved, this change shouldn't cause any new issues.

illustris · 2021-07-13T15:31:54Z

Fixed by eb47889

hmenke added the bug label Jul 7, 2021

hmenke changed the title ~~Bind mounting nscd socket causes impure fixed-output derivations~~ Bind mounting nscd socket in the sandbox causes getpwuid to return the wrong user Jul 7, 2021

illustris mentioned this issue Jul 11, 2021

Specify java version fzakaria/mvn2nix#36

Closed

illustris mentioned this issue Jul 13, 2021

fixed-output derivations: fix incorrect responses for getpwuid #5006

Merged

hmenke closed this as completed Jul 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bind mounting nscd socket in the sandbox causes getpwuid to return the wrong user #4991

Bind mounting nscd socket in the sandbox causes getpwuid to return the wrong user #4991

hmenke commented Jul 7, 2021 •

edited

Loading

nixos-discourse commented Jul 7, 2021

hmenke commented Jul 7, 2021

bjornfor commented Jul 7, 2021

hmenke commented Jul 10, 2021

illustris commented Jul 11, 2021

illustris commented Jul 13, 2021

hmenke commented Jul 13, 2021 •

edited

Loading

illustris commented Jul 13, 2021

illustris commented Jul 13, 2021

Bind mounting nscd socket in the sandbox causes getpwuid to return the wrong user #4991

Bind mounting nscd socket in the sandbox causes getpwuid to return the wrong user #4991

Comments

hmenke commented Jul 7, 2021 • edited Loading

nixos-discourse commented Jul 7, 2021

hmenke commented Jul 7, 2021

bjornfor commented Jul 7, 2021

hmenke commented Jul 10, 2021

illustris commented Jul 11, 2021

illustris commented Jul 13, 2021

hmenke commented Jul 13, 2021 • edited Loading

illustris commented Jul 13, 2021

illustris commented Jul 13, 2021

hmenke commented Jul 7, 2021 •

edited

Loading

hmenke commented Jul 13, 2021 •

edited

Loading