Local installs now require write permissions

[rpkilby]

Environment

• pip version: 20.1
• Python version: 3.6.8
• OS: CentOS 8

Description
The new build-in-place behavior (ref #7882) for local project installs requires that a user have write permissions in the project directory. This normally makes sense, since a user is typically installing their own local work, but this breaks when users are installing a project owned by another user.

Without write permissions, other users will get a permissions error. However, giving them write permissions isn't ideal, and the build artifacts left behind are owned by them instead of the author.

Related to #8168

Expected behavior
Users should not require write permissions to install a local project. Possibly, pip tests for write access, and falls back to building in a temp dir? Possibly, pip could have an option for not building in place, or for providing a directory for where the build should occur (note that the --build option does not fix this).

How to Reproduce

At a high level:

1. Create a local project
2. Ensure other users don't have write permissions chmod 755 -R <project>
3. Switch to another user
4. pip install -e <project> (note that non-editable installs also fail similarly)

Example error:

$ pip install -e /work/test/test/
Obtaining file:///work/test/test
Could not build wheels for test, since package 'wheel' is not installed.
Installing collected packages: test
  Running setup.py develop for test
    ERROR: Command errored out with exit status 1:
     command: /home/secondary/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/work/test/test/setup.py'"'"'; __file__='"'"'/work/test/test/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /work/test/test/
    Complete output (4 lines):
    running develop
    running egg_info
    creating test.egg-info
    error: could not create 'test.egg-info': Permission denied
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/secondary/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/work/test/test/setup.py'"'"'; __file__='"'"'/work/test/test/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

Detailed steps:

Vagrantfile contents:

Vagrant.configure("2") do |config|
    config.vm.box = "bento/centos-8.0"
    config.vm.provision "shell", inline: "yum install -y python36"
end

Create primary/secondary users and test project owned by primary user:

[vagrant@localhost ~]$ sudo su
[root@localhost vagrant]# useradd primary
[root@localhost vagrant]# useradd secondary
[root@localhost vagrant]# mkdir -p -m 755 /work/test
[root@localhost vagrant]# chown primary /work/test/
[root@localhost vagrant]# exit
exit
[vagrant@localhost ~]$ sudo -iu primary
[primary@localhost ~]$ mkdir /work/test/test
[primary@localhost ~]$ echo "from setuptools import setup; setup(name='test', version='1', py_modules=['foo'])" > /work/test/test/setup.py
[primary@localhost ~]$ touch /work/test/test/foo.py
[primary@localhost ~]$ exit
logout

Verify project is reachable and attempt to install as secondary user:

[vagrant@localhost ~]$ sudo -iu secondary
[secondary@localhost ~]$ ls /work/test/test/
foo.py  setup.py
[secondary@localhost ~]$ python3 -m venv venv
[secondary@localhost ~]$ source venv/bin/activate
(venv) [secondary@localhost ~]$ pip install -U pip setuptools
Collecting pip
  Downloading https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 1.1MB/s 
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl (582kB)
    100% |████████████████████████████████| 583kB 1.6MB/s 
Installing collected packages: pip, setuptools
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
  Found existing installation: setuptools 39.2.0
    Uninstalling setuptools-39.2.0:
      Successfully uninstalled setuptools-39.2.0
Successfully installed pip-20.1 setuptools-46.1.3
(venv) [secondary@localhost ~]$ pip install -e /work/test/test/
Obtaining file:///work/test/test
Could not build wheels for test, since package 'wheel' is not installed.
Installing collected packages: test
  Running setup.py develop for test
    ERROR: Command errored out with exit status 1:
     command: /home/secondary/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/work/test/test/setup.py'"'"'; __file__='"'"'/work/test/test/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /work/test/test/
    Complete output (4 lines):
    running develop
    running egg_info
    creating test.egg-info
    error: could not create 'test.egg-info': Permission denied
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/secondary/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/work/test/test/setup.py'"'"'; __file__='"'"'/work/test/test/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
(venv) [secondary@localhost ~]$ exit
logout

Make project writable and reattempt install as secondary user:

[vagrant@localhost ~]$ sudo su
[root@localhost vagrant]# chmod -R 777 /work/test/
[root@localhost vagrant]# exit
exit
[vagrant@localhost ~]$ sudo -iu secondary
[secondary@localhost ~]$ source venv/bin/activate
(venv) [secondary@localhost ~]$ pip install -e /work/test/test/
Obtaining file:///work/test/test
Could not build wheels for test, since package 'wheel' is not installed.
Installing collected packages: test
  Running setup.py develop for test
Successfully installed test
(venv) [secondary@localhost ~]$ pip list
Package    Version Location
---------- ------- ---------------
pip        20.1
setuptools 46.1.3
test       1       /work/test/test

Build artifacts are owned by secondary user:

(venv) [secondary@localhost ~]$ ls -la /work/test/test/
total 4
drwxrwxrwx. 3 primary   primary   57 May  5 20:03 .
drwxrwxrwx. 3 primary   root      18 May  5 20:01 ..
-rwxrwxrwx. 1 primary   primary    0 May  5 20:01 foo.py
-rwxrwxrwx. 1 primary   primary   82 May  5 20:01 setup.py
drwxrwxr-x. 2 secondary secondary 90 May  5 20:03 test.egg-info

[uranusjr]

I’m starting to think maybe we should revert this change. PEP 517 supporting out-of-tree builds is the only way this can be improved, but that will take a significant amount of time to roll out. In the meantime, the previous method is slow, but at least works for everyone.

[sbidoul]

at least works for everyone

Not everyone. There were also correctness issues. Let's discuss in #7555.

[n8henrie]

Hope this doesn't excessively clutter this issue, but I've been struggling for a day or so to figure out why I can't $ $HOME/.venv/bin/python -m pip install /app in a docker container, where /app is a pyproject.toml-based project that installs fine on my host machine -- I keep getting permissions errors. This issue explains the problem, but it was someone difficult to find.

It looks like the problem is that I am running as an unprivileged user, but the /app directory with my source code is root-owned. I thought it would be fine because I chown -R $HOME, making .venv and all subdirectories owned and writable, and couldn't imagine why I wouldn't be able to install from a read-only directory (especially since I can pip install git+https://github.com/foo/bar which obviously doesn't require write permissions).

Looks like the inability to create the appname.egg-info (src/appname.egg-info in my case) and build directories in /app are the problem; if I manually create these directories with user permissions everything else works as is. I had assumed (for no specific reason) these would get created somewhere in /tmp or in the virtualenv and am surprised to see them needing to be created in the read-only directory containing the source code (kind of feels like the issues people run into trying to /usr/bin/python -m pip install stuff without --user or venv -- except I'm using a venv).

A few breadcrumbs from my search history that my help other users find this issue; the fact that the error output lists egg_info and build as issues without the parent directories made this harder to figure out, as I assumed these would be created somewhere in the venv.

• Getting requirements to build wheel exited with 1
• running egg_info

 × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]
      running egg_info
      error: [Errno 13] Permission denied
      [end of output]

  creating build
  error: could not create 'build': Permission denied
  error: subprocess-exited-with-error

  × Building wheel for fauxmo (pyproject.toml) did not run successfully.

[sbidoul]

@n8henrie that is a known issue with setuptools since pip switched to building in place.
This change in pip was necessary for various reasons and the tradeoffs have been weighted carefully.

So there are several options

• adjust the permissions of your /app directory as you noticed already
• address the question of building from a read-only directory to the setuptools project (be sure to search their tracker as this issue has probably been raised already)
• use another build backend that supports that

I am going to close this issue as there is really nothing we can do in pip for this, it must be addressed by the build backends.

[n8henrie]

In this context adjusting permissions works fine. In a different context, it seems like copying to /tmp also works fine (since one must have read permissions obviously):

RUN python3 -m venv --system-site-packages "${VENV}" \
    && "${PYTHON}" -m pip install --upgrade pip \
    && cp -a /app /tmp/app \
    && "${PYTHON}" -m pip install /tmp/app

address ... to the setuptools project

pypa/setuptools#3237

I am going to close this issue

No worries, thanks for your time.