Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Surprising behaviour of relative directory references in requirement files #8765

Open
sbidoul opened this issue Aug 15, 2020 · 4 comments
Open
Labels
C: requirement file Using `requirements.txt` S: needs triage Issues/PRs that need to be triaged state: needs discussion This needs some more discussion type: enhancement Improvements to functionality

Comments

@sbidoul
Copy link
Member

sbidoul commented Aug 15, 2020

Environment

  • pip version: 20.2
  • Python version: 3.8 (most probably irrelevant)
  • OS: linux (most probably irrelevant)

Description

In requirement files, relative references to other requirement files (via -r or -c) are interpreted to be relative to the base url of the including file. However requirements expressed as relative directories (with our without -e) are interpreted to be relative to the current working directory.

Expected behavior

Relative directory requirements should be interpreted as relative to the base url of the requirement file.

How to Reproduce

The following bash scripts fails installing ./pkga, where I'd expect it to succeed.

#!/bin/bash
set -ex
cd $(mktemp -d)
mkdir -p testdir/subdir/pkga
echo "from setuptools import setup; setup(name='pkga')" > testdir/subdir/pkga/setup.py
echo "-r subdir/subreqs.txt" > testdir/reqs.txt
echo "./pkga" > testdir/subdir/subreqs.txt
pip install -r testdir/reqs.txt

output:

ERROR: Invalid requirement: './pkga' (from line 1 of testdir/subdir/subreqs.txt)
Hint: It looks like a path. File './pkga' does not exist.

Compatibility considerations

Modifying this behaviour would be a breaking change.
Assuming there is agreement this needs changing, could this be introduced as part of the 2020-resolver feature flag?

@uranusjr
Copy link
Member

The requirement objects are constructed before the resolver is reached, and both implementations get the same things. It is possible to only fix in the new resolver, but that feels hacky to me. Maybe it’d be better to deprecate relative paths entirely, and introduce a couple of custom environment variables to use (e.g. PWD and FOLDER) in each situation?

@pradyunsg pradyunsg added the S: needs triage Issues/PRs that need to be triaged label Aug 30, 2020
@pradyunsg
Copy link
Member

pradyunsg commented Aug 30, 2020

Or... we could maintain status quo where (almost) no one is complaining, and the explanation of how the feature works is straightforward enough. :)

@pradyunsg pradyunsg added C: requirement file Using `requirements.txt` state: needs discussion This needs some more discussion type: enhancement Improvements to functionality labels Aug 30, 2020
@sbidoul
Copy link
Member Author

sbidoul commented Aug 31, 2020

Well, that might be straightforward to explain but quirky nevertheless.

Especially if you consider requirement files located at a remote URL.

requirements.txt:

-r ./subreq.txt

subreq.txt:

./pkga-1.0.0.tar.gz
./pkgb-1.0.0.tar.gz

pip install -r https://somewhere.org/requirements.txt is going to lookup a remote subreq.txt but local files for pkg{a,b}-1.0.0.tar.gz.

@pradyunsg
Copy link
Member

pradyunsg commented Aug 31, 2020

Especially if you consider requirement files located at a remote URL.

That feels like a case special enough that I'm personally not interested in solving this until someone comes in to say "it's important for my workflow to improve behaviour in this situation" and even then, only if they ask nicely. :P

ttung pushed a commit to chanzuckerberg/czgenepi that referenced this issue Feb 3, 2021
### Description
Standard requirements.txt files have a few issues with a third party library installed from source:
1. pip does not handle [relative paths](pypa/pip#8765) in a predictable way.  It is relative to current working directory instead of relative to the path of the requirements.txt.  As we cannot constrain that, it leads to unpredictable behavior.
2. `pip freeze` does not generate the correct output when there is a local library.

### Test plan
1. tried to deploying to elastic beanstalk, and it seems to work.
2. ran `make update-deps`
3. ran `make update-deps` with a local library and `Pipfile.lock` references that instead of just a version number.
ttung pushed a commit to chanzuckerberg/czgenepi that referenced this issue Feb 3, 2021
### Description
Standard requirements.txt files have a few issues with a third party library installed from source:
1. pip does not handle [relative paths](pypa/pip#8765) in a predictable way.  It is relative to current working directory instead of relative to the path of the requirements.txt.  As we cannot constrain that, it leads to unpredictable behavior.
2. `pip freeze` does not generate the correct output when there is a local library.

### Test plan
1. tried to deploying to elastic beanstalk, and it seems to work.
2. ran `make update-deps`
3. ran `make update-deps` with a local library and `Pipfile.lock` references that instead of just a version number.
ttung pushed a commit to chanzuckerberg/czgenepi that referenced this issue Feb 3, 2021
### Description
Standard requirements.txt files have a few issues with a third party library installed from source:
1. pip does not handle [relative paths](pypa/pip#8765) in a predictable way.  It is relative to current working directory instead of relative to the path of the requirements.txt.  As we cannot constrain that, it leads to unpredictable behavior.
2. `pip freeze` does not generate the correct output when there is a local library.

### Test plan
1. tried to deploying to elastic beanstalk, and it seems to work.
2. ran `make update-deps`
3. ran `make update-deps` with a local library and `Pipfile.lock` references that instead of just a version number.
ttung pushed a commit to chanzuckerberg/czgenepi that referenced this issue Feb 3, 2021
### Description
Standard requirements.txt files have a few issues with a third party library installed from source:
1. pip does not handle [relative paths](pypa/pip#8765) in a predictable way.  It is relative to current working directory instead of relative to the path of the requirements.txt.  As we cannot constrain that, it leads to unpredictable behavior.
2. `pip freeze` does not generate the correct output when there is a local library.

### Test plan
1. tried to deploying to elastic beanstalk, and it seems to work.
2. ran `make update-deps`
3. ran `make update-deps` with a local library and `Pipfile.lock` references that instead of just a version number.
ttung pushed a commit to chanzuckerberg/czgenepi that referenced this issue Feb 4, 2021
### Description
Standard requirements.txt files have a few issues with a third party library installed from source:
1. pip does not handle [relative paths](pypa/pip#8765) in a predictable way.  It is relative to current working directory instead of relative to the path of the requirements.txt.  As we cannot constrain that, it leads to unpredictable behavior.
2. `pip freeze` does not generate the correct output when there is a local library.

### Test plan
1. tried to deploying to elastic beanstalk, and it seems to work.
2. ran `make update-deps`
3. ran `make update-deps` with a local library and `Pipfile.lock` references that instead of just a version number.
michalsieron added a commit to michalsieron/mamba that referenced this issue Jan 21, 2023
Suppose we have the following dependencies in the environment.yml:

```
dependencies:
  - pip
  - pip:
    - -r requirements.txt
    - -e deps/my_local_package
```

Without this commit, it is impossible to install such dependencies with
micromamba. That is because micromamba was creating its own temporary
requirements.txt file in the /tmp directory and called pip to install them.

For some reason pip resolves `-r` paths as relative to the file that
contains them, but `-e` paths as relative to the current working directory.
It is a known issue in pip: pypa/pip#8765.

So in the case of mamba, pip looks for `/tmp/requirements.txt` and
`directory/of/env/yml/deps/my_local_package` and unless whole project
is thrown right into `/tmp` this installation will fail.

This commit makes it so installing pip deps matches the behavior of conda.
michalsieron added a commit to michalsieron/mamba that referenced this issue Jan 21, 2023
Suppose we have the following dependencies in the environment.yml:

```
dependencies:
  - pip
  - pip:
    - -r requirements.txt
    - -e deps/my_local_package
```

Without this commit, it is impossible to install such dependencies with
micromamba. That is because micromamba was creating its own temporary
requirements.txt file in the /tmp directory and called pip to install them.

For some reason pip resolves `-r` paths as relative to the file that
contains them, but `-e` paths as relative to the current working directory.
It is a known issue in pip: pypa/pip#8765.

So in the case of mamba, pip looks for `/tmp/requirements.txt` and
`directory/of/env/yml/deps/my_local_package` and unless whole project
is thrown right into `/tmp` this installation will fail.

This commit makes it so installing pip deps matches the behavior of conda.
wolfv added a commit to mamba-org/mamba that referenced this issue Feb 2, 2023
Suppose we have the following dependencies in the environment.yml:

```
dependencies:
  - pip
  - pip:
    - -r requirements.txt
    - -e deps/my_local_package
```

Without this commit, it is impossible to install such dependencies with
micromamba. That is because micromamba was creating its own temporary
requirements.txt file in the /tmp directory and called pip to install them.

For some reason pip resolves `-r` paths as relative to the file that
contains them, but `-e` paths as relative to the current working directory.
It is a known issue in pip: pypa/pip#8765.

So in the case of mamba, pip looks for `/tmp/requirements.txt` and
`directory/of/env/yml/deps/my_local_package` and unless whole project
is thrown right into `/tmp` this installation will fail.

This commit makes it so installing pip deps matches the behavior of conda.

---------------------

Co-authored-by: Wolf <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: requirement file Using `requirements.txt` S: needs triage Issues/PRs that need to be triaged state: needs discussion This needs some more discussion type: enhancement Improvements to functionality
Projects
None yet
Development

No branches or pull requests

3 participants