pip mistakenly reports recursion error even when there is no recursion

[d3dave]

Description

I have three requirements files, as follows:

# requirements.txt
...

# test-requirements.txt
-r requirements.txt
...

# lint-requirements.txt
-r requirements.txt
-r test-requirements.txt
...

When attempting to pip install -r lint-requirements.txt, pip complains:

ERROR: .../requirements.txt recursively references itself in .../test-requirements.txt and again in .../lint-requirements.txt

even though there is no recursive reference here. This used to work fine prior to pip 24.3.

Expected behavior

pip should not report an error and instead should proceed with installation as usual.

pip version

24.3

Python version

3.9

OS

Linux

How to Reproduce

1. Place three example requirements files in a directory as above.
2. Run pip install -r lint-requirements.txt
3. Observe error.

Output

$ python3.9 -m pip install -r lint-requirements.txt -c constraints.txt
Defaulting to user installation because normal site-packages is not writeable
ERROR: <omitted>/requirements.txt recursively references itself in <omitted>/test-requirements.txt and again in <omitted>/lint-requirements.txt

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[xmatthias]

Same here ... breaks all our CI runs - as we do update pip, but don't pin pip in ci so far ...
it's also not limited to python 3.9 - we're seeing this from 3.10-3.12.

[sbidoul]

Likely due to #12877

[notatallshaw]

Hi all, if you are getting this error right now then I reccomend pinning your pip to pip==24.2 until there is a chance to investigate, decide how to handle, and probably release a 24.3.1.

Trying to create an MRE I found something interesting:

a.txt:

requests

And then b.txt:

-r a.txt

And run pip install -r a.txt -r b.txt --dry-run I don't get an error. But if I add:

c.txt:

-r a.txt
-r b.txt

And run pip install -r a.txt -r b.txt -r c.txt --dry-run I do get an error (and c.txt must refer to both a.txt and b.txt)

[notatallshaw]

I suspect the logic in #12877 is wrong, as I don't think it considers you can receive many requirements files and they can reference each other as a DAG.

I think to fix it you need to build a Directed Graph and show there are no cycles, I suspect that's not a minor code change. Perhaps just better to revert?

[d3dave]

I think to fix it you need to build a Directed Graph and show there are no cycles

@notatallshaw Should be sufficient to remove files from the "parsed" list once they are parsed completely. The current implementation keeps the file in the list even once it's not being parsed any more.

[sbidoul]

Tentative fix in #13047. @d3dave @xmatthias it would be great if you could test it.

[sbidoul]

@notatallshaw thanks for the MRE !

[xmatthias]

@sbidoul seems to work in my test (installed via pip install git+https://github.com/sbidoul/pip.git@fix-13046) - so i'd say great job 👍 - ran against the same repository i encountered the initial problem with.

[sbidoul]

Ok, thanks! I'll cut a 24.3.1 tonight.