Resolution conflict due to the PEP 508 requirement with superfluous #egg fragment specified in a package’s metadata

[ajelinski]

Description

A conflict occurs when trying to install packages based on the requirements.txt file which has a git-based package and another package that has the same git-based package as a dependency. It happens even if there are no git revisions specified.

What's more, it means that if a git-based package with a git-based dependency is installed, then the pip freeze output is unusable because of this conflict. The example of such a package is https://github.com/QuickLogic-Corp/quicklogic-fasm with its install_requires:

    install_requires = [
        'fasm @ git+https://github.com/symbiflow/fasm#egg=fasm',
        'fasm-utils @ git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils#egg=fasm-utils'
    ],

With such a package installed, the output of pip freeze can't be used successfully in pip install -r.

Expected behavior

Packages should be successfully installed.

The conflict is only reasonable if git revisions are specified for the same git-based package and are different. For example, if after resolving dependencies it turns out such packages are going to be installed:

fasm-utils @ git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils@REVISION
fasm-utils @ git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils@ANOTHER_REVISION

but if only one specifies a revision it should be compatible with a package without a revision specified.

pip version

21.1.1

Python version

3.7 / 3.9

OS

Linux

How to Reproduce

python3 -m pip install git+https://github.com/QuickLogic-Corp/quicklogic-fasm
python3 -m pip freeze >requirements.txt
python3 -m pip uninstall quicklogic-fasm fasm fasm-utils
python3 -m pip install -r requirements.txt

but also without git revisions it doesn't work:

python3 -m pip install git+https://github.com/QuickLogic-Corp/quicklogic-fasm
python3 -m pip freeze | sed 's#@[^ ]\+##g' >requirements-without-revisions.txt
python3 -m pip uninstall quicklogic-fasm fasm fasm-utils
pip install -r requirements-without-revisions.txt

Output

root@5a910819b958:/# pip install -r requirements.txt 
Collecting fasm@ git+https://github.com/symbiflow/fasm@99f199f9e32fd30c8adffcc73c13caf95a951c35
  Cloning https://github.com/symbiflow/fasm (to revision 99f199f9e32fd30c8adffcc73c13caf95a951c35) to /tmp/pip-install-31i9zc00/fasm_389620f524324d759e68423b3f833b59
  Running command git clone -q https://github.com/symbiflow/fasm /tmp/pip-install-31i9zc00/fasm_389620f524324d759e68423b3f833b59
  Running command git rev-parse -q --verify 'sha^99f199f9e32fd30c8adffcc73c13caf95a951c35'
  Running command git fetch -q https://github.com/symbiflow/fasm 99f199f9e32fd30c8adffcc73c13caf95a951c35
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting fasm-utils@ git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils@3d6a375ddb6b55aaa5a59d99e44a207d4c18709f
  Cloning https://github.com/QuickLogic-Corp/quicklogic-fasm-utils (to revision 3d6a375ddb6b55aaa5a59d99e44a207d4c18709f) to /tmp/pip-install-31i9zc00/fasm-utils_77d8f9716d15454fbb4dec7204d8caaa
  Running command git clone -q https://github.com/QuickLogic-Corp/quicklogic-fasm-utils /tmp/pip-install-31i9zc00/fasm-utils_77d8f9716d15454fbb4dec7204d8caaa
  Running command git rev-parse -q --verify 'sha^3d6a375ddb6b55aaa5a59d99e44a207d4c18709f'
  Running command git fetch -q https://github.com/QuickLogic-Corp/quicklogic-fasm-utils 3d6a375ddb6b55aaa5a59d99e44a207d4c18709f
Collecting quicklogic-fasm@ git+https://github.com/QuickLogic-Corp/quicklogic-fasm@b86a3f926c84f96e32e6dc51e0679a42d0f98063
  Cloning https://github.com/QuickLogic-Corp/quicklogic-fasm (to revision b86a3f926c84f96e32e6dc51e0679a42d0f98063) to /tmp/pip-install-31i9zc00/quicklogic-fasm_907a7665943043c0909a3200aee79f10
  Running command git clone -q https://github.com/QuickLogic-Corp/quicklogic-fasm /tmp/pip-install-31i9zc00/quicklogic-fasm_907a7665943043c0909a3200aee79f10
  Running command git rev-parse -q --verify 'sha^b86a3f926c84f96e32e6dc51e0679a42d0f98063'
  Running command git fetch -q https://github.com/QuickLogic-Corp/quicklogic-fasm b86a3f926c84f96e32e6dc51e0679a42d0f98063
  Running command git submodule update --init --recursive -q
Requirement already satisfied: Arpeggio==1.10.2 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 1)) (1.10.2)
Requirement already satisfied: textX==2.3.0 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 5)) (2.3.0)
Collecting fasm-utils@ git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils#egg=fasm-utils
  Cloning https://github.com/QuickLogic-Corp/quicklogic-fasm-utils to /tmp/pip-install-31i9zc00/fasm-utils_ad67eecd460a46f2878c8cb548170041
  Running command git clone -q https://github.com/QuickLogic-Corp/quicklogic-fasm-utils /tmp/pip-install-31i9zc00/fasm-utils_ad67eecd460a46f2878c8cb548170041
Collecting fasm@ git+https://github.com/symbiflow/fasm#egg=fasm
  Cloning https://github.com/symbiflow/fasm to /tmp/pip-install-31i9zc00/fasm_284e261cc2924e599dcf7ec899bef64e
  Running command git clone -q https://github.com/symbiflow/fasm /tmp/pip-install-31i9zc00/fasm_284e261cc2924e599dcf7ec899bef64e
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: pip is looking at multiple versions of fasm-utils to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of fasm to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements.txt (line 4) and fasm-utils 0.0.1 (from git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils@3d6a375ddb6b55aaa5a59d99e44a207d4c18709f) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested fasm-utils 0.0.1 (from git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils@3d6a375ddb6b55aaa5a59d99e44a207d4c18709f)
    quicklogic-fasm 0.0.1 depends on fasm-utils 0.0.1 (from git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils#egg=fasm-utils)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies



Without git revisions in `requirements.txt` the conflict information makes even less sense:

The conflict is caused by:
    The user requested fasm-utils 0.0.1 (from git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils)
    quicklogic-fasm 0.0.1 depends on fasm-utils 0.0.1 (from git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils#egg=fasm-utils)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[PaulBenn-UnlikelyAI]

for what it's worth, this also happens with file:// URLs (not just git-based).

If project A depends on B and C, which both have file D as a dependency, pip will state that D conflicts with itself.

[uranusjr]

pip does not try to resolve Git commits for you. A URL is a URL, if the URL is different, they conflict. If you don’t actually want to use that exact URL, you should use version ranges in the dependency instead. IMO this is working as designed and there’s nothing to fix.

[PaulBenn-UnlikelyAI]

Hi @uranusjr , indeed, if the URLs were different I wouldn't be reporting this. However, this happens even when the URLs are exactly identical, which seems like it may need more attention. Could you please re-open this?

[uranusjr]

The URLs are different in the original report, so I’m responding based on what I have. If your case is different, please provide it.

[ajelinski]

Hi @uranusjr, perhaps I gave too many versions of the problem but, to sum it up, I don't think that the additional #egg=fasm-utils part should create a conflict:

The conflict is caused by:
    The user requested fasm-utils 0.0.1 (from git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils)
    quicklogic-fasm 0.0.1 depends on fasm-utils 0.0.1 (from git+https://github.com/QuickLogic-Corp/quicklogic-fasm-utils#egg=fasm-utils)

Do you really think it's a correct behaviour?

There's also an additional problem that the requirements.txt file causing a conflict during installation was created by issuing pip freeze in a working and stable environment. Shouldn't these always work in pip install -r without a conflict?

[uranusjr]

Thanks for the clarification. To be honest, yes, the conflict from different egg= values is condiered correct, since it is the standard compliant implementation in pip maintianers’ views. But we also understand it is counter-intuitive and have been discussing the possibility of loosening the check (practicality beats purity and things like that). I have a work-in-progress locally that implements a loosened check, and will try to push it this weekend. It is still far from ready, but I think it may be possible to get it ready for 21.2 (due next month) if I put enough time on it.

[ajelinski]

@uranusjr Thanks.That's great it'll be changed. It'd be great if you could inform here when the change gets merged.

[uranusjr]

I’ve posted my local branch in the draft PR linked above. Feel free to give it a go and see if it works for you. (You can install from a Git branch with the @branch syntax.)

[piotr-dobrogost]

@uranusjr

But we also understand it is counter-intuitive and have been discussing the possibility of loosening the check (practicality beats purity and things like that).

Could you please point us to this discussion? Funny enough I vaguely recall there was such a discussion and I think I even proposed there to not treat different egg fragments as an error and instead treat it as a warning only. Strangely enough I was trying to find it but with no success.

[uranusjr]

I vaguely recall there was such a discussion and I think I even proposed there to not treat different egg fragments as an error and instead treat it as a warning only

I think we are thinking about the same discussion, but I couldn’t find it either. I do intend to dig it out (along with other related threads) as a part of #10079.

[piotr-dobrogost]

I think we are thinking about the same discussion

This is probably #1289.
Btw, could we change title of this issue to reflect the real problem being raised?
Also, when using name @ url syntax adding #egg= fragment not only is not needed but it can cause problems as raised in this and other issues (#9429). That's why I would encourage OP to raise issue against https://github.com/QuickLogic-Corp/quicklogic-fasm to remove #egg= fragments from its install_requires.

EDIT
https://github.com/QuickLogic-Corp/quicklogic-fasm is forked from https://github.com/antmicro/quicklogic-fasm so maybe the issue should be raised against the latter not the former?

[uranusjr]

Yes, that’s the one! I can do the modification, but am unsure what title is best.

[uranusjr]

I thought a bit about a warning and now think pip should not display one for now. There are still legistimate reasons for user to use that in user inputs; we should eventually refactor those out as per #1289, but we shouldn’t show the warning until we implement valid alternatives. The #egg usages we want to warn about are in distribution metadata (where PEP 508 is required and that value is ignored in all cases), but those are generally outside the user’s control, so the warning would be useless. Instead, we should lobby for build tools (e.g. setuptools, flit, poetry) to warn package maintainers about this when they build sdists and wheels.

[piotr-dobrogost]

Yes, that’s the one! I can do the modification, but am unsure what title is best.

Resolution conflict due to the PEP 508 requirement with superfluous #egg fragment specified in install_requires?

[uranusjr]

It covers more than that though. The conflict can be resolved by ignoring egg= by comparison, but the issue is also tracking PEP 508 conversion for editables. We need to break them out. (Considering the unneeded context in #1289, maybe we should instead create new issues for each thing we want to do and close that old issue.)

[piotr-dobrogost]

It seems I was unclear. When I wrote

Btw, could we change title of this issue to reflect the real problem being raised?

I meant this issue (#10002) not that other, old issue (#1289) :)

[uranusjr]

Oh, I see, sorry for the consusion. Will do.

[uranusjr]

I also changed the labels. Further discussions should go to either the PR #10079 (which ignores #egg unconditionally so resolves the symptom) or #1289 (which is broader in scope and deals with the root cause).