New Resolver unable to handle multiple versions of git dependencies

[Tbilgere-td]

What did you want to do?
New Resolver is unable to handle cases where private Git Dependencies are shared but do not match "version" (i.e. Branch, Tag, Commit). This not only makes repo management much more tedious but also makes feature testing in repos much harder or impossible.

For example if we have:

• a repo with common shared code called common
• a repo with logic to operate a particular service called service
• a repo for a app called app

Requirements.txt in service calls:

commoncode @ git+git://github.com/repoOrg/[email protected]

While requirements.txt in app calls:

commoncode @ git+git://github.com/repoOrg/[email protected]
service @ git+git://github.com/repoOrg/[email protected]

Output

ERROR: Cannot install -r requirements.txt (line 70), -r requirements.txt (line 71) and common 1.0.1 (from git+git://github.com/repoOrg/[email protected]) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested common 1.0.1 (from git+git://github.com/repoOrg/[email protected])
    service 2.1.0 depends on common 1.0.0 (from git+git://github.com/repoOrg/[email protected])

Additional information
I have spent a considerable amount of time reading though available documentation and have not been able to identify a solution for this use case (such as version ranging). Am I missing something? Ideally there would be some way to specify compatible range in service, with concrete versions specified in app.

[pradyunsg]

Thanks for filing an issue!

I'm unsure what you're asking for tho. pip is correctly rejecting to install 1.0.0 and 1.0.1, which (from pip's resolvers' perspectives) are two different URLs that don't hold the same artifact. This is exactly what the error message says: that there's a conflict.

Could you elaborate on what you are trying to do, and what kind of solution you are looking for?

[Tbilgere-td]

What you are trying to do

Trying to maintain our current (seemingly common) workflow with the new resolver.

What kind of solution you are looking for?

Ideally one that doesn't disrupt our ability to share common code across multiple repos. This doesn't seem like it should not be a covered use case by pip. If that requires specifying compatibility ranges as I stated before that would be fine but currently there appears to be no way to do so.

[andylamp]

Hello all,

Actually @pradyunsg, it seems I've stumbled across the same issue albeit this happens with the same artifacts.

The conflict is caused by:
    The user requested myorg-logger 0.0.7 (from git+https://****@github.com/myorg/myorg-logger.git#egg=myorg-logger)
    myorg-utilities 0.0.6 depends on myorg-logger 0.0.7 (from git+https://****@github.com/myorg/myorg-logger.git@main)

The workflow is the same as described above, however all of the packages try to pull the same common package myorg-logger.git@main and still results in the same issue; there is no issue with the dependencies as it pulls the same versions for both instances.

This, obviously, started happening after the upgrade to 20.3.3 😄 -- any clues on how to fix this?

[sbidoul]

@andylamp in your case it's probably because one URL explicitly mentions the @main branch and the other not.

[andylamp]

@sbidoul it happens regardless if I have the main branch there or not, unfortunately 😢

edit, @sbidoul here is a log that has the branch removed:

Collecting myorg-logger@ git+https://[email protected]/myorg/myorg-logger.git
  Cloning https://****@github.com/myorg/myorg-logger.git to /tmp/pip-install-sc3bdyo9/myorg-logger_c5ccc5f317fa4e5ab51b4bb315bdd340
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done

The conflict is caused by:
    The user requested myorg-logger 0.0.7 (from git+https://****@github.com/myorg/myorg-logger.git#egg=myorg-logger)
    myorg-utilities 0.0.7 depends on myorg-logger 0.0.7 (from git+https://****@github.com/myorg/myorg-logger.git)

The issue is still there, as I said...

[aaraney]

I am also facing a related problem. I think that @sbidoul is potentially hinting at it. In my case, I have two packages A and B, B depends on A. Both of the packages are strictly on GitHub and not published on pypi. But both have setup.py etc. so they are installable via GitHub using https, git, or ssh protocols.

Since B depends on A, in B's setup.py, A is listed in the install_requires list using the following (notice protocol used https):

[
"A@ git+https://github.com/someuser/A.git",
]

The B package is installable like so, pip install git+https://github.com/someuser/B. Simple enough.

However, if you place both of the packages in a requirements.txt file:

git+git://github.com/someuser/A
git+git://github.com/someuser/B

and change the protocol that is used to install (git or ssh), a conflict is raised.

Naively, it seems that pip is comparing the string literal to determine conflicts. I am probably wrong about this, but if this is the case, functionally equivalent dependencies that use different protocols will have a conflict. This is also inclusive of if as @andylamp pointed out, @main is included. From my working hypothesis, the string literal comparison would fail because, you know the @main part.

[andylamp]

@aaraney in my case all packages strictly use git+https to fetch them in all instances, just to provide some clarity on my end 😄

[Tbilgere-td]

From my working hypothesis, the string literal comparison would fail because, you know the @main part.

or in @andylamp 's case I think the issue is the #egg=myorg-logger and/or @main

[goodspark]

I have a similar problem. pip 20.3.0 and 20.3.3

Obfuscated the names, but hopefully it's clear enough:

The conflict is caused by:
    The user requested thing 1.2.3 (from git+https://github.com/fake/[email protected]#egg=thing)
    blah 4.5.6 depends on thing 1.2.3 (from git+https://github.com/fake/[email protected])

It looks like the egg is being ignored from blah. I wonder if that's making pip fail? In the requirements for blah, it's definitely there though. Again, obfuscated:

pytz
thing @ git+https://github.com/fake/[email protected]#egg=thing

Does not happen on pip 20.2.4.

[andylamp]

@Tbilgere-td just to add to the discussion, reverting back to pip 20.2.4 everything works as expected.

[aaraney]

@goodspark

From my working hypothesis, the string literal comparison would fail because, you know the @main part.

It may be that pip is doing an atomic comparison between the deps, so if #egg=thing is included, then the comparison is false.

@andylamp having the same behavior here on < 20.3.*

[Tbilgere-td]

@andylamp right as described in documentation here 20.3 moves to the "new resolver" by default you can also add the --use-deprecated=legacy-resolver flag to <20.3.0 but that goes away when 21 drops...

[hellkite500]

@aaraney @andylamp @goodspark it isn't just the protocol or differences in the repo branch, it looks like ANY difference in the URL will trigger this. Confirmed by trying each of the following variants, simplified to aaraney's example:
B's setup.py

install_requires=[
"A@ git+https://github.com/someuser/A.git",
]

Then to install both from requirements.txt (or in any other way where dependencies are resolved, I would imagine)
You would have to write the requirement verbatim as defined in the install_requires

git+https://github.com/someuser/A.git
git+git://github.com/someuser/B

Even a simple change in the url causes an unresolvable dependency, i.e ommit the .git or just change case of part of the url

git+https://github.com/someuser/A
git+git://github.com/someuser/B

git+https://github.com/SOMEUSER/A.git
git+git://github.com/someuser/B

git+https://github.com/SOMEUSER/A.git#egg=thing
git+git://github.com/someuser/B

You can install A using any of the above variants by itself, but not in conjunction with B which defines install_requires

[Tbilgere-td]

Can confirm @hellkite500 's findings we have consistency throughout our git installs and do not hit this case either in any of our repos.

[andylamp]

@hellkite500 I think that's the problem I am hitting - I am using two packages A and B that depend on the same package C; C resides both in the requirements.txt of A and B as well as in their install_requires.

[hellkite500]

@hellkite500 I think that's the problem I am hitting - I am using two packages A and B that depend on the same package C; C resides both in the requirements.txt of A and B as well as in their install_requires.

If the URL strings aren't exact in all places then most likely. The quick workaround is making then match I think. Not a good sustainable solution though IMHO.

[andylamp]

For now, I think reverting to an older version of pip is the way to go; I think a proper fix is warranted for this however...

[hellkite500]

I stumbled across this issue when my CI workflows for testing packages started acting up. Many of those workflows default update pip to the latest version before setting up the testing env, so downgrading isn't something I want to try on each of those.

[andylamp]

ha! this is how I came across that, basically the badge of shame turned red instead of green for initially unknown reasons 😄

[uranusjr]

I described in #9229 how the URLs from dependencies must match exactly to resolve correctly, and why this is an important restriction.

My suggestion is to to use PEP 508 instead, which is functionally the same as the #egg= fragment, but better in every other ways.

[andylamp]

@uranusjr I see that there has already been significant discussion on the matter over another issue, however I still think this poses a significant problem going forward - especially with the deprecation of the old resolver.

[uranusjr]

If you ask me, URL specifications are the significant problem going forward, so there’s no surprise here 🙂

There’s no way to know whether different URLs point to the logical same thing without actually downloading and potentially building the content, which is very problematic in more than one way. So systems using URLs for dependency specification generally impose restrictions over its usage. Go (for which URLs are one of the two primary ways to specify dependencies), for example, behave the same as pip here.

The legacy resolver was way too loose, relying on the user doing the right thing (only the user knows foo @ FOO and foo @ foo point to the same thing), but then things blow up too badly when the user inevitably does something wrong. The new resolver tries to bring things to where they should, at the expense of packages not “playing nice” with the newly enforced (but IMO reasonable) rules. Eventually we do want to offer a way for users to get around the issue and have a way to still use those problematic packages (#8076), but first we need to actually put up the rules to get an idea what we need to allow users to do.

[Tbilgere-td]

@uranusjr I'm not clear if your comments are directed solely towards the issue of resolving the same dependencies with different URL formatting or my issue of being unable to update common library repos without breaking EVERY subsequent repo can you clarify for me? Thanks.

[pradyunsg]

This is the same as #9229. Closing and merging this issue into that.

---

Since this hasn't been stated directly yet: folks depending on the legacy resolver's "eh, ignore the mismatched URLs" approach -- you'll need to change your workflow to adapt to this change. Going forward, pip is not going to allow URL-based packages to have mismatched URLs during installation.

This is a part of the changes made as part of pip's new dependency resolver, which makes it stricter about edge cases like this.

[Tbilgere-td]

"eh, ignore the mismatched URLs" approach -- you'll need to change your workflow to adapt to this change

@pradyunsg Hold on a minute here we are trying to manage repo upgrades without having to release dozens of repos everytime a single change is made to ONE common repo. That doesn't seem like a crazy far out and/or lazy idea but, just to be clear, that is the stance pip is taking here?

[andylamp]

@pradyunsg @Tbilgere-td I think I need a clarification on that as well - as I find this to be a legitimate use-case.

[uranusjr]

@Tbilgere-td Could you clarify what you mean by “unable to update common library repos without breaking every subsequent repo”? From what I can see, your service depend on commoncode 1.0.0 exactly, so it’s only natural that app specifying commoncode 1.0.1 would break it. I am not sure how you expect things to work.

[Tbilgere-td]

@uranusjr I would expect it to work in the same fashion as any other dependency does, I should have someway of describing something like a allowable range and/or explicit allows/denials.

[uranusjr]

But is the current behaviour not what “other dependencies” do? If you have

# service
commoncode==1.0.0

# app
commoncode==1.0.1
service==2.1.0

The installation is supposed to fail, because there’s no way to accept both commoncode 1.0.0 and 1.0.1.

I should have someway of describing something like a allowable range and/or explicit allows/denials.

You do, that’s what versions are for. You can specify

# service 2.1.0
commoncode~=1.0.0

# app
commoncode @ git+git://github.com/repoOrg/[email protected]
service @ git+git://github.com/repoOrg/[email protected]

(Or you can specify the URL in service and use version in app if that’s what you want instead.)

[Tbilgere-td]

You do, that’s what versions are for. You can specify

# service 2.1.0
commoncode~=1.0.0

# app
commoncode @ git+git://github.com/repoOrg/[email protected]
service @ git+git://github.com/repoOrg/[email protected]

@uranusjr okay but how does this work for the case where the app doesn't call for common but still calls for Service? This would also mean we wouldn't be able to build Service independently correct?

[uranusjr]

Then you specify the URL in service and apps that don’t depend on service?

But it sounds to me like you have a more complicated project dependencies than the URL is designed to handle. I would strongly encouraged switching to a setup with internal (i.e. non-PyPI) package index, and specify the index to fetch commoncode etc. with --index-url instead.

[aaraney]

@uranusjr to steer the conversation back to a remark you made earlier:

There’s no way to know whether different URLs point to the logical same thing without actually downloading and potentially building the content, which is very problematic in more than one way.

First off, I don't disagree from a fundamentalist stand point, however my problem is with treating edge case behavior like it is the straight and narrow. Sure, there are edge cases where minute differences in a URL causes a logical difference in retrieved content -- but again this just seems like an edge case. I see the utility of leaving this as an optional feature to capture those edge cases, but this doesn't seem appropriate to force on all users. Additionally,

So systems using URLs for dependency specification generally impose restrictions over its usage. Go (for which URLs are one of the two primary ways to specify dependencies), for example, behave the same as pip here.

I don't think it's a fair comparison between python and go, the two languages serve different purposes and communities. Personally, the more difficult it is for a new user or any user to accomplish their goal using python is non-pythonic. The growing prevalence of type hints is a perfect example of this, in many ways type hints greatly benefit the community; however they are optional and without, python still operates just fine. Of course there is a need for rules and guidelines, but this level of strictness seems like it's gains don't outweigh the benefits for the majority of users. With a language that supports so many open source projects, these kinds of restrictions -- I project -- will only decrease the number of functioning projects over time. The burden should not be placed on the package maintainer, as in many cases, they are not responsive and shouldn't have to be.

Also, as a counter to first argument:

There’s no way to know whether different URLs point to the logical same thing without actually downloading and potentially building the content, which is very problematic in more than one way.

It is also true that there is a case when a user may be restricted from using, for example, port 22 and would run into issues when a package specifies its dependencies using, in this case, the ssh protocol. Likewise, to speculate, it's also possible for a server using a reverse proxy to direct a client to different information at GET time. Both of these are highly unlikely and seem, frankly, silly but the important thing to note is that they are also an edge case. To come full circle, in the case of resolving dependencies via URL should this level of restraint be applied? Personally I don't think so. No matter is I specify that I want the information over ssh, https, or git this should not break packages.

[andylamp]

@uranusjr then could you please then suggest on how to fix this in both requirements.txt as well as in setup.py while adhering to the new resolver rules?

For example within setup.py the links in

install_requires=[
{name}@git+https://{tok}@github.com/{my_acc}/{name}.git
# more packages...
]

Whereas in requirements.txt we have the following:

git+https://{tok}@github.com/{my_acc}/{name}.git#egg={name}

How can I align these with the new resolver? We do not use different version but rather always the latest once for all internal packages.

[uranusjr]

From what I can tell, all you need to do is to use the same format in both setup.py and requirements.txt?

[andylamp]

is either format allowed in both instances?

[uranusjr]

The egg= format cannot be used in setup.py. The @ format (PEP 508) is universal.

[andylamp]

@uranusjr ok, that's helpful; would it be possible to add this to the documentation as a migration note or something along those lines? I am sure it would save a lot of time in many instances where this type of conflict exists.

[Tbilgere-td]

Per @uranusjr 's comments above I found that I was able to define a range in install_requires of setup.py and then a git repo URL in requirements.txt. This allows us to mostly maintain the workflow we had before, makes version bumping slightly more confusing but, as long as we are able to continue to point to a branch and/or commit for testing and proof of concepts and only have to deal with version consistency in the case that breaking changes are introduced we're pretty happy. Thanks uranusjr!

E.g.

Requirements.txt in Service:

commoncode @ git+git://github.com/repoOrg/[email protected]

Setup.py in Service:

install_requires=[
'commoncode~=1.0.0',
]

Requirements.txt in App:

commoncode @ git+git://github.com/repoOrg/[email protected] # or @commit_hash or @branch_name 
service @ git+git://github.com/repoOrg/[email protected]

[justvanrossum]

I'm trying to figure out whether an issue we're facing is the same as this issue or not. Our situation:

• Package A (on PyPI) depends on package B (also on PyPI)
• Application depends on A and B, but would like to use the latest dev version of B from git, and therefore specifies a git url in its requirements.txt

This used to work, but doesn't anymore with the new resolver: ResolutionImpossible.

Is this case no longer supported or am I doing something wrong?

[sbidoul]

@justvanrossum I use this scenario daily with the old and new resolver and it works just fine. Can you provide more details about your issue ?

[andylamp]

actually, this the new resolver is super problematic; in some instances, it goes into backtracking loops and I have reported multiple examples through the pip feedback form that this happens. In its current form, it is unusable - in my opinion, it should not be used in production by default unless these issues are sorted out.

This is of course, apart from all issues surrounding installing packages from source that are mentioned in the thread.

[justvanrossum]

It looks like my issue has to do with extra requirements: A depends on B[extra], and the application installs A and <git-url-to-B> (without the extras). It does lots of backtracking and eventually fails.

In some cases, writing B[extra]@<git-url-to-B> appears to help, in other cases it doesn't.

The smallest example I can create (using public packages) is the following.

1. This one backtracks unnecesarily, doesn't fail, but obsolete stuff gets installed:
   pip install fontParts defcon@git+https://github.com/robotools/defcon

2. This one appears to work correctly:
   pip install fontParts defcon[pens]@git+https://github.com/robotools/defcon

3. This one also backtracks unnecesarily, doesn't fail, but obsolete stuff gets installed:
   pip install fontParts defcon[pens,lxml]@git+https://github.com/robotools/defcon

[justvanrossum]

The fact that the third version of my example behaves badly baffles me. Does anyone have some insights regarding using the new resolver + git URLs + extra requirements? Should I open a new issue? I'm less and less sure how my problem relates to this issue.