Provide a way to allow explicit overrides of subdependency versions of top level packages in pipfile

[juanformoso]

I apologize if this was raised before, but if it was I was unable to find it.

We're trying to start using pipenv but we ran into a use case in which we'd need to override a declared version of a library in another library, knowing well that the first library would continue to work with the newer version even though it declares a specific one.

For context, the problem is that a library requires pyopenssl==0.13, but in high sierra the minimum version that works is 0.14 because of the OpenSSL version (cloudera/hue#205) so we need to force at least that version

Right now we have our project working because pip supports overriding depedencies: pypa/pip#988 (comment)

However with pipenv, I'm having problems.

If I explicitly install a version that is higher than the one declared in a library, I get a conflict error

New dependencies found in this round:
(...)
** adding [u'pyopenssl', '==0.13,==0.14', '[]'] **
(...)
Could not find a version that matches u'pyopenssl==0.13,==0.14

So I'd need pipenv to let me override a version, taking full responsibility about the depedency that could potentially not work but willing to take the risk.

Is there a way to do that already? If not, could this be added as a feature request?

[techalchemy]

Your best bet currently is to put the dependency you want to override in dev-packages -- these will be overridden by dependencies in packages. So you can put the library that requires pyopenssl==0.13 in dev-packages and then pin pyopenssl==0.14 in packages and you should be able to pipenv lock --dev and pipenv install --dev with no problems

[jtratner]

I think this is a pretty legit concern btw - I spent about a week upgrading all the little packages inside and outside my company that inaccurately pinned stuff so we could finally use pipenv. Not sure there’s a great solution per se (except for fixing all the setup.py’s of the world)

…

On Thu, Apr 5, 2018 at 7:33 PM Dan Ryan ***@***.***> wrote: Your best bet currently is to put the dependency you want to override in dev-packages -- these will be overridden by dependencies in packages. So you can put the library that requires pyopenssl==0.13 in dev-packages and then pin pyopenssl==0.14 in packages and you should be able to pipenv lock --dev and pipenv install --dev with no problems — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1921 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABhjqwcuJV6tzMDkoDzBgHaErSE0Y_b4ks5tltQVgaJpZM4TIyH-> .

[juanformoso]

Not sure there’s a great solution per se (except for fixing all the setup.py’s of the world)

In my very humble opinion, the solution would be to handle it like pip does, let me (the developer of an application that uses a library) decide if I want to take the risk of using a newer version of a sub-dependency, under all the disclaimers of the world.
In my example I know for a fact that the library works ok with a newer version of pyopenssl

Your best bet currently is to put the dependency you want to override in dev-packages

Thank you for this workaround @techalchemy, I'll see if I can make it work like that. However this was but a minuscule example, there are other use cases (a newer version of Flask to use a new feature, or other dependencies for the same reason... the bigger the app, the bigger the chance I'd need to override something), it feels kind dirty solving it like that.

Perhaps a new section called overrides to handle this? All that'd be needed is for pipenv to not "break" when it encounters a conflict if the dependency is listed here? Or maybe a simple argument to skip this validation and just use the newest found?

[techalchemy]

Yeah I agree, I left this issue open because I don't like workarounds and I think it is the best issue we've had on the topic in terms of articulating the exact nature of what would be desired here, and precisely why the current behavior can't accommodate it. If I touched the issue and I didn't close it, I was already somewhat convinced

[megahall]

I just ran into this myself, where a package requires an outdated version of dj-database-url and it's impossible to override in Pipfile while it would be trivial to override in requirements.txt.

[uranusjr]

Since requirements.txt is mentioned I want to mention that it being easy to override is not exactly a good thing. Generally you want to detect incompatibilities ahead of time. Say if

• I am developing a site with Django 2.0
• I’m looking into django-myuberplugin, which requires django<2.0

This setup would never work.

So I’d want pipenv install to fail with this Pipfile:

[packages]
django = ">=2.0"
django-myuberplugin = "*"

instead of overriding the sub-dependency, resulting in a broken (and potentially subtly buggy) environment.

I’m not objecting to a way to override subdependency versions—although I’d be satisfied if a tool does not allow it. The version conflict should be resolved by the package maintainer (in this example the author of django-myuberplugin). If a way is provided, it needs to be explicit, and make it clear it is the user’s intention to do so. Something like this could work:

[packages]
django = { version = ">=2.0", override = ["django-myuberplugin"] }
django-myuberplugin = "*"

[juanformoso]

@techalchemy would you like me to propose a PR for this?

I'm far from an expert in python (I only started working with it about 2.5 months ago) but I can give it a try.

If so, which solution would you prefer? A new section at the dev-packages level with a list of overrides maybe?

[ncoghlan]

I think @uranusjr's suggestion of an override at the level of individual package<->requirement pairs would make sense, since that can be read as "use this dependency declaration to override the corresponding declaration for these other packages".

So you'd still need to do the detective work to figure out which packages are causing conflicts, but once you've found them, you could work around the issue immediately in Pipfile, rather than needing to wait for new releases of the projects with overly specific dependency declarations.

[techalchemy]

Yeah this feels like a pipfile level change— an extra package argument like override = “<thing to override>” or some better word I haven’t thought of

[techalchemy]

After discussing this, I think we want to go with django-myuberplugin = {version = ">=x.y", ignore = ["django"]}

[uranusjr]

I don’t think it is the same. Please open a new issue.

[till]

I have an interesting issue which seems to be a dependency resolution problem (or circular dependency somewhere) — or a case for overriding/enforcing dependencies of dependencies.

My objective is to replicate the setup of a vendor (done on a remote server), so I am using Python and pipenv locally to install the required software — down to the Python version. And most of this works great!

When I am done, I have incompatibilities though (errors from the openstack CLI) which I think come from an update to one of the packages.

I started with a clean install, trying to figure out how or where the openstacksdk package is updated (despite it having a ==0.9.13 version pinned in my Pipfile) and it turns out, one of the dependencies of it, probably requests an upgrade because someone though >= is a good idea in requirements.

In my case, it turns out that one of the dependencies lists a dependency on osc-lib as >=1.2.0, so in case, it installs 1.11.x and then upgrades the openstacksdk again. The fix/work-around, is to install/list it first with an explicit requirement on 1.2.0 and then it proceeds with that and openstack --help returns meaningful output (instead of module errors).

So I am guessing that it's impossible to go back and fix releases, especially those 1+ year old, etc.. But how to deal with it in the present?

I thought I'd add this as another use-case of where to override dependencies in packages. Maybe the output could be improved to list which dependency requested the output? Or something like "select lowest version" or "honour Pipfile"? Selecting the lowest version (1.2.0) would have solved this as well.

I am also confused a bit by the output, "successfully installed", followed by "unable to resolve dependencies".

$ pipenv install --keep-outdated openstacksdk==0.9.13
Installing openstacksdk==0.9.13…
Collecting openstacksdk==0.9.13
  Using cached https://files.pythonhosted.org/packages/d2/c0/4ed0e87c58191a56b54d8d1f35c3df930e17e574083c93d7a3bcea4a266b/openstacksdk-0.9.13-py2.py3-none-any.whl
Collecting pbr>=1.8 (from openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/f3/04/fddc1c2dd75b256eda4d360024692231a2c19a0c61ad7f4a162407c1ab58/pbr-5.1.1-py2.py3-none-any.whl
Collecting keystoneauth1>=2.18.0 (from openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/39/e8/bd409d9ed0dde60d5f2db50398e11ec95dd7eeed62fd40a7a319720faa6f/keystoneauth1-3.11.1-py2.py3-none-any.whl
Collecting six>=1.9.0 (from openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting os-client-config>=1.22.0 (from openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/e5/af/2f3bf859a25c98993f85ff1eb17762aa49b2f894630ebe99c389545b0502/os_client_config-1.31.2-py2.py3-none-any.whl
Collecting stevedore>=1.17.1 (from openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/35/fa/8683fab2a6e15ecfe107996e56fab91e52fe3ec0b40ca9440a0e1ffe6892/stevedore-1.30.0-py2.py3-none-any.whl
Collecting iso8601>=0.1.11 (from keystoneauth1>=2.18.0->openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/ef/57/7162609dab394d38bbc7077b7ba0a6f10fb09d8b7701ea56fa1edc0c4345/iso8601-0.1.12-py2.py3-none-any.whl
Collecting requests>=2.14.2 (from keystoneauth1>=2.18.0->openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl
Collecting os-service-types>=1.2.0 (from keystoneauth1>=2.18.0->openstacksdk==0.9.13)
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.14.2->keystoneauth1>=2.18.0->openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.14.2->keystoneauth1>=2.18.0->openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.8,>=2.5 (from requests>=2.14.2->keystoneauth1>=2.18.0->openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.14.2->keystoneauth1>=2.18.0->openstacksdk==0.9.13)
  Using cached https://files.pythonhosted.org/packages/56/9d/1d02dd80bc4cd955f98980f28c5ee2200e1209292d5f9e9cc8d030d18655/certifi-2018.10.15-py2.py3-none-any.whl
Installing collected packages: pbr, six, stevedore, iso8601, urllib3, chardet, idna, certifi, requests, os-service-types, keystoneauth1, os-client-config, openstacksdk
Successfully installed certifi-2018.10.15 chardet-3.0.4 idna-2.7 iso8601-0.1.12 keystoneauth1-3.11.1 openstacksdk-0.9.13 os-client-config-1.31.2 os-service-types-1.3.0 pbr-5.1.1 requests-2.20.0 six-1.11.0 stevedore-1.30.0 urllib3-1.24.1

Adding openstacksdk to Pipfile's [packages]…
Pipfile.lock (a258b9) out of date, updating to (ad4c3a)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches openstacksdk==0.9.13,>=0.13.0
Tried: 0.6.0, 0.6.0, 0.6.1, 0.6.1, 0.6.2, 0.6.2, 0.7.0, 0.7.0, 0.7.1, 0.7.1, 0.7.2, 0.7.2, 0.7.3, 0.7.3, 0.7.4, 0.7.4, 0.8.0, 0.8.0, 0.8.1, 0.8.1, 0.8.2, 0.8.2, 0.8.3, 0.8.3, 0.8.4, 0.8.4, 0.8.5, 0.8.5, 0.8.6, 0.8.6, 0.9.0, 0.9.0, 0.9.1, 0.9.1, 0.9.2, 0.9.2, 0.9.3, 0.9.3, 0.9.4, 0.9.4, 0.9.5, 0.9.5, 0.9.6, 0.9.6, 0.9.7, 0.9.7, 0.9.8, 0.9.8, 0.9.9, 0.9.9, 0.9.10, 0.9.10, 0.9.11, 0.9.11, 0.9.12, 0.9.12, 0.9.13, 0.9.13, 0.9.14, 0.9.14, 0.9.15, 0.9.15, 0.9.16, 0.9.16, 0.9.17, 0.9.17, 0.9.18, 0.9.18, 0.9.19, 0.9.19, 0.10.0, 0.10.0, 0.11.0, 0.11.0, 0.11.1, 0.11.1, 0.11.2, 0.11.2, 0.11.3, 0.11.3, 0.12.0, 0.12.0, 0.13.0, 0.13.0, 0.14.0, 0.14.0, 0.15.0, 0.15.0, 0.16.0, 0.16.0, 0.17.0, 0.17.0, 0.17.1, 0.17.1, 0.17.2, 0.17.2, 0.18.0, 0.18.0, 0.18.1, 0.18.1, 0.19.0, 0.19.0
There are incompatible versions in the resolved dependencies.

[techalchemy]

The thing is, installation is not the same as dependency resolution. If all you care about is installation, pip already has you covered. If you don't want your dependencies to get obliterated by one another in the process of serially installing them, then dependency resolution is rather important.

In any event, all you've shown is your attempt at pinning a version of openstacksdk which you already knew was not compatible with the dependencies you have installed. Like the message says, you have something in your packages that is demanding a higher version. You can't simply brute force a lower version until that problem is solved. You can run pipenv install --skip-lock and then pipenv graph to inspect the situation and see the hierarchical structure involved, or you can run pipenv lock --verbose (but you definitely need to unpin your openstacksdk dependency).

[till]

@techalchemy my problem is that even though I pinned openstacksdk in my Pipfile, it got updated because one of the sub(-sub)-dependencies requested something else. And there was no error for that. The install "completed" as dependencies could be resolved. But the result was broken software.

The problem of the constraints is not something that you can fix. That's PRs to the individual projects, etc..

The problem is that it didn't honour my pinned version. It just upgraded it.

Maybe that's a different issue? If so I apologise for adding noise here.

(And yes, I ran --skip-lock and the graph to inspect it and then installed other dependencies first so I could work around it with --keep-outdated (as I said).)

[altendky]

@till, the previously proposed solutions would roughly allow you to specify that the particular package's dependency on >=0.13.0 should be ignored. The effect would be that a pin you do to ==0.9.13 could be satisfied (assuming no other conflicts). Would this be satisfactory?

I generally like the idea of being able to override a particular individual thing, a single package's requirement for another package/version. I would note that there are various possible things you may want to express when overriding though so a general solution that let's you specify a substitute requirement including regular version specifiers and even multiple dependencies would be good.

1. The sub-dependency isn't actually required at all for this use case
2. The sub-dependency version restrictions are incorrect and should be replaced
   • Including keeping the dependency but with no version restriction)
3. Something else could be used to satisfy the requirement
   • PIL->pillow or other forks or compatible package alternatives etc
4. Multiple other packages
   • Say a package gets split in a newer version and we need to specify multiple things, or their versions

So it's really like we want to be able to take the existing dependency, say openstacksdk>=0.13.0 and replace it with pretty much any other arbitrary dependencies. It seems like it would be good to keep this substitution related to that package rather than simply deleting the dependency and making our own unrelated dependency elsewhere. But perhaps it can be argued that that causes significant complication with minimal value, I haven't thought through it in detail yet.

Now we get more complicated because really this should probably be reviewed if the package with the overridden dependency ends up at a new version... In other words, a notification/warning/error in this case would be helpful. Good fun.

This seems like a messy rabbit hole but I can't convince myself that it's not an important use case. There's too much room for opinion and pragmatic 'there will be errors' to just say 'go fix the original package'.

[till]

@altendky yes, that! ;) It's a follow the white rabbit thing! :D

On a more serious note, I realise the problem is also with the packages I am using. E.g. >= to say the least. Maybe it would also suffice to add a warning and not add the complication of fixing package requirements from the installer?

"Your Pipfile contains ==foo-1.3.8, but bar.foo.foobar requests >=foo-1.4.0."

bar.foo.foobar could be a tree to say:

• my first dependency is bar
• bar's dependency is foobar

It just doesn't do that at all, instead I had broken code.

Maybe that doesn't introduce a whole world of new hurt to this project? Then I can still figure out how to make things work, which is not ideal (for me), but would have been much more straight forward. The graph seems to have most of the info already, maybe it can be presented in a different form.

[ctaggart]

@techalchemy, I like your suggestion. I would be really useful to me right now if I could use ignore do this:

azure-cli = { {version = ">=2.0.66", ignore = ["cryptography"] }

because of setup.py dependency errors like Azure/azure-cli#9629.

[techalchemy]

Thanks for bringing this back up -- I've spent a lot of time thinking this over and I currently am happy we never implemented this. I think that since the tool we maintain is pipenv we need to respect Pipfile as the single source of truth

That means that if you specify a pin in your pipfile, I'm thinking we should probably raise a warning or something to let you know that it is in conflict with something else we encountered, but to prefer the explicit dependency you provided and assume you are doing so to override something else.

Currently, pipenv has a significant usability issue around this and I encounter it myself somewhat regularly -- it's pretty challenging to override a conflict you know is invalid. That's not something I'm ok with & I feel the best approach (per @altendky's insights above among others) is to just allow overrides.

/cc @ncoghlan @uranusjr -- this is mainly a UX concern on the design here because it definitely needs to be possible

[tevansuk]

Since requirements.txt is mentioned I want to mention that it being easy to override is not exactly a good thing. Generally you want to detect incompatibilities ahead of time. Say if

• I am developing a site with Django 2.0
• I’m looking into django-myuberplugin, which requires django<2.0

This setup would never work.

So I’d want pipenv install to fail with this Pipfile:

[packages]
django = ">=2.0"
django-myuberplugin = "*"

instead of overriding the sub-dependency, resulting in a broken (and potentially subtly buggy) environment.

I’m not objecting to a way to override subdependency versions—although I’d be satisfied if a tool does not allow it. The version conflict should be resolved by the package maintainer (in this example the author of django-myuberplugin). If a way is provided, it needs to be explicit, and make it clear it is the user’s intention to do so. Something like this could work:

[packages]
django = { version = ">=2.0", override = ["django-myuberplugin"] }
django-myuberplugin = "*"

I think this is grabbing the donkey by the wrong end. The dependency that we want to override is not on django, it is on django-myuberplugin. I'd much prefer to see something like this:

[packages]
django = ">=2.0"
django-myuberplugin = { version = "*", override=["django>=2.0"] }

That clearly says to me that the project:

• wants django >= 2.0
• wants django-myuberplugin, and pipenv should consider that library to have a dependency of django>=2.0, regardless of what version of django is actually specified by django-myuberplugin

For reference, I came across this issue when trying to work out a way to use a library (flake8-black) that has a dependency (black) that has no release that is not pre-. The latest flake8-black specifies that it wants black>=19.3b0; the project Pipfile specifies that it wants black==19.3b0. This causes pipenv to look for the latest released version of black, there is no released version and so kabloom. If I was able to specify an override to the deps for flake8-black, then I would be able to lock.

[uranusjr]

To me the two are the same intention expressed differently (using the same keyword):

• I want django>=2.0, and this intention overrides django-myuberplugin’s.
• I want django-myuberplugin, but override its django intention to django>=2.0.

FWIW both makes sense to me and I don’t have a preference either way. But the more important issue here is to find someone interested in actually implementing this; design details don’t really matter otherwise 😛

[ncoghlan]

@tevansuk For the specific case of pre-release only packages, see #1760 for some further discussion. Fixing that doesn't need an override syntax, it needs improvements to the way pip handles pre-releases.

[timorthi]

I am running into the same issue between two of my dev-packages. apache-airflow==1.10.7 and faker>=2.0.2 yields the following resolution:

adding ['text-unidecode', '==1.2,==1.3', '[]']

which is a non-breaking constraint (API-wise) because only the license changed between 1.2 and 1.3

Unfortunately @techalchemy 's workaround did not work for me, so I resorted to pinning faker==2.0.1 which requires the same text-unidecode version as apache-airflow==1.10.7, otherwise I'd be stuck and unable to use apache-airflow and faker at the same time with Pipenv.

Is there still any plan to implement explicit overrides? Looks like there was some discussion in this thread but it fizzled out with no resolution. If resources are lacking, I'm also happy to take a stab at a prototype to jumpstart this feature.

[uranusjr]

Would it be an option to bring this up to either apache-airflow or faker maintainers? I’m quite sure they have explicit reasons to pin versions like this (likely licensing), and in that case it actually serves as an example against explicit version overwrite, because that is exactly the thing you shouldn’t do in this situation.

[timorthi]

@uranusjr That's a fair point, thanks for pointing that out. apache-airflow is on text-unidecode==1.2 which looks like the version with the more restrictive license so I'll bring it up to them.

That aside, how do pipenv maintainers feel about this override feature? I'd imagine that this situation comes up quite often, as this thread and other GH issues have alluded to.

[jperras]

It comes up semi-often. I've had to sometimes spend days trying to track down the maintainers of some package dependencies and ask/beg them to widen their proverbial net, so to speak.

[uranusjr]

@timorthi I am not against the idea. I think you can start by discussing this with pip-tools maintainers for directions (Pipenv currently uses it internally for dependency resolution). It would likely be a very involved task though.

[n1ngu]

Isn't the new dependency resolution enforced by the latest versions of pip making this issue a no-go? This started as "pip permits this overrides" but AFAIU this won't be so anymore and I don't see how pipenv could work around that in any way.

[djmore4]

I think @timorthi 's point about faker is even more relevant now than it was at the time of their comments. If a dependency in my project stops being maintained or stops being an acceptable package to use, forking that project to create a drop in replacement or finding another project which operates as a drop in replacement is a use-case that should be supported in my opinion.

Given that this is fully based on user experimentation however, it should be up to the programmer to determine what sub-dependencies from the package described in the Pipfile should be included. This can simply be done by including those packages themselves in the Pipfile.

pip allows for installation of packages with the --no-deps flag which could be used with the methodology I've described to allow developers to manage their sub-dependencies with more control and without needing to wait for packages to update their own dependencies.

[xsrvmy]

On the topic of drop in replacements, this is already supported in NPM 8 via overrides. It's also something that's required for GPL/LGPL compliance (although falling back to pip in that case also works).

[ddevault]

If a dependency in my project stops being maintained or stops being an acceptable package to use, forking that project to create a drop in replacement or finding another project which operates as a drop in replacement is a use-case that should be supported in my opinion.

This is a really important use-case to consider and most other package managers handle this case. Why was this closed as "not planned" without addressing this need?