Ability to override/ignore sub-dependencies

[zehauser]

• I have searched the issues of this repo and believe that this is not a duplicate.

(However, it is related to #436).

Issue

In the dark, old world of Python packaging, sub-dependencies are handled very poorly. If I recall correctly, pip will happily install a sub-dependency despite conflicting versions being specified by two direct dependencies... in fact I think which version it ends up installing depends on the order in requirements.txt. Yuck! Only very recently has it even started issuing a warning for cases like this.

In contrast, poetry does this right. It computes the entire dependency tree and will complain if there are conflicts anywhere in the tree.

But... many packages out there are not specifying their dependencies properly. Even if they are, there's always the possibility that their specified dependencies are a tighter range than they strictly need to be.

Is there a way to tell Poetry to force a specific version (or version) range of a dependency in cases like this — or in other words, to ignore a dependency specification of another dependency somewhere in the tree? If not, should there be?

[swarmer]

+1
Looks like this can be enough of a problem to make poetry unusable for many non-library projects, so this is quite important.

[zehauser]

@sdispater You must be super busy, but what are your initial thoughts on this?

[sdispater]

I don't think this is desirable. This would require a lot of work and add complexity to the resolver - which is already complex due to the Python specific ecosystem - because of some packages specifying their dependencies poorly. I don't want Poetry to have to make up for a lack of proper tools or proper specifications, there is already a lot of work to be done as it is.

This is the job of each package's maintainers to ensure their dependencies are correct and loose enough to not create conflict.

If we ever want to have an ecosystem similar to what other languages already have, we have to draw the line somewhere and enforce everyone to contribute to the common goal. Poetry helps with that by making it easier to build and manages Python projects.

[danielkza]

@sdispater While you are completely correct in principle, in reality there are cases where not being able to perform overrides means being completely unable to install some packages, specially when multiple projects specify conflicting version constraints.

Waiting for the whole ecosystem to improve will take years, and in the meanwhile Poetry is unusable in some cases, which is a shame considering it brings lots of improvements in other areas.

As a practical example, awscli requires a particular version range colorama while docker-compose requires another. That makes it completely impossible to install both simultaneously, and neither of the projects are willing to make a change. But in practice both programs work correctly (since I've been using them in the same venv for years).

Even other package managers that work with more "well-behaved" ecosystems allow overrides e.g. Maven, SBT and others in Java world.

[fsworld009]

Would also like to have this option, especially as of now there are packages cannot be installed because poetry fails to parse their setup.py, like #904.

[drunkwcodes]

Uncompromising claim is good to formalize the package versioning.

But the conflicts are existing for sure. And packages are ofter more useful than it claims.
An error mitigation strategy is a must.

I think it's good to have a export option to write those conflicts in a requirements.txt.
And we can review & edit them like dealing a merge conflict.

So the work flow will be like:

1. poetry export --ignore-conflicts
2. Resolve conflicts in requirements.txt.
3. pip install -r requirements.txt

This can be a plug-in if necessary. Maybe related to #558 #663 & #955.

[RXminuS]

I've ran into exactly the same issue as @danielkza. But I don't think we need to make it that complicated, if you could just tell poetry to completely ignore the dependencies specified by a package then you can always just specify your own.

Also at the moment, it can take a really long time for poetry install to try every combination of dependencies, there must be a better way than iterating down the version numbers?

[floer32]

I am sympathetic to the points about having some way to override or accelerate or skip resolution on an opt-in basis, but also always want to avoid making compromises to the "real, thorough" way of doing a task like this for the sake of proactively optimizing some efficient way...

Explicit control seems best... So you can do real/thorough at maintenance times, in master CI/CD or base docker/VM image baking, etc etc, then more efficient when there has not been a change in the reqs. (And that 'has there been a change?' part is well-solved by things like git-checks and/or docker image caching, and I do think it's best to keep that burden on those tools, and really avoid complicating concepts of cache,state, or history in Poetry itself.)

---

I like @drunkwcodes 's workflow suggestion, and agree that #558 seems related. It really seems there is another subcommand or workflow waiting to be fleshed out...

As @drunkwcodes says maybe it could be a plugin. Personally I am fine "dropping down" to pip for these cases, and then continuing to rely on Poetry as the "master" and what I rely on for longevity and maintenance.

[floer32]

brainstorming what this new subcommand or plugin may look like

This more-locked installation should probably be tied in with wheels (re: #558). It may even directly involve using pip to keep things explicit (kinda relates to #1012), but just have more known-good ways to handle in the poetry commands or flags.

two related/compatible use-cases

(a) poetry -> reqs, per @drunkwcodes example, and helps address #697

poetry export --ignore-conflicts
# [as needed] "Resolve conflicts in requirements.txt" - manually, or interactive prompt option? 
pip install -r requirements.txt

(b) reqs -> wheels -> install-with-pip

... installing with pip doesn't necessarily conflict with Poetry, you can do the pip-install-from-wheels / with-whatever-options, then call poetry install and it will do almost nothing if things look good, otherwise it'll install just the little bit of drift.

... seems to help address #558, and by directly exposing pip may make sense for cases like #1012

poetry export --ignore-conflicts 
# [as needed] "Resolve conflicts in requirements.txt"  - manually, or interactive prompt option? 
pip wheel -r requirements.txt  # assumption: you've set PIP_WHEEL_DIR
pip install --find-links "file://$PIP_WHEEL_DIR" -r requirements.txt

imagining new options to bring things together

implicit in example below: poetry export should take an argument for what its output is named (currently it is hardcoded to requirements.txt, it should take a filename or allow stdout output (convention being -))

intermediate idea 1

pip wheel -r <( poetry export --ignore-conflicts --output - )  # pip respects PIP_WHEEL_DIR or --wheel-dir, see pip docs
poetry install --find-links="file://$PIP_WHEEL_DIR"  # ..? does it need reqs again?

intermediate idea 2 - seems clean to me, but could be disagreeable to some

poetry export --ignore-conflicts --fetch-wheels --wheel-output=./wheelhouse --output=poetry.wheelhouse.lock
poetry install --find-links=./wheelhouse --extra-lock=./poetry.wheelhouse.lock

the idea in that second one being, in the event of a conflict, poetry would fall back to the --extra-lock file that was given. Or maybe call it --extra-constraints and .txt instead of .lock, if it seems like concepts are getting blurry.

intermediate idea 3 - does require one pip call, but very explicit?

If that's too weird, this is a fairly clean compromise:

poetry export --ignore-conflicts --wheel-output=./wheelhouse --output=poetry.wheelhouse.txt
pip install --find-links "file://./wheelhouse" -r poetry.wheelhouse.txt
poetry install --find-links=./wheelhouse

... in that last idea, poetry install is not even paying attention to poetry.wheelhouse.txt (which is just a requirements.txt, just trying to name it so its origin and usage context are clear). it's using its own pyproject.toml/poetry.lock, only, in that one; but it would prefer the wheelhouse before using PyPI.

poetry install would just happen to ignore already-installed cases and not hit conflicts, based on that pip install step.

(I also omitted --fetch-wheels because maybe that can just be implicit from the --wheel-output flag being present.)

interactive conflict resolution?

In those three examples I was trying to suggest ways that the intermediate manual step of hand-resolving conflicts, could be skipped, yet still having determinate results (if one checks the export'd artifact into VCS.

There could still be cases where something like,

poetry export --interactive-conflicts

and/or

poetry lock --interactive-conflicts

could be relevant; with prompts asking what choice is preferred. Since they are supervised maintenance tasks they could be interactive ...

p.s. pip flags in general

Also there still seems to be a general need to allow most/all pip options/env-vars/flags to be controlled from Poetry config and/or CLI. Will help a number of use cases.

[drunkwcodes]

I think the command can even be renamed to poetry resolve, which only outputs conflict warnings or errors by default, and has -o, -v, --tree options. Since somtimes we need the dependency resolution result but not to export.

• -o option is same as poetry export -f
• -v option means verbose and shows debug messages of the resolver.
• --tree option will lists dependency graphics in terminal.

P.S Reply to interactive mode:

interactive conflict resolution?

In those three examples I was trying to suggest ways that the intermediate manual step of hand-resolving conflicts, could be skipped, yet still having determinate results (if one checks the export'd artifact into VCS.

Spawn a process to open text editor like vi[m], emacs, nano etc. right after file generation sounds adequate.

[floer32]

That sounds really great. Less steps the better and your suggestion coheres very well with the existing functionality and the intentions we are discussing.

[epage]

So the use cases we've been running into with proper dependency resolution: packages we get value out of that have low activity or are unmaintained that happen to work with newer versions of dependencies. This is much worse than in Rust because Rust supports multiple versions of a dependency existing in your dependency graph. Here, we need one compatible version through the entire graph.

The proposed solution for this seems to rely on manually resolving the conflcit on every upgrade. This seems tedious for a process (poetry update) that we should be able to automate (i.e. via Dependabot). Personally, my preference would be for the override to be recorded in the pyproject.toml file so it can be a persistent input to the process. For example, we could have a table of patches to the dependency graph that override the version constraint.

[danields761]

Strict dependency resolution reveals a lot of issues with a lot of packages in Python ecosystem. There should be a least a way to handle dependencies by yourself without the need to contact with package maintainer, wait for patches etc.

I am agree with @epage that there should be a way to override dependency-graph

[kylebebak]

This makes poetry very difficult to use in all sorts of real-world projects.

This problem exists with other package managers, and has been solved before. yarn, for example, handles a similar (but more complicated) problem with resolutions: https://yarnpkg.com/lang/en/docs/selective-version-resolutions/.

It's also worth noting that pip will let you install deps that are incompatible with other deps, and will print something like the following to the console:

django-stubs 0.12.1 has requirement mypy<0.710,>=0.700, but you'll have mypy 0.711 which is incompatible.
Installing collected packages: mypy
Successfully installed mypy-0.711

I totally agree that poetry's default behavior should be to prevent users from installing packages whose deps can't be properly resolved, but users need to be able to opt out of this restriction (as they can with pip), because it's a common real-world situation. Telling users that their best option is to open issues with every package that has stale or incorrect version requirements is not realistic.

My vote is to keep the implementation as simple as possible, and to make the user responsible for managing the risk of ignoring conflicts. After all, it would be opt-in behavior.

I propose making this a flag on poetry add: poetry add <package> --ignore-conflicts. This would add an ignores-conflicts flag to the dependency in pyproject.toml, in much the same way --allow-prereleases adds an allows-prereleases flag.

[zehauser]

Unfortunately this continues to be an enormous issue that prevents using poetry in many real-world projects. 😢Which sucks, because poetry is awesome and getting awesomer!

Of course there won't be any solution overnight, but it would be great to get an updated opinion on this subject from @sdispater or another maintainer/triager (@stephsamson?).

[kylebebak]

@sdispater @stephsamson

Any update on this? I think this thread makes it clear that being able to ignore conflicts on an opt-in basis is necessary to make Poetry usable as a dependency manager for a wide range of projects.

I just want to reiterate that pip does it. It sensibly warns the user that there are version conflicts, but it doesn't prevent them from installing dependencies. There's no reason Poetry should prevent users from doing this either.

It's obviously outside the purview of Poetry to make sure that the dependency requirements of every Python package ever made are defined correctly.

How difficult would it be to implement this as proposed above, so that it works like --allow-prereleases?

[sdispater]

@kylebebak pip does it as a side-effect of not having a proper dependency resolver and, as far as I know, it's not a feature.

I understand the points made here but like I said before this is not a trivial change and would require a lot of work on the dependency resolution front that I am not willing to do. If someone want to step in and implement it in an intuitive manner I will gladly take a look at it.

But this leaves a lot of uncertainties, let's take an example:

• A depends on C >= 1.0.0, <2.0.0
• B depends on C >= 2.0.0, <3.0.0
• C intoduced API-breaking changes in version 2.0.0
• You force C to the range >= 2.0.0, <3.0.0

Now, you might introduce bugs in the behavior of A because of the API change while using a lower version of B might be the solution to the issue. The resolver and its conflict reporting is here to make you think about your dependencies and understand what is going on. Dependency constraints are here for a reason and introducing a way to override them introduces a risk of it being misused instead of finding the proper solution.

That being said, if someone find a good solution to this problem and implement it, I'd gladly review it.

[kylebebak]

@sdispater

When he created this issue, @zehauser mentioned a reason that provides a very good justification for this feature:

But... many packages out there are not specifying their dependencies properly. Even if they are, there's always the possibility that their specified dependencies are a tighter range than they strictly need to be.

Imagine you depend on A and B. Let's say A depends on B >= 1.1.0, <1.3.0, but it turns out A works just fine if you use B v1.9 instead of B v1.1.

Because A has specified an excessively narrow version range for B, you now can't ever upgrade B's version if you use Poetry, even though newer versions of B work just fine with A.

Your only option is to ask A's maintainer to broaden the version range. Maybe he'll do this in a week, maybe he'll do it in a month, and maybe he'll do it never...

[brycedrennan]

This has also prevented me from using poetry for projects.

I think part of the tension here is between the differing needs of applications vs libraries. For libraries, allowing conflicting dependencies would be bad. For applications, its sometimes necessary to use libraries that claim to be conflicting because you have to get stuff done and cant wait on a maintainer to update their library.

Perhaps any solution we come up with should allow overriding conflicts for features used by apps (poetry install) but not for building packages (poetry build)?

[nikordaris]

I think the bottom line is if your application doesn't directly use library C but A and B use conflicting versions then poetry should fail the install. But if through testing you discover that the limited usage of C by A and B turns out to only use portions that were non breaking changes then you should be allowed to set the version of C that makes sense in your application. The expectation of a dep manager isn't an enforcement tool, it's a convenience tool for efficiently installing all my indirect dependencies such that it won't cause conflicts. Poetry does a great job of this. The only thing missing is the ability for the dev to say I know better for this edge case so actually install this version instead, I'll assume all risk in my decision.

With that said, it sounds like this is a hard change so can we start talking about ways we could solve this now instead of whether we should?

[fakyras]

This also would help to resolve issues like #1330 by manually adding functools32 to ignored package list - a little bit hacky, but a valid solution.

I have been using poetry for over a year now, but this situation is very disappointing.
It is practically impossible to use poetry together with tensorflow2 at current state :(

[windviki]

This also would help to resolve issues like #1330 by manually adding functools32 to ignored package list - a little bit hacky, but a valid solution.

I have been using poetry for over a year now, but this situation is very disappointing.
It is practically impossible to use poetry together with tensorflow2 at current state :(

+1
By now tensorflow2 has functools32 dependency issue and it prevents me to use poetry.

[q-wertz]

Stumbled into the problem that poetry also takes sub-dependencies into account that are in a dev group.

Not sure at which places that is useful, but as far as I understand it, dependencies such as mypy do not really matter, right? Because to upstream author maybe just wants to make sure to always test with the same tool but for my package I don't care about that.

In my case nbconvert requires bleach = * which has a package.extras in my poetry.lock file that looks like:

[package.extras]
css = ["tinycss2 (>=1.1.0,<1.2)"]
dev = ["Sphinx (==4.3.2)", "black (==22.3.0)", "build (==0.8.0)", "flake8 (==4.0.1)", "hashin (==0.17.0)", "mypy (==0.961)", "pip-tools (==6.6.2)", "pytest (==7.1.2)", "tox (==3.25.0)", "twine (==4.0.1)", "wheel (==0.37.1)"]

which is derived by poetry from bleach from the following setup.py block:

EXTRAS_REQUIRE = {
    "css": [
        "tinycss2>=1.1.0,<1.2",
    ],
    "dev": [
        "black==22.3.0; implementation_name == 'cpython'",
        "build==0.8.0",
        "flake8==4.0.1",
        "hashin==0.17.0",
        "mypy==0.961; implementation_name=='cpython'",
        "pip-tools==6.6.2",
        "pytest==7.1.2",
        "Sphinx==4.3.2",
        "tox==3.25.0",
        "twine==4.0.1",
        "wheel==0.37.1",
    ],
}

In my dev group I need a mypy>=0.981 which fails to install because

1. The bleach developers have too strict dependencies?
2. The bleach developers should not have a group called dev?
3. poetry should ignore package.extras.dev blocks?

Where do you see the problem? Or am I misunderstanding something?

---

Using the deprecated [tool.poetry.dev-dependencies] block works, but using the new poetry add "mypy>=0.981" which would add it to [tool.poetry.group.dev.dependencies] and thus lead to this kind of naming conflict...

[neersighted]

There's a couple things going on here:

• Poetry solves universally; as things stand, a major feature of Poetry is that all groups/extras must be mutually compatible; to do otherwise requires allowing very surprising behavior or a lot more work solving the entire matrix of possible combinations. See Support for mutually exclusive dependencies across groups (e.g. path dependencies during development only) #1168 and Support for conflicting dep extras, when they are exclusive of each other. #6419 for asks for selective opt-outs.
• Tools like mypy and pytest that directly import/evaluate your source code should be locked in your tree; as they interact at a 'Python level' they really should be resolved. They are intended for the current groups.dev/old dev-dependencies feature.
• Not all tools make sense for this; e.g. pre-commit, black, and other tools which do not actually import your code/get imported by your code. poc: isolated project plugins & tools #5740 is a prototype for how to handle installing/managing these tools (and plugins!) with Poetry without them being in the main dependency tree.
• Poetry should not resolve unused extras unless you ask for them -- I'm not sure if the dev name is triggering some latent collision; I suggest you reach out in Discussions or Discord to determine if you're hitting a bug, or just not using the tooling quite right.
• When inspecting a package, I always suggest reviewing the metadata on PyPI as sometimes authors upload incorrect code and there's a mismatch between PyPI and source control. That hasn't happened here however.

For reference, here is bleach's requirements:

Requires-Dist: six (>=1.9.0)
Requires-Dist: webencodings
Provides-Extra: css
Requires-Dist: tinycss2 (<1.2,>=1.1.0) ; extra == 'css'
Provides-Extra: dev
Requires-Dist: build (==0.8.0) ; extra == 'dev'
Requires-Dist: flake8 (==4.0.1) ; extra == 'dev'
Requires-Dist: hashin (==0.17.0) ; extra == 'dev'
Requires-Dist: pip-tools (==6.6.2) ; extra == 'dev'
Requires-Dist: pytest (==7.1.2) ; extra == 'dev'
Requires-Dist: Sphinx (==4.3.2) ; extra == 'dev'
Requires-Dist: tox (==3.25.0) ; extra == 'dev'
Requires-Dist: twine (==4.0.1) ; extra == 'dev'
Requires-Dist: wheel (==0.37.1) ; extra == 'dev'
Requires-Dist: black (==22.3.0) ; (implementation_name == "cpython") and extra == 'dev'
Requires-Dist: mypy (==0.961) ; (implementation_name == "cpython") and extra == 'dev'

@q-wertz, I suggest that any subsequent discussion occur somewhere else, as you are either encountering usage or documentation issues, or a weird and nasty bug. In any case, this is an illustration of this proposed feature often being a footgun; your instinct to try and solve it with this extremely sharp-edged and brittle (proposed) tool is exactly why no core maintainer has been interested in exploring this feature themselves; it screams persistent headache for end users and Poetry developers alike.

[q-wertz]

Thank you for the extensive explanation and help.

You are probably right with your assumption there might be an issue on my side. I now refactored the pyproject.toml file, removed the mix between old and new syntax and it seems to work now. Not exactly sure where exactly the issue was…

Gave me (minor) headaches for a few days 😅

[neersighted]

Sorry for the noise; I am updating the issue labels/status to be more accurate.

[earonesty]

if anyone gets here, here's how you solve it.... fork the offending 10 levels upstream repo, relax constraints as needed, then use the forked repo instead, submitting a PR if appropriate.

[pierresouchay]

@earonesty Yes, making poetry not suitable for large dependencies and long term projects

[tombohub]

I just discovered that if you have sub package in your project with another pyproject.toml and you run poetry add some-package from there it will happily upgrade or downgrade already installed packages to satisfy some-package, but if you run from project root it will conflict

[frankhli843]

Is there any resolution to this issue? This is causing issues with pydantic

[bcliang]

As a point of reference, pdm now supports both overrides and exclusions in dependency resolution. Really hoping for poetry to introduce something similar.
https://pdm-project.org/latest/usage/config/#override-the-resolved-package-versions
https://pdm-project.org/latest/usage/config/#exclude-specific-packages-and-their-dependencies-from-the-lock-file

[bellini666]

Not only pdm, but uv by astral (the creators of ruff) does support dependency override as well:

https://github.com/astral-sh/uv?tab=readme-ov-file#dependency-overrides

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.