resolver dependency contamination

[gsemet]

Hello

I see a weird behavior on the version resolution on pipenv update, for dependencies where the latest version is to be avoided by another dependency.

Given a Pipfile with no special restriction:

[dev-packages]
"flake8" = "*"
pycodestyle = "*"
pyflakes = "*"
pylint = "*"
yapf = "*"
cryptography = "*"
isort = "*"
PyYAML = "*"
pytest = "*"
tox = "*"
"flake8-comprehensions" = "*"
asynctest = "*"
rope = "*"
pytest-sugar = "*"
pytest-mock = "*"
pytest-watch = "*"
mock = "*"
"autopep8" = "*"
pytest-cov = "*"
bandit = "*"

[packages]

For example, pycodestyle latest version is 2.4.0. Some dependencies such as flake8 adds the restriction "<2.4.0". Another one ask for "pycodestyle<1.7.0". But the resolver fails at the end with the message

Could not find a version that matches pyflakes<1.7.0,==2.0.0,>=1.5.0

While reasonably the "==2.0.0" have nothing to do here since no one ever restrict or freeze to this version. This looks like the latest found version on pypi in the previous resolver stage is taken are a hard restriction (with ==), so any later resolution stager should follow it, or fail.

This appears to me as a bug since later stages can only add new dependencies, not restrict the version of a package already resolved.

To me, each resolver stage should keep the resolution context (for pycodestyle, keep the "require=*" information instead of "require==2.4.0").

We can see this behavior in the logs:

$ pipenv update --clear --verbose
...
pycodestyle (from -r /tmp/pipenv-bvm1pyac-requirements/pipenv-jtt7r2p0-constraints.txt (line 4))
...

Finding the best candidates
...
  found candidate pycodestyle==2.4.0 (constraint was <any>)
...

Finding secondary dependencies:
..
  flake8-comprehensions==1.4.1 requires flake8!=3.2.0, flake8-comprehensions==1.4.1, mccabe<0.7.0,>=0.6.0, pycodestyle<2.4.0,>=2.0.0, pyflakes<1.7.0,>=1.5.0
...
pycodestyle==2.4.0        requires pycodestyle==2.4.0
  ^=== This is were the issue happen, the ==2.4.0 should not be, it should be something like pycodestyle==2.4.0 (restriction="*"), to keep the information
...
  flake8==3.5.0             requires flake8==3.5.0, mccabe<0.7.0,>=0.6.0, pycodestyle<2.4.0,>=2.0.0, pyflakes<1.7.0,>=1.5.0
...

New dependencies found in this round:
...
  adding ['pycodestyle', '<2.4.0,==2.4.0,>=2.0.0,>=2.3', '[]']

From now on, the resolution couldn't succeed since "<2.4.0" and =="2.4.0" are obviously incompatible.

Workaround: recopy the restriction "<2.4.0" in my Pipfile. From now, the resolver will take the latest version that respect <2.4.0 and everybody will be happy later on.

Happens with pycodestyle and pyflakes (restricted to "<1.7.0" by flake8 3.5.0)

[frostming]

Don't repeat dependencies in your Pipfile, pycodestyle and pyflakes are dependencies of flake8, please remove them. A good practice is to list top level dependencies only.

This will solve most issues since Pipenv's resolver is not smart enough to handle cases like this.

[gsemet]

I understand, but this is just an example on how to easily reproduce it. The first round of dependencies résolution kind of freeze the version of what has been found while it should not.

[gsemet]

great news! looking forward to testing that ! thanks for the amazing work on pipenv :)

[gsemet]

Saldy, I still have this issue.

[idchlife]

Suddenly project started showing failed dependency of

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies. First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again. Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation. Hint: try $ pipenv lock --pre if it is a pre-release dependency. Could not find a version that matches pycodestyle<2.4.0,>=2.0.0,>=2.4.0 Tried: 2.0.0, 2.0.0, 2.1.0, 2.1.0, 2.2.0, 2.2.0, 2.3.0, 2.3.0, 2.3.1, 2.3.1, 2.4.0, 2.4.0 Skipped pre-versions: 1.8.0.dev0, 1.8.0.dev0, 2.0.0a1, 2.0.0a1 There are incompatible versions in the resolved dependencies.

But I don't have this dependency inside Pipfile D:

Project was working month ago, not suddenly stopped. Could it be my upgrade to Python 3.7?

[gsemet]

not, it is still happening. What I called "contamination" still happens, because autopep8 depends on a version of flake8 that requires pycodestyle<2.4.0. Your solution is to update flake8, or prevent update of autopep8 with autopep8<1.4.1.

The resolver algorithm is pretty hard to debug, probably needs to be completely rewritten. I actually do not understand why they do not delegate completely this to pip which is able to find the right version if you pip install all direct dependencies on a single line.

[techalchemy]

Because pip isn’t a resolver and doesn’t have one to resolve things. That’s why we don’t delegate it. If you want pip to instal whatever it feels like you can use pip. Pipenv is backed by dependency resolution which doesn’t just do randomly ordered installation of whatever it encounters.

We did rewrite the entire thing from scratch btw, https://github.com/sarugaku/passa — feel free to try it out

[gsemet]

Good to know! I hope it will fix this issue. Because autopep8 releases a new version 1.4.1 which may introduce an incompatibility in resolution, the resolver should be able to find it needs to use 1.4.0, even if it involves several round and retries. I understand this complicates a lot, but would greatly improve the adoption of pipenv for python app.

[idchlife]

@gsemet yes, exactly! Learned about dependency which breaks the installation by removing-returning parts of Pipfile and locating dependency.

It will help much if pipenv would print like parent dependency, so developer can learn right now which one should be updated or with frozen version.

Anyway, except this pipenv worked flawlessly. Hope my way of resolving issue would help newcomers (as myself) into pipenv.