Adding Python-esque test cases

[pradyunsg]

One of the things that can be found in zazo's issue tracker, is a bunch of Python-specific failure cases for the current pip resolver.

It might be useful to add test cases for those situations. Another case that would be really handy would be to have test cases for some successful situations, like installing https://libraries.io/github/aws/chalice -- which depends on 35+ packages directly, including botocore, which has 900+ releases.

[pradyunsg]

I suggest creating a script that fetches metadata from PyPI's JSON API for a package for all versions of that package, and dumps those details into a file (skipping packages without dependency metadata from the API). Then running this script for with all packages that get installed when you do pip install chalice will give you all the information needed to construct the test case.

[uranusjr]

I suggest creating a script that fetches metadata from PyPI's JSON API for a package for all versions of that package, and dumps those details into a file

Many of Monilino’s test cases are built this way (from RubyGems), and I’m 100% on board. It wouldn’t be a terrible idea to read from wheels even IMO since the spec generation is a one-off event.

[pradyunsg]

pypa/pip#7096

^ another simple-but-interesting case. :)

[pradyunsg]

/cc @pfmoore in case he's interested in taking a look at this.

[uranusjr]

Note: There are some cases raised in pypa/pip#7406 as well.

[pradyunsg]

You can probably directly use https://github.com/chrahunt/packages. :)

[uranusjr]

One more pypa/pip#7714

[uranusjr]

(BTW I started implementing this in the pypi-tests branch.)

[pradyunsg]

File a draft PR for the WIP branch?

[pradyunsg]

One place worth "mining" is... pypa/pip#988. A lot of issues related to dependency resolution on pip's issue tracker were closed by cross-linking to that issue -- if you're looking for additional test cases, that'd be a good place to dig up for them.

[uranusjr]

List of cases to pull (growing):

• chalice
• virtualenv>=20 + pre-commit (Adding Python-esque test cases #27 (comment))
• Two entries for one package, on with extra and one without (Fail to resolve dependency when install_requires and extras_require have same package with different extras. pypa/pip#7714)
• pip needs a dependency resolver pypa/pip#988
  • TODO: Read the thread
• pyrax==1.9.8 (New Resolver: technical choices pypa/pip#7406 (comment))
• Two extras required separately for one package (Unable to install two extras in a project making use of different extras of a second library pypa/pip#7096)

[uranusjr]

To reproduce the virtualenv + pre-commit error:

$ pip install "virtualenv<20" "six<1.12.0" "tox==3.14.2"
...

$ pip install -U "tox<3.14.4" virtualenv
...
ERROR: virtualenv 20.0.3 has requirement six<2,>=1.12.0, but you'll have six 1.11.0 which is incompatible.
...

---

Quoting @pfmoore from Zulip:

If a user installs something like tox, which depends directly on six, as well as indirectly via virtualenv, and the user has an older version of six installed, the current resolver sees the first dependency, notes that it's satisfies, and ignores the subsequent dependency. Result - virtualenv breaks because it wants to call a version of six that isn't installed.

Quoting myself:

The current resolver would “successfully” resolve if you swap the packages to update (pip install -U virtualenv "tox<3.14.4") because “first found wins”.

[uranusjr]

Two improvements (bugfixes?) are needed:

• Can’t find any links for PyYAML, but there are clearly compatible wheels. Need to investigate
• Need a way to provide dependencies for sdist-only projects. Two are known: bokeh and toolz.

[pradyunsg]

Another case: python-poetry/poetry#2094

• allennlp == 0.9.0

[uranusjr]

We’d probably want to add a timeout to the new resolver for the initial rollout, to avoid users waiting indefinitely. (If the resolver takes too long, give up and tell them to report the issue.)

[pradyunsg]

I think we should keep track of number of backtracks done, which would allow us to be insensitive to network latency, and the "oh, it's taking too long" would be dependent on the graph, rather than other system/external characteristics.

[uranusjr]

There’s currently not a hook to report how many backtrack steps were taken :p I think it would be acceptable to use “rounds” instead though, which counts forward steps.

[pradyunsg]

Pinging @sdispater for inputs here!

@sdispater Do you have any suggestions for here, other than pyrax and oslo.utils?

I see user reports for things like allennlp and newspaper3k on poetry's issue tracker and I'm wondering if you remember edge cases that you had to fix in poetry's resolver as part of user reports that aren't covered by the aforementioned cases.

[uranusjr]

I’m closing this since the repo has acquired several tests from pip users now. We can always add more when they come up.