pip-compile skips transitive dependencies on python 3.7

[tuukkamustonen]

pip-compile skips transitive dependency on python 3.7.

Environment Versions

1. OS Type: Linux
2. Python version: 3.7.4
3. pip version: 19.2.3
4. pip-tools version: 4.1.0

Steps to replicate

$ echo "pydocstyle" > requirements.in

Works on python 3.6 (and 3.5):

$ pyenv install 3.6.6 py36
$ pyenv activate py36
$ pip install -U pip pip-tools
$ pip-compile --no-header --no-index
pydocstyle==4.0.1
snowballstemmer==2.0.0    # via pydocstyle

(snowballstemmer is the transitive dependency we are after here. Read on.)

Fails on python 3.7:

$ pyenv install 3.7.4 py37
$ pyenv activate py37
$ pip install -U pip pip-tools
$ pip-compile --no-header --no-index
pydocstyle==4.0.1
$ pip install pydocstyle  # the missing requirement is actually needed on py3.7, too
...
Installing collected packages: snowballstemmer, pydocstyle
Successfully installed pydocstyle-4.0.1 snowballstemmer-2.0.0

Interestingly, pydocstyle < 4 installs just fine! Still within python 3.7 env:

$ echo "pydocstyle<4" > requirements.in
$ pip-compile --no-header --no-index
pydocstyle==3.0.0
six==1.12.0               # via pydocstyle
snowballstemmer==2.0.0    # via pydocstyle

Took a look at pydocstyle changes into 4.x, but they have only added python_requires='>=3.5' to setup.py as far as I can tell.

Also reproduces with pylint (where it skips even more dependencies).

[tuukkamustonen]

Uh, I think --rebuild fixed this issue (didn't remember #835 right away).

Annoying, next time this happens, what information (and how) can I provide, to get to the root of it?

[atugushev]

Hello @tuukkamustonen,

Aw, you've caught it again. Next time this happens run pip-compile with --verbose flag and preserve the output. Also, having a dependency cache file attached to the issue would help to investigate the problem. The cache file you may find here:

• Unix: ~/.cache/pip-tools
• Mac OS: ~/Library/Caches/pip-tools/
• Windows: <CSIDL_LOCAL_APPDATA>\pip\Cache

[tuukkamustonen]

Thanks! I'll try to remember this and embed full details :| Closing this now.

[atugushev]

@tuukkamustonen

Just wondering, have you used constraints files with pip-tools (e.g., -c constraints.txt)? This might cause a bug with the cache. See #1037 where we've caught it.

[tuukkamustonen]

@atugushev Appreciate for checking back on this 🙏. Yeah, I'm using pip constraints these days, but I'm not sure if I had those back when I encountered the problems.

I'll check the linked ticket! (And thanks for the awesome tool btw.)