Is it safe to pin setuptools/distribute/pip in requirements.txt?

[jcushman]

• Pip version: pip 19.1
• Python version: Python 3.5.3
• Operating system: macOS 10.14.4

As documented in this issue over at pip-tools, if you install packages like pytest or Markdown with --require-hashes you have to pin a specific version of setuptools as well, because it's included in their dependencies:

$ cat requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
markdown==3.1 \
    --hash=sha256:fc4a6f69a656b8d858d7503bda633f4dd63c2d70cf80abdc6eafa64c4ae8c250 \
    --hash=sha256:fe463ff51e679377e3624984c829022e2cfb3be5518726b06f608a07a3aad680
$ pip install -r requirements.txt
Collecting markdown==3.1 (from -r requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/f5/e4/d8c18f2555add57ff21bf25af36d827145896a07607486cc79a2aea641af/Markdown-3.1-py2.py3-none-any.whl
Collecting setuptools>=36 (from markdown==3.1->-r requirements.txt (line 7))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools>=36 from https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl#sha256=c7769ce668c7a333d84e17fe8b524b1c45e7ee9f7908ad0a73e1eda7e6a5aebf (from markdown==3.1->-r requirements.txt (line 7))

The pip-tools project does not pin setuptools without the scary-sounding --allow-unsafe flag, because "it may cause conflicts with pip itself". But the maintainers don't currently have a failing test case or know what the conflict would be. It also appears that setuptools was vendored by pip at some point, which might have changed the equation.

Can anyone on the pip side clarify under what circumstances it might be safe or unsafe to pin setuptools, and thus what the correct behavior would be for pip-compile --generate-hashes in the linked bug? To promote the use of --require-hashes, it would be great to either narrow pip-compile's definition of "unsafe", or provide clearer instructions about how to handle the unsafety when needed.

The other two packages on pip-compile's "unsafe" list are distribute and pip itself, so I guess the same question applies to those as well -- are there any risks as far as pip is concerned to pinning them if necessary to satisfy --require-hashes?

Thanks!

[uranusjr]

The pinning issue aside, why is a Markdown module depending on setuptools in the first place? :(

[jcushman]

Python-Markdown requires setuptools>=36 because it wants to set markdown.__version__ = pkg_resources.extern.packaging.version.Version(...).

pytest requires setuptools>=40.0 because it wants to use py_modules in setup.cfg.

I don't know that either of those reasons was necessary, but are they a problem? It seems better to tell package authors "go ahead and depend on features of a minimum setuptools if you want to," rather than "be sure to support all versions of setuptools in use" -- otherwise new features of setuptools will take many years to be usable.

But if package authors are allowed to set minimum setuptools versions, then it also seems better (safer?) to pin that version in a requirements.txt file rather than letting each deployment end up with a different version.

[uranusjr]

I’d say there are better ways to do what the packages want to do. Maybe conceptually there is a case pip should handle itself (and setuptools) more gracefully in hash mode, or maybe pip-tools should allow pip/setuptools to be locked, that’s up for discussion. These particular usages do no make a strong case, however, IMHO.

[jcushman]

The pytest usecase is pretty reasonable, isn't it? setuptools introduced an updated config file format and they want to use it -- that seems OK.

I poked around a bit more via a Google search on github. Some other popular packages setting minimum setuptools versions: kubernetes-client requires setuptools>=21.0.0; ipython requires setuptools>=18.5

I'm not sure it really matters how good the reasons are, does it? It's valid syntax; it's harmless, as far as anyone knows; it presumably solved a problem when introduced; and it's used in popular packages. Trying to convince package authors to never use it seems like a harder approach than tweaking pip-tools to handle it -- unless for some reason that wouldn't work.

[uranusjr]

I believe setup_requires or the PEP 518 build requirement is better for describing their intention 🙂

[uranusjr]

But regardless of whether Pytest can do better, I believe setuptools should be able to be pinned, since it does have a Python API (pkg_resources) that is useful for certain kind of projects. Whether pip-tools should allow pinning it, or pip should allow skipping it in hash mode, that needs to be discussed.