Python tag should reflect python_requires

[jaraco]

In jaraco/zipp#37 and others, I've learned that universal wheels are essentially meaningless as they declare "work on all Python 2 and Python 3 versions" when in fact they have minimum Python 2 and Python 3 versions.

The baseline packaging default for generating wheels for pure-python distributions was bad enough when it required specifying universal=1 for wheels to get py2/3 compatibility, but now I learn that behavior is unsatisfactory for projects targeting Python 3 and now users are demanding that the python tag be updated to declare the minimum supported Python version(s).

The Requires-Python (python_requires distutils option) was created to declare these requirements more directly, but this functionality didn't supersede the PEP 425 expectations that the Python tag should indicate the minimum supported Python version (or versions) that the build supports.

It is recommended that installers try to choose the most feature complete built distribution available (the one most specific to the installation environment) by default before falling back to pure Python versions published for older Python releases.

What this means ultimately is that package maintainers are required to declare their supported Python versions in at least two, maybe three places:

• In the Requires-Python spec
• In the python-tag for wheels
• In Trove classifiers

I'm excluding Trove classifiers from this discussion.

I propose that bdist_wheel, instead of defaulting to the py3 or py2 should default to a value matching the Requires-Python directive, and determine the minimum python 2 and python 3 versions relevant to the project and use that. So if python_requires = '>=3.6', it should use py36 for the tag. And if python_requires = '>=2.7.13,!=3.0.*,!=3.1.*', the python tag should be py27.py32.

This scheme would allow the packager to declare the supported versions in one place and derive the (default) python tags for the build.

[agronholm]

I've been wondering: how can wheel conclusively determine from python_requires which tags can be added?

[jaraco]

One way could be to construct a list of tags and their relevant supported versions:

dict(
  py2='2.0.0',
  py3='3.0.0',
  py27='2.7.0,
)

Then for each tag, if its associated version satisfies the python_requires, include that tag.

[agronholm]

I thought of that too, but given your example of python_requires = '>=2.7.13,!=3.0.*,!=3.1.*', that would not include py27 since the test '2.7.0' >= '2.7.13' would fail.

[henryiii]

I've used patch=99 for this, that is,

dict(
  py27='2.7.99',
)

python_requires really shouldn't be used with patch versions IMO, but if it is, it's often something like >=3.6.1, and the largest possible patch version is about 18, so 99 is safe.

I would really like to see at least a warning for projects that set universal=True but have a >=3.x Python requirement, I'll open an issue for that.

[agronholm]

Is there a function in packaging that I could to parse python_requires?

[henryiii]

Sure, you get a SpecifierSet out of it. https://github.com/joerick/cibuildwheel/blob/d223db13d833ec81f6303b042029a46a2462b916/cibuildwheel/__main__.py#L173

Then you check if a version is "contained": https://github.com/joerick/cibuildwheel/blob/d223db13d833ec81f6303b042029a46a2462b916/cibuildwheel/util.py#L76-L77

[henryiii]

PS. Since you are making a wheel, I assume you'd parse Requires-Python, the metadata slot, not the input python_requires in setup.py/setup.cfg, or requires-python in pyproject.toml; for cibuildwheel, I didn't have that luxury, because it's using it to decide what Python's to build for. (Edit: yes, it's noted at the top).

[agronholm]

Perhaps an option would be to ignore universal=True if python_requires conflicts, and print a warning?

[henryiii]

How about this concrete proposal (cross posted from pypa/twine#739 since it involves wheel more than twine):

• No Requires-Python, no [bdist_wheel] universal=1: produce py3 wheels, as now.
• No Requires-Python, has [bdist_wheel] universal=1: produce py2.py3 wheels, as now. No warning, fully allowed for now.
• Requires-Python, no [bdist_wheel] universal=1: Use the smallest allowed tag for each valid Python major version.
  • >=2.7 would produce py27.py3.
  • >=2.6, !=3.0.*, !=3.1.* would produce py26.py32.
  • >=3.6 would produce py36.
  • When Python 4 comes out, this starts including Python 4 if allowed by Requires-Python, just like 2/3.
• Python-Requires, has [bdist_wheel] universal=1: produce a warning from wheel, maybe eventually an error (after Python 2 is dropped from wheel). This is logically over constrained - it shouldn't universally work on all Pythons and be limited to some Pythons. When producing a warning, it still produces py2.py3 wheels for backward compatibility.

[pfmoore]

Just as a minor note, PEP 425 does not say anywhere that Python 3.7 must support the py36 compatibility tag. According to the PEP, that's 100% an installer decision.

The reality is, of course, that the sys_tags implementation in packaging.tags does do this, so for all practical purposes it should be OK. But from a "behaviour should be backed by standards" perspective, it's not valid to presume that >=3.6 implies that a py36 tag is correct. Updates to the standards to make such an inference valid are, of course, welcome 😉

[henryiii]

Excellent point, I didn't realize that - in fact, I originally didn't know that py36 was loadable on py37, as I'm more familiar with built wheels where it's one-version only.

Stage 1: Implement the above proposal, but only with py2 / py3 in step 3. >=2.6, !=3.0.*, !=3.1.* would produce py2.py3, etc.
Stage 2: Update the standard to add "pyXY" is valid on any future version of Python with the same major version. Also update the standard to add a "py" tag for later "universal" use; when 4 comes around (in no less than three years, I expect, as at least 3.12 is planned, and maybe in many more), there will be a nice universal tag for that.
Stage 3: Update to the original version of the proposal above, where you can read the minimum versions of Python from the filenames. (Optional, really, but was the main point of this issue originally)

[agronholm]

Closing as not relevant to this project anymore as bdist_wheel is now canonically part of setuptools.