packaging.tags does not support `-none-any` wheels for non`-py` interpreter tags #311

pganssle · 2020-05-28T14:17:54Z

Environment

pip version: 20.1.1 (19.3.1 is the last working version)
Python version: pypy3.6.9-7.3.1
OS: Arch Linux

Description
It appears that pip does not consider pp3-none-any a valid wheel tag on PyPy 3 after pip==19.3.1. This is a problem for the PEP 615 backport, since it has both a C and a pure Python implementation, and I want to ship a wheel containing only the pure Python implementation only for PyPy (anyone else who doesn't have a wheel should fall back to the sdist).

Here is a minimal reproducing bash script, which downloads an arbitrary universal wheel and renames it to end with pp3-none-any.whl:

cd $(mktemp -d)
pypy3 -m venv venv
source venv/bin/activate
pip install -U pip

pip --version
# pip 20.1.1 from /tmp/tmp.IV5fhdUrQt/venv/site-packages/pip (python 3.6)

python --version
# Python 3.6.9 (2ad108f17bdb, Apr 07 2020, 02:59:05)
# [PyPy 7.3.1 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]

pip wheel -w wheelhouse tzdata==2020.1
mv wheelhouse/tzdata-2020.1-py2.py3-none-any.whl\
   wheelhouse/tzdata-2020.1-pp3-none-any.whl
pip install -t target --no-index -f wheelhouse tzdata
# Looking in links: wheelhouse
# ERROR: Could not find a version that satisfies the requirement tzdata (from versions: none)
# ERROR: No matching distribution found for tzdata

Expected behavior
The expected behavior is to install the pp3 wheel when on PyPy, which was the behavior in 19.3.1:

$ pip --version
pip 19.3.1 from /tmp/tmp.IV5fhdUrQt/venv/site-packages/pip (python 3.6)
$ pip install -t target --no-index -f wheelhouse tzdata
Looking in links: wheelhouse
Processing ./wheelhouse/tzdata-2020.1-pp3-none-any.whl
Installing collected packages: tzdata
Successfully installed tzdata-2020.1

It's possible that this is another manifestation of pypa/pip#7629.

The text was updated successfully, but these errors were encountered:

This is a tricky situation, because the PyPy wheel is special — it does not contain the C extension. We do not want this being picked up by anything except pypy. Unfortunately, after pip version 19.3.1 (and until at least version 20.1.1), pip will not pick up the `pp3-none-any.whl` tag (see https://github.com/pypa/pip/issues/8347 ), so pypy users with a recent version of pip will get the sdist (which is our only alternative really anyway). We'll ship these wheels anyway, since this seems to be an issue with pip, and this is indeed the proper tag for PyPy 3 wheels.

uranusjr · 2020-05-28T14:52:20Z

This seems to be an issue with packaging.tags:

>>>> from packaging import tags
>>>> for t in tags.generic_tags(): print(t)
pp36-pypy36_pp73-manylinux2014_x86_64
pp36-pypy36_pp73-manylinux2010_x86_64
pp36-pypy36_pp73-manylinux1_x86_64
pp36-pypy36_pp73-linux_x86_64
pp36-none-manylinux2014_x86_64
pp36-none-manylinux2010_x86_64
pp36-none-manylinux1_x86_64
pp36-none-linux_x86_64
>>>> for t in tags.compatible_tags(): print(t)
[... snip]
py36-none-any
py3-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any

It only lists ppXY, not ppX. The any platform is only available for py-tags, not pp. This is actually the same on CPython (only cpXY are listed, and there are only pyX-none-any, not cpX-none-any). I am not familiar with the tags standard to decide whether this is correct or not, however.

pganssle · 2020-05-28T14:55:33Z

Hm, this was apparently done deliberately, based on the idea that pp3-none-any would be a nonsensical tag, but I'm not sure what I should use in this case. I'd be fine with pp-none-any.whl here as well, since python_requires will prevent it being picked up on PyPy 2.7.

@brettcannon Is this use case, where I have a PyPy-specific build not using any version-specific features of PyPy, something you considered already? Is there a workaround? If not, I think it would be a good idea to add support for at least pp, if not pp3. I'd really rather not have to build separate, identical wheels for every version of PyPy that's released just so I can tag them with pp36, pp37, pp38, etc.

(P.S. We can probably move this issue over to pypa/packaging, since it seems this was a deliberate choice over there.)

pradyunsg · 2020-05-28T16:52:01Z

This is basically pypa/pip#7629?

pradyunsg · 2020-05-28T16:55:06Z

It is, indeed. Closing as a duplicate of pypa/pip#7629. If someone understands this issue well enough, they're welcome to make a pip-specific fix for this change. There's an approach that there's reasonable consensus on, and a PR for that which needs updating -- pypa/pip#7655.

uranusjr · 2020-05-28T16:55:10Z

They are both caused by #184, but I feel each tag should be discussed separately. pp272 (#7629) makes less sense to me than pp2 (this issue).

pradyunsg · 2020-05-28T16:57:09Z

I'd bundle them together TBH -- pypa/pip#7629 is the right place within pip's issue tracker for discussing the PyPy tags w/ packaging.tags situation.

pganssle · 2020-05-28T17:01:29Z

@pradyunsg I'm having a bit of trouble disentangling them, but it seems from the description of pypa/pip#7655 that these are "legacy" wheel tags for specific binaries. I do not think "pip supports them because it used to support them, but you should stop generating them" would solve my use case. I think we should migrate this issue to the packaging repo.

pradyunsg · 2020-05-28T17:08:57Z

Moved as requested, and, well, /cc @mattip @brettcannon for thoughts. :)

brettcannon · 2020-05-28T18:47:30Z

Three things. One, @pganssle why don't you want CPython users to use the pure Python wheel and have to build from source otherwise? If the pure Python wheel is good enough for PyPy why isn't it good enough for e.g. Alpine users? And if they really care about performance couldn't they choose to skip the wheel to build from source?

Two, using cp3 as an example, where do you put it in the tag list (and remember you need to insert it twice; none-<os> and none-any as I'm assuming zero ABI support)? And do note that the ordering is a bit funky when it comes to interpreter version so you will have to be very specific about where in the list below you suggest it goes.

And three, if you add cp3 and pp3, what about cp39-none-<OS> and cp39-none-any? Where do they go?

cp38-cp38-win_amd64
cp38-abi3-win_amd64
cp38-none-win_amd64
cp37-abi3-win_amd64
cp36-abi3-win_amd64
cp35-abi3-win_amd64
cp34-abi3-win_amd64
cp33-abi3-win_amd64
cp32-abi3-win_amd64
py38-none-win_amd64
py3-none-win_amd64
py37-none-win_amd64
py36-none-win_amd64
py35-none-win_amd64
py34-none-win_amd64
py33-none-win_amd64
py32-none-win_amd64
py31-none-win_amd64
py30-none-win_amd64
py38-none-any
py3-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any

I still don't think the ABI-agnostic interpreter versions are worth it, but people keep bringing them up just often enough I'm going to stop arguing against them if we can agree as to where they should land in the order and someone submits a PR for whatever we agree to.

pganssle · 2020-05-28T20:11:49Z

Three things. One, @pganssle why don't you want CPython users to use the pure Python wheel and have to build from source otherwise? If the pure Python wheel is good enough for PyPy why isn't it good enough for e..g Alpine users? And if they really care about performance couldn't they choose to skip the wheel to build from source?

This is because the pure python version isn't just "good enough" for PyPy, it's better, but only on PyPy, because PyPy needs to use the cpyext compatibility layer in order to make use of any C extensions, which is dramatically slower than JIT-compiled PyPy code. See the benchmarks in the PyPy support PR; generally the pure Python version is ~10x faster (here's one example):

Running utcoffset in zone America/New_York
c_zoneinfo: mean: 3.45 µs ± 52.71 ns; min: 3.40 µs (k=5, N=100000)
py_zoneinfo: mean: 336.64 ns ± 11.60 ns; min: 325.69 ns (k=5, N=1000000)
pytz: mean: 213.31 ns ± 5.82 ns; min: 204.82 ns (k=5, N=10000000)

Those numbers are reversed on other platforms, and I would prefer other platforms to build from source. If I want to add the ability to opportunistically fall back to the pure python library on musl, etc, I'll put that in the setup.py.

Two, using cp3 as an example, where do you put it in the tag list (and remember you need to insert it twice; none-<os> and none-any as I'm assuming zero ABI support)? And do note that the ordering is a bit funky when it comes to interpreter version so you will have to be very specific about where in the list below you suggest it goes.

And three, if you add cp3 and pp3, what about cp39-none-<OS> and cp39-none-any? Where do they go?

I am not sufficiently familiar with the issues of ordering (and in my case it doesn't matter, since I won't be shipping any py3 wheels — it would only matter if I needed to ship both a pp3 and a pp38 wheel or something of that nature, in which case it's just important that pp38 come before pp3). Could we not use the example ordering laid out in PEP 425, which does include these abi-agnostic interpreter-specific versions?

cp33-cp33m-linux_x86_64

cp33-abi3-linux_x86_64

cp3-abi3-linux_x86_64

cp33-none-linux_x86_64*

cp3-none-linux_x86_64*

py33-none-linux_x86_64*

py3-none-linux_x86_64*

cp33-none-any

cp3-none-any

py33-none-any

py3-none-any

py32-none-any

py31-none-any

py30-none-any

By that ordering, I think (though I could be wrong) it would go:

cp38-cp38-<os>
cp38-abi3-<os>
cp3-abi3-<os>
cp38-none-<os>
cp3-none-<os>
cp38-none-any
cp3-none-any
cp37-abi3-<os>
…

brettcannon · 2020-05-29T17:15:45Z

One other wrinkle in this (and one of the reasons I dropped support for this tag that I just remembered), is there is no consistent way to get the 3 in pp3. The generic code that calculate the interpreter tag is:

packaging/packaging/tags.py

Lines 776 to 787 in 59e1488

    
           def interpreter_version(**kwargs): 
        
               # type: (bool) -> str 
        
               """ 
        
               Returns the version of the running interpreter. 
        
               """ 
        
               warn = _warn_keyword_parameter("interpreter_version", kwargs) 
        
               version = _get_config_var("py_version_nodot", warn=warn) 
        
               if version: 
        
                   version = str(version) 
        
               else: 
        
                   version = _version_nodot(sys.version_info[:2]) 
        
               return version

You will notice it reads from the py_version_nodot config variable; there isn't an explicit major/minor split here. And with 3.10 coming we know that minor versions can be more than a digit so you can't just slice of the last digit.

So to make this work you will either have to special-case PyPy for interpreter version support (which we just stopped doing at PyPy's request), or someone will have to introduce some other way to consistently get the interpreter version as discrete components (possibly config variables for the major and minor versions separately which may help with #308

brettcannon · 2020-09-09T18:24:49Z

Due to this continuing to come up, if someone wanted to added a cp38-none-any tag back in, I won't object (I just won't do the work either 😄 ). Pragmatically, the issue of having an odd tag triple totally outweighs the number of times I have explained why I left it out. 😉

pradyunsg closed this as completed May 28, 2020

pradyunsg reopened this May 28, 2020

pradyunsg transferred this issue from pypa/pip May 28, 2020

pganssle changed the title ~~pip does not pick up pp3-none-any wheels on pypy~~ packaging.tags does not support pp3-none-any wheels May 28, 2020

pganssle mentioned this issue May 29, 2020

Build wheels for PyPy pganssle/zoneinfo#75

Closed

brettcannon added enhancement packaging.tags labels Sep 8, 2020

brettcannon mentioned this issue Sep 8, 2020

It's not obvious how to construct a set of compatible tags for a system #333

Closed

mattip mentioned this issue Nov 1, 2020

Better support for PyPy nedbat/coveragepy#1049

Closed

brettcannon changed the title ~~packaging.tags does not support pp3-none-any wheels~~ packaging.tags does not support *-none-any wheels for non-py* interpreter tags Mar 29, 2021

McSinyx mentioned this issue Sep 19, 2021

Expose various function in tags publicly #231

Merged

mattip mentioned this issue Oct 14, 2021

add a pp3-none-any tag #466

Merged

mmckerns mentioned this issue Oct 14, 2021

installs wrong multiprocess wheel for pypy uqfoundation/pathos#222

Closed

brettcannon closed this as completed in #466 Oct 29, 2021

This was referenced Feb 24, 2022

packaging.tags.compatible_tags() includes cpxx-none-any tags if interpreter is provided #511

Closed

Pip allows cpxx-none-any tags (and maybe it shouldn't) pypa/pip#10923

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

packaging.tags does not support `-none-any` wheels for non`-py` interpreter tags #311

packaging.tags does not support `-none-any` wheels for non`-py` interpreter tags #311

pganssle commented May 28, 2020

uranusjr commented May 28, 2020 •

edited

Loading

pganssle commented May 28, 2020

pradyunsg commented May 28, 2020

pradyunsg commented May 28, 2020

uranusjr commented May 28, 2020

pradyunsg commented May 28, 2020 •

edited

Loading

pganssle commented May 28, 2020

pradyunsg commented May 28, 2020

brettcannon commented May 28, 2020 •

edited

Loading

pganssle commented May 28, 2020 •

edited

Loading

brettcannon commented May 29, 2020

brettcannon commented Sep 9, 2020

packaging.tags does not support *-none-any wheels for non-py* interpreter tags #311

packaging.tags does not support *-none-any wheels for non-py* interpreter tags #311

Comments

pganssle commented May 28, 2020

uranusjr commented May 28, 2020 • edited Loading

pganssle commented May 28, 2020

pradyunsg commented May 28, 2020

pradyunsg commented May 28, 2020

uranusjr commented May 28, 2020

pradyunsg commented May 28, 2020 • edited Loading

pganssle commented May 28, 2020

pradyunsg commented May 28, 2020

brettcannon commented May 28, 2020 • edited Loading

pganssle commented May 28, 2020 • edited Loading

brettcannon commented May 29, 2020

brettcannon commented Sep 9, 2020

packaging.tags does not support `-none-any` wheels for non`-py` interpreter tags #311

packaging.tags does not support `-none-any` wheels for non`-py` interpreter tags #311

uranusjr commented May 28, 2020 •

edited

Loading

pradyunsg commented May 28, 2020 •

edited

Loading

brettcannon commented May 28, 2020 •

edited

Loading

pganssle commented May 28, 2020 •

edited

Loading