Proposal: cli for adding tags to an existing wheel?

[henryiii]

Something that comes up in several contexts is the ability to add or change tags to wheels. When using python -m build, you don't have access to the --plat-name setting, and it also only supports one setting and other parts of the tag are missing.

Use case 1

When packaging code that does not rely on CPython, like cmake and ninja, the wheels do not depend on the Python version. This can also be the case for packages that build an SO and load it with types; one refactor I'm about to work on is pullout out the common code from awkward to sit in a single package that is depended on by the regular package, dramatically reducing the total size required on PyPI for a release.

For CMake, for example, the tags look like this:

cmake-3.20.2-py2.py3-none-win_amd64.whl 
cmake-3.20.2-py2.py3-none-macosx_10_10_universal2.macosx_10_10_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl
cmake-3.20.2-py2.py3-none-manylinux1_x86_64.whl
cmake-3.20.2-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 

Linux is mostly handled by auditwheel, but auditwheel has to internally do this too, using auditwheel.wheeltools. And in CMake's case, the py2.py3 part still has to be manipulated.

macOS is actually not very special here - all Universal2 wheels should ideally be tagged with x86_64 as well to be loadable in older versions of Pip if one wants to only release a single binary. (AFAIK). This leads into use case 2.

Another case I've seen is with wcmatch, which depends on the Python version, but not the platform.

Use case 2

In cibuildwheel, producing a single universal wheel that actually works on all versions of pip requires a platform tag like macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2. The final pip 20 release wasn't able to read "macosx_11_0_arm64" (as well as the default pip on macOS 11), etc. Being able to change/set the wheel tags would make this much simpler. This is much more important now with macOS on Arm.

This was done incorrectly by simply renaming the wheel for a while during development, just to get the wheel to load for testing before pip updated. The metadata was not being corrected.

Proposal

A new command could be added to wheel, wheel tags <name> --set <new_tag> --add <new_tag> --add <newtag>. You could 0 or 1 --set, which replaces the current tags, and 0 or more --add, which add tags. This would add the tags to the wheel and rename it to match the new tags. An API for it would be nice too, I think.

The implementation could be basically an unpack, change tags, then pack, as in scikit-build/ninja-python-distributions#49?

Alternatives

This could be made part of a different package, but this seems like a general enough problem to be part of wheel, and isn't that different from the existing unpack/pack - the code is already mostly in wheel. Or maybe there's another solution already? At least in the scikit-build projects, there is code doing this being dragged around scikit-build/ninja-python-distributions#49 .

There could also be better support in specifying tags when making the wheel in the first place, but it tends to be rather dynamic, depending on "universal2 on macOS" and things like that. This could be a longer-term project vs. the extra utility?

Maybe there's already something I'm not seeing, etc?

CC @mayeut (currently on vacation). @brettcannon might be interested.

---

Related to #161, #153, #175, #394.

[brettcannon]

What is <new_tag> to be? Based on the example it would have to be the entire tag set, e.g. cmake-3.20.2-py2.py3-none-macosx_10_10_universal2.macosx_10_10_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl because you can't infer what type of tag something is (i.e. interpreter, ABI, or platform). So you would need to have options for all of the types of tags to do this accurately.

And is <name> the file name? Are you expecting this to rename in-place?

[henryiii]

Sorry, yes, I was combining them incorrectly - my focus was a bit more on "is this a good idea and an acceptable thing to work on", vs. the details. New proposal:

The options would be:

usage: wheel tags [-h] [options...] [WHEEL, ...]

Make a new wheel with given tags. Any tags unspecified will remain the
same. Tags that are specified will replace existing tags unless --append
is given. The original file will remain unless --remove is given.

positional arguments:
  WHEEL           Path to an existing wheel. Can pass more than one.

optional arguments:
  -h, --help            show this help message and exit
  --rm                  remove the wheel, performing a rename
  --append              add to existing tags
  --python-tag TAG      specify an interpreter tag. Can be given multiple times.
  --abi-tag TAG         specify a ABI tag. Can be given multiple times.
  --platform-tag TAG    specify a platform tag. Can be given multiple times.

Example usage:

# Given: cmake-3.20.2-py3-none-macosx_10_10_universal2.whl
wheel tags cmake-3.20.2-py3-none-macosx_10_10_universal2.whl \
                   --rm --append \
                   --python-tag py2 \
                   --platform-tag macosx_10_10_universal2 \
                   --platform-tag macosx_10_10_x86_64 \
                   --platform-tag macosx_11_0_arm64

# Makes
cmake-3.20.2-py2.py3-none-macosx_11_0_universal2.macosx_10_10_universal2.macosx_10_10_x86_64.macosx_11_0_arm64.whl
# And removes cmake-3.20.2-py3-none-macosx_10_10_universal2.whl

Defaulting to --rm and removing a way to keep the original file would be fine with me. Also, we could flip the option --append option, making it --reset, that way the "add a tag" usage is more natural.

The output should be the new wheel name(s). That would allow later tools to capture and use the new wheel names.

[brettcannon]

I would call it platform-tag or plat-tag for consistency with the other options.

[henryiii]

Before doing any work on this I'll wait for @mayeut's input, as he's had to work with this quite a bit. Might be up to two weeks.

[mayeut]

Sorry it took so long to get to this.

The overall idea seems like a good addition. The proposed usage of CLI is enough I think.
There are however 2 things I'm thinking about:

• CLI:

1. as mentioned by @henryiii there are use cases where we want to append and ones where we want to replace. Those are not exclusive use cases. If it can be done somehow, choosing replace or append per tag kind would be great. If this seems too convoluted, then it's just a matter of multiple calls to the tool, nothing too complicated.
   e.g.

wheel tags cmake-3.20.2-cp39-cp39-macosx_10_10_universal2.whl \
    --replace --python-tag py2 \
              --python-tag py3 \
              --abi-tag none \
    --append --platform-tag macosx_11_0_universal2 \
             --platform-tag macosx_10_10_x86_64 \
             --platform-tag macosx_11_0_arm64

instead of

wheel tags cmake-3.20.2-cp39-cp39-macosx_10_10_universal2.whl \
    --python-tag py2 \
    --python-tag py3 \
    --abi-tag none
wheel tags cmake-3.20.2-py2.py3-none-macosx_10_10_universal2.whl \
    --append --platform-tag macosx_11_0_universal2 \
             --platform-tag macosx_10_10_x86_64 \
             --platform-tag macosx_11_0_arm64

2. I think the parser shall accept compressed tag sets:
   e.g.

wheel tags cmake-3.20.2-cp39-cp39-macosx_10_10_universal2.whl \
    --replace --python-tag py2.py3 \
              --abi-tag none \
    --append --platform-tag macosx_11_0_universal2.macosx_10_10_x86_64.macosx_11_0_arm64

• API: must have in 1.0 IMHO (I've still not tested the on-going branch in auditwheel...)

[henryiii]

I'll try to work on this in a week or so. Compressed tags should be fine (though I'd think both should be accepted), and I'll consider point 1; if we can do it in a not-too-surprising way, it would be fine.

[brettcannon]

Another thing to consider is a way to add/change the build number (been asked that multiple times on Twitter thanks to https://snarky.ca/what-to-do-when-you-botch-a-release-on-pypi/).

[henryiii]

(That was exactly what was mentioned in pypa/cibuildwheel#791, actually)

I just saw that article on PyCoder Weekly a few seconds ago in my mail (where I saw this), didn't even notice it was yours yet. :)

[henryiii]

Just curious, what about appending tags if they start with .? That is,

wheel tags cmake-3.20.2-cp39-cp39-macosx_10_10_universal2.whl \
    --python-tag py2.py3 \
    --abi-tag none \
    --platform-tag .macosx_11_0_universal2.macosx_10_10_x86_64.macosx_11_0_arm64

Or is that too hard to see / too confusing? Just not liking the --replace/--append flags affecting the flags after them.

[brettcannon]

My brain noticed the leading ..

I guess the question is whether people will most likely be doing substitutions or additions of tags and thus which to optimize for?

[brettcannon]

I will say, though, that the suggestion of a leading . in the new match keyword was a controversial one that ultimately was left out to punt on the discussion.

[mayeut]

While looking at #422, I remembered seeing https://github.com/nightlark/swig-pypi/blob/f7fdab94cf58b1c53670bfee47e14a5a402f3fd4/setup.py#L12-L20 not long ago (and forgot the linked issues #161, #175 before that). I gave it a try, and updated the snippet for the macOS case in ninja:
scikit-build/ninja-python-distributions#85

It seems to work well but how recommendable is this ? Will subclassing bdist_wheel be stable in time ?

[agronholm]

The plan is to migrate bdist_wheel to setuptools itself and make this project a generic library (and CLI) for creating and manipulating wheel files.

[agronholm]

The optimal way to implement this would be to modify the metadata files in the existing wheel file. Have you looked into that yet?

[henryiii]

I thought zip files can only be appended to, not modified? Am I wrong? Would like to be.

[agronholm]

Quote from PEP 427:

Archivers are encouraged to place the .dist-info files physically at the end of the archive. This enables some potentially interesting ZIP tricks including the ability to amend the metadata without rewriting the entire archive.

[henryiii]

I would prefer "required" over "encouraged". :) Okay, I'll try that next. This was also useful to get a feel for the CLI interface, etc.

[henryiii]

I've reimplemented this more directly, though it still doesn't do "tricks" with modifying the zip - that could be a later optimization if required, I'd think?

[JuniorJPDJ]

Would be amazing to have command for deleting, adding and replacing tags!

[agronholm]

Is there currently a use case that would not be covered by setting the platform tag right when creating the original wheel?

[mara004]

Is there currently a use case that would not be covered by setting the platform tag right when creating the original wheel?

@henryiii mentions two use cases in the issue description.

Another point: To my understanding, the only Python APIs for setting wheel tags reside in this package and are considered non-public. So one advantage in providing a CLI for wheel tags would be to finally provide an official and easy to use interface for projects that need to set wheel tags directly and do not use a PyPA package like cibuildwheel to do it.

[agronholm]

From the description, it seems like use case 2 might not be relevant anymore, as pip has been since been fixed to deal with the tags correctly. Use case 1 would be better served by either specifying the platform tag during creation of the initial wheel, or by cibuildwheel using the upcoming public API of wheel.

[JuniorJPDJ]

Is there currently a use case that would not be covered by setting the platform tag right when creating the original wheel?

Manipulation of already created wheels. Eg. when you want to re-tag wheels for internal use-case or fix something with workaround something that 3-rd party packager broke. ATM I'm doing sed over WHEEL file in dist-info, which is far from ideal.

[agronholm]

Is there currently a use case that would not be covered by setting the platform tag right when creating the original wheel?

Manipulation of already created wheels. Eg. when you want to re-tag wheels for internal use-case or fix something with workaround something that 3-rd party packager broke. ATM I'm doing sed over WHEEL file in dist-info, which is far from ideal.

Ok, so can you describe your use case in more detail?

[JuniorJPDJ]

I'm fixing tags created by maturin due to PyO3/maturin#1289.

This is workaround, but I need to have it working in meanwhile when issue is being fixed.

There is my use-case scripted: https://dev.funkwhale.audio/funkwhale/funkwhale/-/blob/7e0aabe62c6efd7687fd9f74819e803adc0f8aeb/.gitlab-ci.yml#L84

[agronholm]

Ok, sold. I'll give the linked PR a look tomorrow.