Add a machine specific wheel tag as highest precedence wheel tag

[pradyunsg]

What's the problem this feature will solve?
A machine-specific tag lets you compile a wheel on your local machine, for your local machine, and have it get picked above the manylinux1 wheel from pypi without disabling manylinux1 completely.

Additional context
#3921 has all the past conversation related to this. #2875 is sorta related.

[dholth]

I designed the PEP 425 tag system so that linux_x86_64 is the machine specific tag, and as long as you limit the distribution of those wheels this works.

Please do fix this somehow. We need to be able to override specific broken manylinux wheels by placing higher precedence wheels in the search path.

[uranusjr]

Is this specific to Linux, or should other systems also get similar tags? I know Linux is the place where this is needed, but it seems somewhat overly specific if linux_x86_64 is used for the abstract idea of machine-specific, or inconsistent if only Linux can have machine-specific wheels.

[dholth]

Interesting never thought of it that way before. Windows wheels have no compatibility guarantee either and a Linux wheels might be widely compatibile whatever the tag; it's a matter of probability.

…

On Fri, May 24, 2019, at 2:23 AM, Tzu-ping Chung wrote: Is this specific to Linux, or should other systems also get similar tags? I know Linux is the place where this is needed, but it seems somewhat overly specific if `linux_x86_64` is used for the abstract idea of machine-specific, or inconsistent if only Linux can have machine-specific wheels. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6523?email_source=notifications&email_token=AABSZEXDCHIJTQ4G344NM3LPW6CVDA5CNFSM4HOZ23E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWEIVYA#issuecomment-495487712>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABSZEVD5SX5ZFUXNSMKDTLPW6CVDANCNFSM4HOZ23EQ>.

[dholth]

It's right there in PEP 425 "Tools should use the most-preferred architecture dependent tag e.g. cp33-cp33m-win32 or the most-preferred pure python tag e.g. py33-none-any by default.", but tags[0] is the manylinux tag. Currently a wheel building tool must skip all those tags to find the correct tag for a new C extension wheel.

Maybe we could get around this problem by replacing the manylinux config file with a Python script that generates or filters the entire list of tags and/or have everyone import packaging.tags. Then BSD users who have also said we generate the wrong tags for them, or cluster users who have a lot of copies of the same environment, could get whatever behavior they need.

[dholth]

If you don't mind this issue should be closed. I'm frustrated that I haven't been allowed to be the authority on my own non-intuitive invention. The proposed solution of a random tag per machine would be less convenient than the status quo.

[cjerdonek]

FWIW, I'm happy to defer to @dholth on this.

Maybe we could get around this problem by replacing the manylinux config file with a Python script that generates or filters the entire list of tags and/or have everyone import packaging.tags.

Can you elaborate on this, by the way? What is the manylinux config file you're referring to? (pardon my ignorance on this)

[dstufft]

It doesn't make sense that linux_x86_64 is the more specific tag, given it will match on literally any Linux machine. It is not more specific, it is broadly more generic, the only thing more generic is any.

[dholth]

It is a semiotic disconnect

…

On Fri, May 31, 2019, at 5:13 PM, Donald Stufft wrote: It doesn't make sense that ` linux_x86_64` is the more specific tag, given it will match on literally any Linux machine. It is not more specific, it is broadly more generic, the only thing *more* generic is `any`. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6523?email_source=notifications&email_token=AABSZEV7ZNFLND73MM5A6O3PYGIFTA5CNFSM4HOZ23E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWWMKAY#issuecomment-497861891>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABSZESQAZYUIGCYIZEM7RDPYGIFTANCNFSM4HOZ23EQ>.

[dholth]

@dstufft I've noticed that I like systems that fail with false positives as "I installed a package but it didn't run" while you tend to build systems that fail with false negatives "I copied the package to another machine but it didn't install at all". I think that is where the communication breaks down unfortunately. I would be open to any static tag that could come with instructions to copy with care because it doesn't guarantee compatibility. For example py3-none-localhost or xunil_x86_64 (because Linux compatibility is so backwards).

[pradyunsg]

@dholth @dstufft I'm okay either way (since this doesn't affect me directly) and I'm happy to defer to you both (in case there's more discussion to be had here) or maintain status quo.

@dholth if you think a "localhost" tag is good enough here, please do reopen this issue since that's what I view the proposed tag as being.

FWIW, I like that name too. :)

[xavfernandez]

👍 for the idea of a localhost (or some other name) tag that would take precedence over all the other machine tags.

[dholth]

If that makes sense to you then it will solve my problem.

Suppose pypi has a broken manylinux wheel for a database driver.

You can compile a wheel for that driver.

driver-1.0-cp27-cp27mu-local.whl

Now place that in ~/var/wheels which happens to be on $PIP_FIND_LINKS and everything works again. If you know that you have many similar machines, you can copy it there without having to configure anything.

If you used a random number instead of a fixed tag, that wouldn't be smart enough to detect when you changed your machine to be incompatible with the shared library the wheel links to; maybe you uninstalled or upgraded the driver. We can't usefully encode the complete set of shared libraries and symbol versions on your system in a tag.

You might want to give your wheel both arch tags so it was easier to keep track of what was going on inside the wheel: driver-1.0-cp27-cp27mu-local.linux_x86_64.whl, but you would still have to make sure the $PIP_FIND_LINKS directory was never shared between dissimilar systems.

[dholth]

’Tis but thy name that is my enemy;
Thou art thyself though, not a Montague.
What’s Montague? it is nor hand, nor foot,
Nor arm, nor face, nor any other part
Belonging to a man. O! be some other name:
What’s in a name? that which we call a rose
By any other name would smell as sweet;
So Romeo would, were he not Romeo call’d,
Retain that dear perfection which he owes
Without that title.

[dholth]

It's inconvenient that wheel tags are not more intuitive. The system was built only to help install the "right" version of a package when there can be multiple builds of a package under consideration, if it also expresses an abstract idea that might be a bonus. The trouble is that these are all separate things:

The one the user wants
The one that was built on a system most-similar-to the installation environment (the heuristic the PEP wants)
The wheel with the highest priority tag in the set of wheels under consideration (the heuristic the PEP uses)
The actual order of the tags
The tags ordered by how many or few systems will accept that particular tag
More?

[pradyunsg]

Corresponding tracker issue for packaging.tags: pypa/packaging#239

[pradyunsg]

Relabeled, since this can now be scoped as an enhancement to an existing PEP (PEP 425).

[pradyunsg]

There's a draft PEP for augumenting the spec: https://discuss.python.org/t/draft-pep-the-local-wheel-tag/3145/.

Since we likely won't need to do anything until packaging.tags gets an update due to this PEP, I'm gonna close this out.