Add a "local"/"localhost" tag that's machine-specific

[pradyunsg]

Context:

• manylinux tags should come after linux tags #160
• Add a machine specific wheel tag as highest precedence wheel tag pip#6523
• add 'local' as highest precedence architecture for wheel pip#6565 (comment)

There is now agreement that we'd want to add a local or localhost platform compatibility tag, which has a higher precedence than all other platform specific tags.

Open questions:

• Does this need amending our existing specification (PEP 425?)?
  • If yes, we should write a new PEP, we should follow https://www.pypa.io/en/latest/specifications/#handling-fixes-and-other-minor-updates.
• Do we want to add the local tag unconditionally (not just on linux)?
  • I think so, yea. I don't see any good reason not to.

[brettcannon]

I agree we should just add the tag unconditionally.

[pradyunsg]

Pinging @pfmoore and @ncoghlan to ask if they have any inputs on the question:

• Does this need amending our existing specification (PEP 425?)?
  • If yes, we should write a new PEP, we should follow https://pypa.io/en/latest/specifications/#handling-fixes-and-other-minor-updates.

[pfmoore]

I think that the PyPA Spec should be updated, probably with a new section "Generic platform tags" that would describe this tag. And yes, that does mean that I think the new tag should be added unconditionally, not just for Linux.

According to the standards process, this would require a new PEP, but I'd hope it would be pretty small and focused. I'm happy to be the sponsor for such a PEP.

[ncoghlan]

Suggested rationale for making the tag "localhost":

• PyPI will always disallow uploads with this tag
• major feature improvement would be to override incorrect or less optimal remote wheels with locally built ones
• most common expected use case would be to tag the auto-generated wheels in the local wheel cache
• when using a private index server, it's better to use the regular tags and upload different wheel archives rather than relying on the localhost override tag

[pfmoore]

@ncoghlan can you clarify? Surely PyPI will always disallow uploads with whatever tag we decide on, and the remaining 3 points don't seem in any way linked to the choice of tag name (to the extent that I understand the points).

Personally, I prefer "local" as it shorter and doesn't have the "network name" implication that "localhost" does (this is a platform tag, not a machine name, although that's a subtle distinction even in my head 🙂)

[ncoghlan]

There's nothing intrinsically "local" about wheels tagged this way, so the name could just as well be "override" to reflect the use case where you're overriding an incorrect or less optimal public wheel.

My preference for "localhost" comes from the fact that if you're dealing with more than one machine, a private index server with proper tag names is likely to be substantially less hassle overall than relying on the override tag. That's only a mild preference, though - I think "local" would also be fine.

[dstufft]

What about custom?

…

Sent from my iPhone

On Jan 5, 2020, at 4:12 PM, Nick Coghlan ***@***.***> wrote:  There's nothing intrinsically "local" about wheels tagged this way, so the name could just as well be "override" to reflect the use case where you're overriding an incorrect or less optimal public wheel. My preference for "localhost" comes from the fact that if you're dealing with more than one machine, a private index server with proper tag names is likely to be substantially less hassle overall than relying on the override tag. That's only a mild preference, though - I think "local" would also be fine. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[ncoghlan]

My only objection to "custom" is that it feels weird for the implicit caching use case. OTOH, it reads better for the explicit override use case.

Perhaps there is a case to be made for two tags? ("local" for the implicit cache, "custom" for explicit overrides)

[pfmoore]

Perhaps there is a case to be made for two tags? ("local" for the implicit cache, "custom" for explicit overrides)

That seems over-complicated. Unless you're thinking of something more elaborate than pypa/pip#6565, I see no particular value in allowing the user to manually rename to any one of a list of values which would have identical semantics. That's bound to result in support questions "what's the difference between the 'local' and 'custom' tags?"

I'd suggest that we keep it simple. Anything more complex than pypa/pip#6565 is probably YAGNI at this point.

[pradyunsg]

I'd suggest that we keep it simple. Anything more complex than pypa/pip#6565

I'll second this sentiment.

[brettcannon]

+1 from me for 'local'. I think that makes it 3 +1s from Paul, Pradyun, and myself and a +0 from Nick and none of the other proposals have garnered as much support from what I can tell.

[ncoghlan]

That seems over-complicated. Unless you're thinking of something more elaborate than pypa/pip#6565, I see no particular value in allowing the user to manually rename to any one of a list of values which would have identical semantics. That's bound to result in support questions "what's the difference between the 'local' and 'custom' tags?"

The difference would be:

• 'custom' overrides everything. Use it explicitly when you want to force the use of a specific custom-built wheel archive over all alternatives, even local builds in the wheel cache.
• 'local' overrides everything except 'custom'. It is used implicitly by the wheel cache to get locally built wheels to take precedence over pre-built remote archives.

Without that separation, I'm not sure how pip would behave in the case where there was a local build in the wheel cache and a local build in a given --find-links directory (or a remote server). With separate tags for the two use cases, confusion over precedence should only arise if there are two copies of the 'same' "custom" wheel in different locations (and that confusion isn't any different from the confusion that arises for multiple copies of any sdist or wheel archive).

[pfmoore]

'local' overrides everything except 'custom'. It is used implicitly by the wheel cache to get locally built wheels to take precedence over pre-built remote archives.

First of all, why would this be needed? Any front end that builds and caches wheels locally should prefer the cache over a download. So why would the front end need to force a non-standard tag?

Second, you'd need to define the behaviour of this tag in a way that doesn't reference pip-specific caching behaviour, if this tag is to be standardised. I'm sure you could, but there would be a lot of wording to discuss caches without assuming anything about frontend behaviour that isn't standardised.

I'm not sure how pip would behave in the case where there was a local build in the wheel cache and a local build in a given --find-links directory (or a remote server)

If "pick any one you like" isn't correct, then either the build process or the tags are at fault. There's supposed to be no difference between the wheels in that case.

I'm still going to claim YAGNI here.

[ncoghlan]

If the most common case of local builds (i.e. caching) isn't going to use it, then I'd be -1 on the tag name being "local".

"custom" would be fine, though.

[pfmoore]

I don't think I said that caching wouldn't use "local". Just that if it did, and didn't match any other "local" build, then someone's building things wrongly.

Maybe we're talking at cross-purposes here? Are we in agreement that any two wheels, both named foo-1.0-py38-none-local.whl, must be semantically identical? To me, that's self-evident from the way the wheel spec is defined. And if so, then I don't see why the cache needs a different platform tag than any other "local" tagged wheel.

[ncoghlan]

I don't think we can make any assumptions about the contents of custom wheel archives. For example, the default build settings for most extension modules won't fully tailor them to the specific hardware they're running on - they'll use some more generically portable defaults. If you do a custom wheel build tailored to your specific hardware, you may be able to get things to run faster than they do with the default build settings. Alternatively, it may be that the public build settings require a hardware feature that your particular system doesn't provide (e.g. maybe the project is assuming the availability of very recent CPU vector operations, and you have an older CPU without those instructions), and you need to build it differently to let your hardware run it.

So if a local build with default settings is picked up from a cache in preference to a custom build with particular build settings, that would be potentially problematic.

The simplest scheme that would cover that is to have a "custom" tag that never gets used implicitly by tools - it would strictly be used when you need a "the default options are wrong for some reason, use this instead" override.

However, having two wheels called foo-1.0-py38-none-custom.whl with different contents (because they've been compiled with different settings) wouldn't be an error per se - it would only be an error to have both of them visible to the same installation command, since the installer would arbitrarily pick one of them.

At that point, I think you're right that also having a dedicated "this was built implicity with the default settings" tag would be a case of YAGNI - there's nothing wrong with using the default tags for caching purposes, so there's no need to change that.

But perhaps it is a sign that custom should be defined as a tag prefix, rather than as a fixed tag? Installers still wouldn't have a way to consistently decide between them, but it would make it possible to communicate directly in the filename why the custom artifacts exist by using names like foo-1.0-py38-none-custom_linux_x86_64_with_avx512.whl or foo-1.0-py38-none-custom_build_for_win10.whl. The plain string "custom" would then mainly be used as a marker in lists of compatibility tags to indicate the relative priority of custom wheel archives.

[pfmoore]

But perhaps it is a sign that custom should be defined as a tag prefix

We've established that we require a PEP for this change. I think that a suggestion like this is sufficiently large that it ought to be handled in that PEP process (i.e., it should be part of the discussion with the broader community on Discourse) rather than here. So I won't comment on your suggestion any further now (other than to simply say that I still personally think this is a case of YAGNI) and I'll defer discussion until we have an actual PEP.

As a general point, I'd suggest that we've made as much progress here as we can, and the next step should be for someone to write an actual PEP.

[aragilar]

In case people haven't seen it, there's a draft PEP for this at https://discuss.python.org/t/draft-pep-the-local-wheel-tag/3145/.