place manylinux1 after linux tag

[dholth]

PEP 425 tags are designed to score 'the one most specific to the
installation environment' above more general tags. Although linux_x86_64
does not mean much globally, within the confines of a given machine
it means compiled for this machine.

Placing linux above manylinux1 lets you compile a wheel on your local
machine, for your local machine, and have it get picked above the
manylinux1 wheel from pypi without disabling manylinux1 completely.

[dholth]

Theory of operation

linux_x86_64 wheels only work on your machine but compatibility information is out of band so you can't really share them. This is how Linux ABI compatibility has traditionally worked - only point RHEL7 at the RHEL7 repository.

If you like manylinux1 but have problems with the occasional package, as I recently did with a database driver, point pip at a local wheel cache ~/wheelhouse and pypi, and recompile the recalcitrant package into ~/wheelhouse. Pip automatically and unambiguously prefers the ~/wheelhouse linux_x86_64 wheel to the pypi version.

Any linux_x86_64 wheel visible to a given invocation of pip is more likely to work correctly on that machine than the manylinux1 alternative.

[dstufft]

The problem is, that pip has not traditionally worked with compatability out of band. The meaning of a linux_x86_64 tag is "this is acceptable on all Linux 64 bit machines". We don't know why it's limited to 64bit Linux machines and isn't a platform independent wheel-- It could simply be that it contains some linux only dependencies or files that shouldn't be installed on Windows or macOS.

This tag is less specific than the manylinux tags, and it should not be preferred over them. This use case should be supported by the ability to define a custom platform tag, something like daniel_holth_linux_x86_64 where it's expected there is some definition of what that platform is (even if that definition is "whatever is on Daniel's Linux 64bit laptop"). If we want, we could also add a default platform tag that is the most specific tag that something like linux_$(cat /etc/machine-id) as well.

[dstufft]

To be clear, I am -1 on merging this.

[dholth]

Enumerated failure states with workarounds

Assume pip is looking at a local repository of your wheels in ~/wheels, plus pypi, and you are making lots of virtualenvs.

Current situation with manylinux1 on the top:

1. Manylinux wheel on PyPI does not work, no local wheel. Workaround: compile wheel locally and manually install it, or disable manylinux1, or disable wheels.
2. Manylinux wheel on PyPI does not work, own linux_x86_64 wheel in ~/wheels. Manually install local wheel for each environment or disable manylinux1.
3. No wheel on PyPI, own linux_x86_64 wheel of same version in ~/wheels. Works fine.
4. linux_x86_64 wheel in ~/wheels does not work, manylinux1 wheel from PyPI does work. Also works.

If manylinux1 comes after linux_x86_64:

1. Manylinux wheel on PyPI does not work. Workaround: compile wheel locally, store in ~/wheels, local wheel is automatically installed in virtualenvs.
2. Manylinux wheel on PyPI does not work, own linux_x86_64 wheel is in ~/wheels. The local wheel is preferred automatically.
3. No wheel on PyPI, own linux_x86_64 wheel of same version in ~/wheels. Works fine.
4. linux_x86_64 wheel in ~/wheels does not work, manylinux1 wheel from PyPI does work. Broken. You need to delete the local wheel.

If an environment variable sets the tag:

• upgraded bdist_wheel required
• easier to put wheels from different systems on a single file server
• but without a strict definition like 'manylinux1', RHEL7 means "RHEL7 plus n shared libraries this particular wheel links to". Does not tell you everything you need to know to publish the wheels.
• one more thing to configure

If the platform tag is a cryptographic hash of the os release plus all installed shared libraries, or all libraries linked to by the wheel: near-zero chance that any system will support the same platform tag.

The meaning of linux_x86_64 has not changed. It means "it might be compatible with your 64-bit linux system, and that's all we know". They are super useful for local caching and controlled sharing between identical systems. They are not useful for publishing to pypi or otherwise sharing with strangers.

When manylinux1 is on the top, if I want to install a particular database driver that has a broken manylinux1 wheel, I have to either disable manylinux1 entirely, or manually install just that one locally compiled wheel to make my virtualenv work.

Here is the very specific situation I see where having manylinux1 first is a win:

• You have configured pip to point at broken linux_x86_64 wheels.
• That same package has a working manylinux1 wheel available.
• Pip installs the working manylinux1 wheel.
• You do not notice that pip is pointing at a bad repository.

[xavfernandez]

This tag is less specific than the manylinux tags, and it should not be preferred over them.

I agree with Donald on this.
I like the linux_$(cat /etc/machine-id) idea tag though :)

[pradyunsg]

I like the linux_$(cat /etc/machine-id) idea tag though :)

Is there anyone who would mind this?

/cc @xavfernandez @pfmoore @dstufft @dholth

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

[pradyunsg]

So, does moving forward with a machine-specific tag added at the first index sounds like a reasonable way forward for everyone here?

note-to-self: linux_machine_<machine-id>

[mboisson]

@pradyunsg :

Note that using some machine ID would not work. On HPC clusters, those would likely be different, yet the environment is the same and the local wheel should still be preferred. Letting one specify a preferred tag in the pip config would be best.

[dholth]

I consider the linux_x86_64 tag to be analogous non-routeable ip addresses e.g. 192.168.x.y. They may overlap globally but are useful within a controlled set of systems.

[pradyunsg]

Closing in favor of #6523.