Build Linux binary wheels

[remram44]

Distributing binary Linux wheels is a thing now, and would allow people to pip-install reprozip without a C compiler and the Python and SQLite3 development libraries.

reprounzip and the rest don't include C extensions and thus are already distributed as wheels.

The usual approach is to build against an old version of the libc so that every distribution can use it. manylinux provides multiple versions of Python ready to build wheels, however reprozip doesn't build on the Docker image (versions of ptrace header and sqlite3 are too old). But perhaps it can be made compatible with it?

[kaczmarj]

Hi @remram44 - I would greatly appreciate it if reprozip distributed wheels. I was able to compile and install reprozip on CentOS 6.9 with libc version 2.12 (circa 2010) and Python 2.7. I haven't tried Python 3.x.

PEP 513 (the manylinux1 PEP) requires compiling on CentOS 5, but due in part to the CentOS 5 EOL, which happened several months ago, PEP 571 was drafted (manylinux2). This specifies CentOS 6 as the build distribution, which reprozip should be compatible with. It seems, however, that manylinux2 is still in development. manylinux2 wheels can be created in the Docker images markrwilliams/manylinux2, but I do not know how broad the support is yet. For example, the readme of the manylinux github repository does not mention manylinux2 or PEP 571. There is discussion about manylinux2 in pypa/manylinux#152.

I was able to build linux wheels using the Docker image markrwilliams/manylinux2:x86_64 and following the build instructions in this example shell script.

$ docker run --rm -it markrwilliams/manylinux2:x86_64
$ yum install -y sqlite-devel
$ git clone https://github.com/ViDA-NYU/reprozip.git
$ cd reprozip/reprozip
$ for PYBIN in /opt/python/*/bin; do "${PYBIN}/pip" wheel . -w wheelhouse; done
$ for whl in wheelhouse/*.whl; do auditwheel repair "$whl" -w wheelhouse; done
$ ls wheelhouse
certifi-2018.1.18-py2.py3-none-any.whl         reprozip-1.0.11-cp27-cp27m-manylinux2_x86_64.whl
chardet-3.0.4-py2.py3-none-any.whl             reprozip-1.0.11-cp27-cp27mu-linux_x86_64.whl
idna-2.6-py2.py3-none-any.whl                  reprozip-1.0.11-cp27-cp27mu-manylinux2_x86_64.whl
PyYAML-3.12-cp27-cp27m-linux_x86_64.whl        reprozip-1.0.11-cp33-cp33m-linux_x86_64.whl
PyYAML-3.12-cp27-cp27m-manylinux2_x86_64.whl   reprozip-1.0.11-cp33-cp33m-manylinux2_x86_64.whl
PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl       reprozip-1.0.11-cp34-cp34m-linux_x86_64.whl
PyYAML-3.12-cp27-cp27mu-manylinux2_x86_64.whl  reprozip-1.0.11-cp34-cp34m-manylinux2_x86_64.whl
PyYAML-3.12-cp33-cp33m-linux_x86_64.whl        reprozip-1.0.11-cp35-cp35m-linux_x86_64.whl
PyYAML-3.12-cp33-cp33m-manylinux2_x86_64.whl   reprozip-1.0.11-cp35-cp35m-manylinux2_x86_64.whl
PyYAML-3.12-cp34-cp34m-linux_x86_64.whl        reprozip-1.0.11-cp36-cp36m-linux_x86_64.whl
PyYAML-3.12-cp34-cp34m-manylinux2_x86_64.whl   reprozip-1.0.11-cp36-cp36m-manylinux2_x86_64.whl
PyYAML-3.12-cp35-cp35m-linux_x86_64.whl        requests-2.18.4-py2.py3-none-any.whl
PyYAML-3.12-cp35-cp35m-manylinux2_x86_64.whl   rpaths-0.13-py2.py3-none-any.whl
PyYAML-3.12-cp36-cp36m-linux_x86_64.whl        urllib3-1.22-py2.py3-none-any.whl
PyYAML-3.12-cp36-cp36m-manylinux2_x86_64.whl   usagestats-0.7-py2.py3-none-any.whl
reprozip-1.0.11-cp27-cp27m-linux_x86_64.whl

[remram44]

Great news! I had tried building on manylinux1, but that was indeed too old. Releasing binaries from now on shouldn't be difficult.

[remram44]

I can't seem to upload the wheels to PyPI:

Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: remram
Enter your password: 
Uploading reprozip-1.0.11-cp27-cp27m-linux_x86_64.whl
HTTPError: 400 Client Error: Binary wheel 'reprozip-1.0.11-cp27-cp27m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://upload.pypi.org/legacy/

Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: remram
Enter your password: 
Uploading reprozip-1.0.11-cp27-cp27m-manylinux2_x86_64.whl
HTTPError: 400 Client Error: Binary wheel 'reprozip-1.0.11-cp27-cp27m-manylinux2_x86_64.whl' has an unsupported platform tag 'manylinux2_x86_64'. for url: https://upload.pypi.org/legacy/

PEP-0513 allows manylinux1 wheels to be uploaded to PyPI. I'm guessing manylinux2 hasn't been enabled yet.

[kaczmarj]

I don't think pip knows what a manylinux2 wheel is yet. pypa/pip#5008 implements preliminary support for manylinux2 but has not been merged yet. I wonder if users would need a very new version of pip to install a manylinux2 wheel.

For what it's worth, PEP 571 is active (python/peps#565 merged 15 days ago) and says that

PyPI should permit wheels containing the manylinux2 platform tag to be uploaded in the same way that it permits manylinux1. It should not attempt to verify the compatibility of manylinux2 wheels.

[remram44]

I don't think a new version of pip is required, but someone needs to whitelist the manylinux2 platform on PyPI. Or did I do something wrong to get manylinux2_x86_64 != manylinux2?

[kaczmarj]

I can replicate your issue. It is likely due to manylinux2 not being implemented yet in pip and auditwheel (pypa/pip#5008 and pypa/auditwheel#88, respectively). manylinux2 is very new, so I think the support is not present yet. We should try again once those PRs are merged.

By the way, I was able to upload the wheel reprozip-1.0.11-cp35-cp35m-manylinux2_x86_64.whl after simply renaming it to reprozip-1.0.11-cp35-cp35m-manylinux1_x86_64.whl (to test.pypi.org, I deleted the project).

[remram44]

Renamed my manylinux2 packages to manylinux1. You should be able to download them from PyPI! Will upload as manylinux2 once PyPI updates.

[remram44]

Marking this as closed. Thank you for your help!

[takluyver]

manylinux2 is now called manylinux2010, but there's still some way to go before it's widely supported (see pypa/manylinux#179 ).

Renamed my manylinux2 packages to manylinux1.

For the record, changing the compatibility tag manually is not recommended. Each manylinux version defines particular compatibility requirements. If you make a wheel on a newer platform and then pretend it's manylinux1 compatible, it can be installed on systems that support manylinux1, even if it's not compatible with them. That could lead to unexpected errors and crashes.

Hopefully it's not too big a deal, because it should only affect quite old systems which are not compatible with manylinux2010. But keep that in mind if you get bizarre issues reported.

[remram44]

I agree, but if it's the only one PyPI will accept, I don't have a choice. Why they decided to validate that field I don't understand.

[takluyver]

PyPI allowing other tags on upload won't help until pip recognises those tags for installation, and it doesn't recognise manylinux2010 yet, even in master (there's a PR, but it needs updating: pypa/pip#5008 ).