PyPI wheels for Python 3.8

[mgedmin]

It would be nice to have Python 3.8 wheels for Windows at least (Linux users tend to have C compilers) published on PyPI.

This is a bit complicated by Appveyor not supporting Python 3.8 yet (appveyor/ci#3142), but I expect that to happen within the next couple of days. In any case it's possible to use 3.8 on Appveyor today if you do something like this: mgedmin/check-python-versions@162f908

[mgedmin]

lol github closed this prematurely.

Appveyor and manylinux scripts are primed, all we have to do to get linux and windows wheels for Python 3.8 on pypi is to make a new release.

[mgedmin]

I'm not going to do it right now, for two reasons:

• should the next version be 4.6.0 or 4.5.1? (is python 3.8 support a feature or a bugfix?)
• travis mac builds don't have python 3.8 yet and maybe there are mac enthusiasts who would like to open a PR for this?

[mgedmin]

I guess nobody cares, so: 4.5.1 is pushed to PyPI. Wheels should be uploaded as soon as CI builds finish.

[mgedmin]

Both manylinux jobs (64-bit and 32-bit) failed, but for different reasons.

64-bit build

+ auditwheel repair wheelhouse/persistent-4.5.1-cp27-cp27m-linux_x86_64.whl -w /io/wheelhouse/
Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 10, in <module>
    sys.exit(main())
  File "/opt/_internal/cpython-3.7.5/lib/python3.7/site-packages/auditwheel/main.py", line 50, in main
    rval = args.func(args, p)
  File "/opt/_internal/cpython-3.7.5/lib/python3.7/site-packages/auditwheel/main_repair.py", line 49, in execute
    from .repair import repair_wheel
  File "/opt/_internal/cpython-3.7.5/lib/python3.7/site-packages/auditwheel/repair.py", line 14, in <module>
    from .wheeltools import InWheelCtx, add_platforms
  File "/opt/_internal/cpython-3.7.5/lib/python3.7/site-packages/auditwheel/wheeltools.py", line 15, in <module>
    from wheel.util import urlsafe_b64encode, open_for_csv, native  # type: ignore
ImportError: cannot import name 'open_for_csv' from 'wheel.util' (/opt/_internal/cpython-3.7.5/lib/python3.7/site-packages/wheel/util.py)
The command "if [[ $TRAVIS_TAG ]]; then bash .manylinux.sh; fi" failed and exited with 1 during .

32-bit build:

+ /opt/python/cp38-cp38/bin/pip install -U pip setuptools wheel cffi
...
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/opt/_internal/cpython-3.8.0/include/python3.8 -c c/_cffi_backend.c -o build/temp.linux-i686-3.8/c/_cffi_backend.o
  c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
...

presumably because there are no 32-bit Python 3.8 manylinux wheels for cffi on PyPI (yet?).

[mgedmin]

So, missing wheels:

• cp38 MacOSX
• all manylinux variations

[mgedmin]

Filed an issue about the missing 32-bit cffi wheel: https://bitbucket.org/cffi/cffi/issues/433/i686-manylinux1-wheel-for-python-38-is

[mgedmin]

Filed an auditwheel bug at pypa/auditwheel#206

[mgedmin]

Incidentally, we should build wheels for all commits (without uploading them to PyPI) so we can discover these problems before making releases.

[jamadden]

Are 32-bit Linux wheels important? I know they still matter on Windows (because I've had complaints if they don't get uploaded) but I don't build 32-bit Linux wheels for gevent or RelStorage and I've never gotten any feedback asking for them. The newer manylinux tags may not even have 32-bit docker images (or they're at least substantially delayed compared to the 64-bit images).

The 64-bit issue is due to a mismatch between dependency versions and is a known problem. I've solved the problem in my projects by not upgrading wheel.

Vaguely related (I should open a new issue), I would suggest that we stop using the manylinux1 docker image.

persistent/.travis.yml

Line 55 in 4b2f8d8

env: DOCKER_IMAGE=quay.io/pypa/manylinux1_x86_64

Instead, we can use the manylinux2010 image (quay.io/pypa/manylinux2010_x86_64). Since we don't use any exotic libraries or functions, our wheels should still be compatible with the manylinux1 tag, and auditwheel will mark them as such, so we won't lose support for any systems. But we'll have the benefit of being built with newer compilers that have better standards support (C++11, for example) and presumably better optimizations. (I've got a new non-CFFI implementation of segmented LRU in RelStorage that requires these newer compilers; potentially relevant because of #45)

[mgedmin]

Are 32-bit Linux wheels important?

shrug

cffi has them on PyPI for all Python versions except 3.8

lxml has them on PyPI for all Python versions including 3.8

somebody wrote the .travis.yml snippet for persistent that builds them

I don't much care personally, and I'm happy to approve a PR that removes the build rules from .travis.yml

[mgedmin]

The 64-bit issue is due to a mismatch between dependency versions and is a known problem. I've solved the problem in my projects by not upgrading wheel.

But we're not upgrading wheel in /opt/_internal/cpython-3.7.5. We're upgrading wheel in /opt/python/*/.

Do any of them share site-packages with /opt/_internal/? Are there symlinks? How can I discover how the Docker image was built?

[mgedmin]

Vaguely related (I should open a new issue)

Please do!

[mgedmin]

I'd completely forgotten about #113! The wheel problems were a known issue.

[icemac]

@mgedmin Is this issue still open or was it fixed in the meantime?

[jamadden]

I think this was fixed with #119 when .travis.yml and manylinux support was refactored. I'll go ahead and close it. Please feel free to reopen if I'm incorrect.

[mgedmin]

Well, the wheel coverage currently available on PyPI is rather spotty (no manylinux, no 3.8 for MacOS), but maybe it'll improve once somebody makes a new release.

[jamadden]

maybe it'll improve once somebody makes a new release.

Which I'm hoping to be able to do soon 😄