Wheels for MacOS and Linux

[horta]

This should close #114.

Wheels built on TestPyPI: https://test.pypi.org/project/cyvcf2/#files

Platforms: MacOS, Linux
Python: 3.5, 3.6, 3.7., 3.8, 3.9

[brentp]

This looks great to me. @ccwang002 have any concerns?

[ccwang002]

Thank you for making this! This is very helpful. I tested the macOS wheels on python 3.8 and 3.9. Both wheels work for me.

I tested the python 3.8 linux wheel inside Docker. Reading VCF from the local file works, but the remote file fails. Here is how I tested it:

docker run -it --rm python:3.8-slim bash -l
pip --version
# pip 20.2.4 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)
pip install https://test-files.pythonhosted.org/packages/9e/f2/746a6cd950ff8bdad6315d8a8244b4a7ec7723e623686a21c78d4b786f09/cyvcf2-0.20.10-cp38-cp38-manylinux2010_x86_64.whl

>>> from cyvcf2 import VCF
>>> v = VCF('https://github.com/brentp/cyvcf2/raw/master/cyvcf2/tests/test.vcf.gz')
[E::hts_open_format] Failed to open file "https://github.com/brentp/cyvcf2/raw/master/cyvcf2/tests/test.vcf.gz" : Input/output error
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cyvcf2/cyvcf2.pyx", line 257, in cyvcf2.cyvcf2.VCF.__init__
  File "cyvcf2/cyvcf2.pyx", line 189, in cyvcf2.cyvcf2.HTSFile._open_htsfile
OSError: Error opening https://github.com/brentp/cyvcf2/raw/master/cyvcf2/tests/test.vcf.gz

Would you be able to investigate this?

[ccwang002]

The linux fails is probably related to how the openssl is linked. If the openssl is linked to a different SSL or lives at the different location to the wheel building environment, cyvcf2 will fail to read all remote files. Like how cyvcf2 now links to htslib, we might need to statically link to openssl as well. We might be able to leverage pyca/cryptography's existing approach (for example, their wheel building workflow or their docker image).

Alternatively, we can just note that the pre-built wheel has limited functionality. Given not everyone uses cyvcf2 to read remote VCFs, it could be a good compromise. If the user really needs this, they should self compile everything (which works using the current instruction), or they should use bioconda.

[horta]

Thanks @ccwang002 for the debugging!
I have now potentially fixed the issue. The root issue was that cyvcf2 was being built (on Linux) against ssl 1.0.x while we were testing it on machines with ssl 1.1.x machine.

I'm now installing openssl 1.1.x on the manylinux2014 image, then building and installing libssh2, curl lib, and then htslib.

I unfortunately cannot run Travis CI right now as my OSS credits is on red, and therefore I cannot use Travis do upload the wheels to test pypi.

If you have a look at this line:

python3 -m twine upload --repository testpypi --skip-existing wheelhouse/*

you might perhaps understand that I'm using the TWINE_USERNAME and TWINE_PASSWORD environment variables, defined in the Travis CI dashboard, to upload the wheels to test pypi. Is anyone willing to fork (or perhaps just merge) this and test upload to test pypi? Is there another solution while I don't have credits?

[brentp]

pending @ccwang002 's approval, I am fine to merge as long as you continue to help with any additional required changes.

[horta]

I'm happy with it, yes.

[ccwang002]

LGTM. Let's merge this and test again with the new wheels.

Once this works, we should also let CI automatically build (and upload) wheels whenever a new release is made.

[horta]

Thanks a lot!

Just to clarify. How can we setup automatic upload to Test PyPI?

[horta]

Also, do you have an ETA for next release? We can also try and build wheels for the current version of cyvcf2.

[brentp]

I can tag a release any time. Can you walk me through what I need to do re wheels? (I just haven't kept up to date with python packaging stuff).

[horta]

Sure. I would first deploy wheels to Test PyPI so we can make sure the new workflow works indeed.

The way I setup .travis.yml, it will try to deploy wheels (and the sdist) every time a new TAG is pushed to GitHub. It happens in two places (one for MacOS and one for Linux):

• cyvcf2/.travis.yml

  Line 45 in cf7991b

  if [ "${TRAVIS_TAG:-}" != "" ]; then

• cyvcf2/.travis.yml

  Line 69 in cf7991b

  if [ "${TRAVIS_TAG:-}" != "" ]; then

The line

python3 -m twine upload --repository testpypi --skip-existing wheelhouse/*

try to upload everything inside wheelhouse, which happens to be wheels only. The line

python3 -m twine upload --repository testpypi --skip-existing dist/*

does the same but with the source distribution (.tar.gz).

It will only work if you had defined the environment variables TWINE_USERNAME and TWINE_PASSWORD in Travis CI: https://travis-ci.com/github/brentp/cyvcf2/settings (their values won't show up in travis logs).

[hammer]

@brentp is there any chance the wheels will be included in a cyvcf2 release soon? We‘d love to make use of them in https://github.com/pystatgen/sgkit!

[brentp]

thanks @horta for clear description (and @hammer for reminder). i just pushed a new tag after updating those env vars.
if that appears to work, let me know and i'll change testpypi to (i assume?) pypi.

[brentp]

seems to be issues with travis-ci right now with using up my free quota. thought it would reset on today, december 1. but that doesn't seem to be the case yet.

[horta]

It might be possible to buy some credits: https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
@hammer

[hammer]

@brentp any luck with Travis credits? If not, I'd be happy to buy some for you to get these wheels published!

[brentp]

I have an email out to travis-ci with no response yet. Not sure why the credits didn't reset. Hopefully we'll hear from them soon. If not, I'll either do it manually or take you up on that. Looks like $69/month https://travis-ci.com/account/plan

[horta]

I should have tried GitHub actions. I did on Travis CI because I have more experience with... (and they didn't have those limits =P)

[hammer]

I see a release was made on December 5 but it does not appear that the wheels made it? https://pypi.org/project/cyvcf2/#files. My apologies if I'm missing something!

[brentp]

Yes, the CI is not running (https://news.ycombinator.com/item?id=25338983)
We'll have to switch to using github actions.

[horta]

But it didn't trigger a Travis building (https://travis-ci.com/github/brentp/cyvcf2/builds) probably because of lack of OSS credits.

[hammer]

We'll have to switch to using github actions.

Should we file an issue to track this work? Would you like help with that?

[brentp]

yes! please!