-
-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build a single wheel for cpython 3 using py_limited_api #32
Conversation
Codecov Report
@@ Coverage Diff @@
## master #32 +/- ##
=====================================
Coverage 100% 100%
=====================================
Files 7 7
Lines 151 142 -9
Branches 15 15
=====================================
- Hits 151 142 -9
Continue to review full report at Codecov.
|
Hm.. I have a feeling this bdist_wheel --py-limited-api option won't work on Windows, as different cpython minor versions may link with different versions of the MSVC runtime. This is the error Appveyor is giving, even after ensuring wheel is up-to-date:
https://ci.appveyor.com/project/hynek/argon2-cffi/build/1.0.195/job/r9ank20duuyc2mun @agronholm could you advise? |
This is specifically one of the problems PEP 384 was intended to overcome – to avoid dependencies on specific C runtime versions. |
milnksnake is nice, but it's just a helper for CFFI in ABI mode (i.e. using shared library and dlopen), whereas this module uses cffi in API mode (i.e. set_source, etc.). |
I will try to see if I can understand why that error message pops up. |
maybe it has to do with this warning printed when running
|
|
no problem, thanks for checking! Shall we open a new issue over at pypa/wheel for this? |
I found this: https://mail.python.org/pipermail/distutils-sig/2013-February/020022.html Please only open issues for solvable problems regarding the software, not the packaging standards! |
PyQt5 provides abi3 wheels for manylinux and macosx and windows wheels with "none" abi tag: https://pypi.python.org/pypi/PyQt5/5.9.2 but they only support cpython 3.5 and above (which I think are built using the same Visual Studio 2015), whereas 3.4 uses 2010. |
|
If you want to get to the bottom of this, I suggest you open a new discussion thread on distutils-sig. |
well according to the thread you linked above, "none" is the only available ABI tag on Windows...
I would like to get to the bottom of this, but I feel like this is a bit over my head. |
Well, it boils down to the question: Should ABI3 be usable on Windows? If so, what will it take? |
No, Windows does support tags like |
I asked the question on distutils-sig. |
Slowly making progress. After ruling out an issue with setuptools, I found another issue, this time with cffi itself. The latter's https://bitbucket.org/cffi/cffi/issues/350/issue-with-py_limited_api-on-windows The only workaround so far is to pass The same argon2._ffi extension module can then imported on Windows CPython 3.5 and 3.6, same as on manylinux1 and macosx. Cool! 😎 However, the issue with pep425tags and abi3 tag is still present on Windows: which means one can't Interstingly, I can build the wheel on Windows without the However, inspecting the |
The issue with cffi and Py_LIMITED_API linking to the wrong python DLL on Windows has already been fixed on cffi master branch. @hynek I hope you don't mind if I'm using your project as a guinea pig for testing the Py_LIMITED_API support in the current python packaging tools :) |
just let me know when you need something ;) |
Status update: cffi decided to revert the patch that fixed support for Py_LIMITED_API on windows. https://bitbucket.org/cffi/cffi/issues/355/importerror-dll-load-failed-on-windows The bug was fixed months ago on virtualenv master, but there hasn’t been a release since November 2016. Pity.. I think we won’t be shipping abi3 wheels for another year. |
@hynek maybe you know somebody at PyPA that could help moving this issue forward? Besides the issue with virtualenv not copying PYTHON3.DLL (easily fixed by tagging a new release, see pypa/virtualenv#1122), the other remaining issue is bdist_wheel not recognizing "abi3" as a valid tag on windows. It's not clear whether this is a limitation of the current packaging tools or rather an unsolved problem in the relevant PEPs (as @agronholm suggested above). Maybe if @hynek steps in, they will listen to him. |
…above wheel 0.30.0 added a new --py-limited-api {cp32|cp33|cp34|...} option that produces cpNN.abi3.{arch} tags on CPython 3, for distributing wheels containing extension modules which define the Py_LIMITED_API: https://docs.python.org/3/c-api/stable.html CFFI does that by default since v1.8: http://cffi.readthedocs.io/en/latest/cdef.html#ffibuilder-compile-etc-compiling-out-of-line-modules This means CFFI extension modules are now named *.abi3.so, and should work on any version of CPython >= 3.2 (that's when the stabe Py_LIMITED_API was first introduced). In setup.cfg, I still kept the minimum version supported to 3.4.
not giving up huh? :D (I got a notification from AppVeyor) |
I saw some activity on the pypa/wheel repo and wanted to see if anything had changed since last time.. but apparently no. |
As wheel maintainer I will do whatever I can to fix this, but I don't know what to do! I posted a message on distutils-sig regarding the state of wheel. |
@agronholm thanks, I really appreciate that! |
I believe we ship abi3/limited_api wheels now using https://github.com/hynek/argon2-cffi/blob/master/.azure-pipelines/wheel-builder.yml – is that what you meant? |
I’m taking the silence as confirmation ;) |
Thanks, I'm glad that that at least for macos and linux we could build abi3 wheels! Also I wonder, why are you pinning an older version of pip in your wheel-builder.yml? |
As far as I know, there's no abi3 wheels for Windows. We need to pin pip because newer version support pep517 but no passing arguments (= abi3) to wheel which makes it explode. This is terrible, but I don't know what to do about it since the pip project is informed since January or February. |
So why do you need to pass it as an argument? This thread mentions setting this option in |
It's about pypa/pip#6304 – according to Paul and Alex there's no other way? I'm more than happy to be corrected. |
To the best of my understanding, the pep517 command should invoke the setuptools build backend which then should invoke wheel, which does read this option in setup.cfg. I have not tested this locally yet, and I could be mistaken about how the processing chain works. |
My understanding of the problem here is that there are no abi3 for Windows, so we need to set the flags on a case to case basis. :( |
wheel
0.30.0 added a new--py-limited-api {cp32|cp33|cp34|...}
option that producescpNN.abi3.{arch}
tags on CPython 3, for distributing wheels containing extension modules which define thePy_LIMITED_API
CFFI does that by default since v1.8 (see cffi docs)
This means CFFI extension modules are now named
*.abi3.so
, and should work on any version of CPython >= 3.2 (that's when the stabe Py_LIMITED_API was first introduced).In
setup.cfg
, I still kept the minimum version supported to 3.4.This new bdist_wheel option should be ignored on implementations different from cpython.