Windows Stable ABI Support

[reaperhulk]

• Pip version: 9.0.1
• Python version: Any py3
• Operating system: Windows

Description:

I'd like to be able to use the stable ABI to lessen the burden of distributing wheels. pip 9 supports this on macOS, but on Windows it does not. The stable ABI is a bit less useful on Windows due to CRT issues, but with the changes to CRT linking in Python 3.5 and above it seems like it should be possible to do a single stable ABI build that works with Python 3.5+? I am only a Windows Python user to the extent required by the build infrastructure for PyCA projects so hopefully someone more knowledgeable can help out here 😄

Assuming the stable ABI should be supported on Windows then we can start digging into the problem. It turns out that pep425tags.get_supported() doesn't show abi3 as a supported tag on Windows. The implementation uses imp.get_suffixes() and on Windows this does not return anything with abi3 in it. Setuptools uses a function get_abi3_suffix that keys on pyd in Windows, but it's unclear to me if this is the correct behavior. Is there a more correct way to detect support for stable ABI on Python 3 in Windows?

(Aside: It'd be nice if pep425tags had a canonical single repo that is then vendored by pip/wheel rather than being divergent modules)

[dstufft]

I suspect either @pfmoore or @zooba will have the most knowledge about what is correct here.

[pfmoore]

I think the first stage would need to be for distutils/setuptools to be able to build extensions that use the stable ABI. That is, the extension would need to link to python3.dll rather than python36.dll. I don't know how that would work, so I can't help there, sorry. Once setuptools could build such binaries, it would be a matter of pip wheel gaining a flag that let it request such things when building, and then as you say the get_supported side of things to allow pip install to consume them.

[anthrotype]

Are you sure it's not supported on Windows?
From what I can understand, PyQt5 seems to have Windows wheels that supports the stable ABI:
https://pypi.python.org/pypi/PyQt5

[pfmoore]

Interesting. They must have renamed those manually (and of course, they'd need to change the name each time a new Python version appeared, but that's not exactly a big issue except around release time). I guess they had to do some manual work when building to make the extensions use only the stable ABI, but there's no indication of how they did that. Their approach looks like a reasonable workaround for the lack of first-class support for the stable ABI, though.

[reaperhulk]

That's a clever workaround (that I'll be interested in using in the interim), but passing none for the ABI tag and specifying multiple interpreter versions doesn't seem like the right long term solution.

[dstufft]

@pfmoore Heh, shows what I know about Windows. So I guess the next step here is to open an issue with setuptools to get it to link against python3.dll instead of python36.dll when py_limited_api=True on the Extension object.

[pfmoore]

Probably. Plus the build would need to define Py_LIMITED_ABI - but it might be worth waiting for @zooba to comment as he knows most about the MSVC support in setuptools in case there's any quirks that need to be considered. Personally, I wasn't even aware that the stable ABI had sufficient features to allow its use for defining extensions - I thought it was purely for embedding.

[zooba]

The stable ABI is meant for extensions as well, and generally works fine. (There's actually no point in using it for embedding on Windows, since you would also have to ship a Python runtime with your application, which means you know which version you will have.) Defining Py_LIMITED_ABI is the build should be sufficient to link against the correct DLL, as that is selected in PC/pyconfig.h and doesn't need to be specified to the linker explicitly.

Potential CRT conflicts should be isolated by the limited ABI, as that was an intentional part of the design. But of course it's fairly easy to spoil that if you are not careful in your extension.

The main caveat I'm aware of right now is bpo-23903 - basically, keeping python3.dll up to date is a manual process, and most people who add to the limited ABI (normally accidentally) do not update the required file. The headers with Py_LIMITED_ABI and the stable functions you can actually link against on Windows do not match, and the only reasonable fix (IMHO) is to make all of the currently stable APIs permanently stable - the people whose APIs accidentally ended up on that list do not like that solution, and so we're at a standstill until someone swoops in with a strong enough opinion to resolve it.

In short:

• some limited-ABI-using code that builds on Linux will not build on Windows (fixed by building with the latest Python on Windows)
• some limited-ABI-using code that works on Windows may not work on earlier versions of Python 3 (fixed by building with the earliest Python on Windows)

So it's a bit of a mess. I love the concept of the limited ABI, but it's hard to recommend it over running multiple builds for supported versions.

[dstufft]

@zooba Ok so if #define Py_LIMITED_ABI is in the extension code, then it should already be linking against python3.dll?

[zooba]

One other requirement for 3.5 and later is to make sure that the extension suffix is .pyd and not (e.g.) .cp35-win_amd64.pyd - distutils defaults to the most specific suffix when building.

But the point of the more specific suffix is to allow you to have multiple extensions in the same folder for different interpreters. So you can have a multi-targeted wheel with separate pyd's in it and the interpreter will load the correct one.

[dstufft]

@zooba With setuptools if you pass py_limited_abi=True to the Extension object, it makes sure that the extension is .pyd, which I found a bit surprising that it wasn't .abi3.pyd like it is on Linux but I digress.

[zooba]

@dstufft Yes to both. When I was adding the more specific suffixes on Windows the .cp3-win32.pyd version was blocked in review (could have been abi3-plat but I was taking the Python version part of the tag rather than the API version).

[dstufft]

So that leaves us back here then, since it seems like Python, setuptools, and wheel are currently capable of producing wheels that are linking to the limited ABI on Windows, but we're ignoring them because our pip.pep425tags takes a linux centric view by looking for an abi3 in the accepted extensions.

[pradyunsg]

If I understand correctly, pip.pep425tags needs to be updated to add support for Windows Stable ABI tags.

Is that all? (I feel like I'm missing something)

[anthrotype]

it seems like Python, setuptools, and wheel are currently capable of producing wheels that are linking to the limited ABI on Windows, but we're ignoring them...

well, in my tests, setuptools doesn't seem to be currently capable of building extension modules that link to the limited ABI on Windows. It is still linking to the versioned PYTHON35.DLL or PYTHON36.DLL instead of PYTHON3.DLL associated with the Py_LIMITED_API.

I wrote to distutils-sig and opened an issue at setuptools (sorry for cross posting, I thought it would be useful to mention this here as well)
pypa/setuptools#1248

[anthrotype]

Sorry, I take that back...

It's not setuptools nor distutils that's linking with the incorrect python DLL.
When compiling extensions for CPython on Windows, it's neither setuptools nor distutils that pass either pythonXY.lib or python3.lib to the linker command line; the path to the the version-specific import library is defined by the pyconfig.h header via a compiler directive like #pragma comment(lib,"python36.lib").

Only when Py_LIMITED_API is already defined before including pyconfig.h, the correct "python3.lib" import library is passed to the linker with that #pragma.

I was incorrectly defining Py_LIMITED_API only after including the Python.h header (which must be in turn including pyconfig.h).

Ok good, so we are back to the original issue that pip/wheel pep425tags module fails with AssertionError on Windows because it

takes a linux centric view by looking for an abi3 in the accepted extensions

---

EDIT: I could have spared a couple of hours if I had read more carefully @zooba's commend above...

Defining Py_LIMITED_ABI is the build should be sufficient to link against the correct DLL, as that is selected in PC/pyconfig.h and doesn't need to be specified to the linker explicitly.

[anthrotype]

What needs to be done to add support for stable ABI tag on Windows?
I’d love to see this in pip 10 (but it’s probably too late, isn’t it?)

[pfmoore]

Sorry but it's way too late for pip 10. We're releasing the beta this weekend.

Step 1 is probably not directly pip related, though. Building wheels that use the limited ABI would be a requirement, so I'd start with confirming (and documenting) precisely how to build such wheels. This may require changes to the wheel project to allow users to supply some form of --stable-abi flag when building the wheel. Once it's possible to easily build (and publish!) wheels using the stable ABI, then would be the right time to add the appropriate tags to pip.

[anthrotype]

yes, building the extension module works fine with setuptools.

It's just a matter of using py_limited_api=True in setuptools.Extension, and adding #define Py_LIMITED_API at the top, before the Python.h include, so the right python3.dll gets linked automatically.

See https://github.com/anthrotype/hello-world-cpython-extension

The bdist_wheel command already has an option --py-limited-api cpXX, however it fails when one attempts to use it on Windows, because abi3 tag is supposedly unsupported.
See my attempts here:
hynek/argon2-cffi#32
https://ci.appveyor.com/project/hynek/argon2-cffi/build/1.0.195/job/fcic8mqi1f856vjc#L481

The only way I can build a stable ABI wheel on Windows is without the --py-limited-api option, and by simply renaming it to *-cp35.cp36.cp37-none-win_amd64 (note the none as the abi tag). Then I can successfully install and import the wheel in all the listed cpython versions.

the appropriate tags to pip

that's the question, exactly what would the appropriate tag be? "abi3" or "none"?

[anthrotype]

cc @agronholm

[agronholm]

I have no idea. I wasn't involved in the definition of these standards.

[pfmoore]

It's just a matter of ...

My point here was that I'd prefer to get the upstream questions resolved, so that there's a clear process for projects to build stable ABI wheels (that doesn't involve the manual processes you describe here). Then, we need agreement on what the correct "stable ABI for Windows" tags should be, and wheel should support them. That may well involve a round of discussion on distutils-sig and an update to the relevant standards. I'm not clear on what's involved in those steps, so that's about all I can say there. Once we have agreement on what the relevant tag is (and it's documented) it's easy to add it to pip. But we don't want to add a tag to pip before we have that agreement.

[anthrotype]

That may well involve a round of discussion on distutils-sig

I did try that a couple of months ago, maybe we should revive the discussion there:
https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html

[jlaine]

Unless I'm mistaken, the virtualenv issue has been fixed since 15.2.0. I'm a bit confused as to what the next step is?

[reaperhulk]

@jlaine the virtualenv issue is indeed fixed, but we now have a nasty long tail of existing virtualenvs (both outdated installs and virtual environments created with older versions) that would exhibit very ugly failure modes in the presence of a limited API wheel. That's not to say we shouldn't figure out how to move forward now that the primary blocker is fixed, but it's going to be a bit tricky.

[anthrotype]

There is still the problem that —py-limited-api option of bdist_wheel command doesn’t work on Windows because abi3 isn’t recognised for that platform, so one has to work around this by using none as the abi tag.
See my comment above #4445 (comment)

[chrahunt]

The pip part of this was resolved for #7327 in #7367, essentially we now just use a plain "abi3" when installing for a supported version of CPython. I will close this now but if there's still any pip-specific action indicated, please say so!