Generate python3.dll import libraries for Windows targets in the build script

[ravenexp]

I'm routinely cross-compiling Python extension modules for Windows from a Linux host using the well-supported x86_64-pc-windows-gnu rustc target. The main issue I'm having is finding a suitable python3.dll import library (for abi3 extensions) to use with PYO3_CROSS_LIB_DIR.

MinGW and MSVC compilers use incompatible import library object formats, so downloading a Windows Python distribution does not work, and compiling Python with MinGW compiler is not supported by the upstream and requires about a hundred patches to work.

I've been working around this by using shell scripts to generate the MinGW-compatible python3.dll import library using dlltool from MinGW binutils. Luckily, the CPython project maintains a list of the stable ABI symbols in a machine-readable format, which can be used to generate a DLL module definition (.def) file and the DLL import library from it.

A few weeks ago I decided to ditch the shell script hacks and package everything into a self-contained Rust crate called python3-dll-a. I'm currently using it in my own projects via build.rs, and the Windows cross-compile command is just this (no Windows Python files required):

PYO3_CROSS_LIB_DIR=target/python3-dll cargo build --target x86_64-pc-windows-gnu

Ideally, I'd like to integrate it into pyo3-build-config, so that cross-compiling abi3 extension modules for Windows would just work out of the box without messing with any PYO3_CROSS* environment variables. This should be already the case for Unix-like platforms where linking extension modules does not require a Python shared library.

The current python3-dll-a version can generate python3.dll import libraries for
x86_64-pc-windows-gnu, i686-pc-windows-gnu, x86_64-pc-windows-msvc, i686-pc-windows-msvc and aarch64-pc-windows-msvc compile targets and can support both cross and native compilation setting. As a limitation of this approach, targeting the -msvc environment ABI requires LLVM binutils to be installed on the compile host.

If there is an interest in integrating this solution, I can start working on a PR for PyO3.
This will be feature-gated at the pyo3 crate level, of course.

[messense]

See also PyO3/maturin#830

[messense]

Ideally, I'd like to integrate it into pyo3-build-config

I'm also open for integrating it into maturin. It should be doable.

[davidhewitt]

I'm happy for it to go in either place. You might find it a bit easier to put it in maturin; to keep pyo3-build-config simple there's a single file impl_.rs which is compiled into both the build script and the library.

[ravenexp]

I'm happy for it to go in either place. You might find it a bit easier to put it in maturin; to keep pyo3-build-config simple there's a single file impl_.rs which is compiled into both the build script and the library.

Thanks!

I'll try to go with pyo3-build-config first, because I like to think that cargo build should always work. It's also a prerequisite for cargo clippy and cargo test to work. If it is implemented in maturin only these commands will also become inaccessible.