bpo-28833: Fix cross-compilation of third-party extension modules

[xdegaye]

https://bugs.python.org/issue28833

[vstinner]

Is there a place where cross-compilation is already documented? If not, did you consider to start to write such documentation?

This change is backward incompatible. It drops support for _PYTHON_xxx environment variables. Would it be feasible to keep the old env var, or would it make the code too complicated? Maybe just document it properly (complete What's New in Python 3.9, see emy comment below).

[xdegaye]

@vstinner

Is there a place where cross-compilation is already documented? If not, did you consider to start to write such documentation?

This has been started in issue https://bugs.python.org/issue28542, the issue is languishing.

This change is backward incompatible.

The PR implementation uses the shell script generated by autoconf from Misc/python-config.sh.in to get the Python configuration variables (see _get_python_config() in Lib/sysconfig.py) needed to import the correct sysconfigdata module (see get_build_time_vars() in Lib/sysconfig.py).

So _PYTHON_HOST_PLATFORM, _PYTHON_SYSCONFIGDATA_NAME and PYTHONPATH defined by PYTHON_FOR_BUILD in configure.ac are not used anymore. If someone is currently handcrafting these environment variables to cross-compile third party extensions, his setup won't be hurt by these changes when PYTHON_PROJECT_BASE is simply replaced by _PYTHON_PROJECT_BASE everywhere in this PR.

Do you suggest that the PR uses _PYTHON_PROJECT_BASE instead of PYTHON_PROJECT_BASE for backward compatibility ? I don't mind.

Maybe just document it properly (complete What's New in Python 3.9, see emy comment below).

I will also add an entry to What's New in Python 3.9.

[vstinner]

So _PYTHON_HOST_PLATFORM, _PYTHON_SYSCONFIGDATA_NAME and PYTHONPATH defined by PYTHON_FOR_BUILD in configure.ac are not used anymore.

Sorry, I didn't know that these env vars were set by configure.ac. I understand that they are "internal" variables and so can be changed anytime without breaking the backward compatibility.

[vstinner]

LGTM.

Honestly, I don't understand fully the change, I never cross-compiled Python. But I trust @xdegaye since he is working on cross-compilation to Android ARM for like 2 or 3 years. Overall, the change seems sane and to cleanup the code.

I don't merge the change to let others review the change. But if nobody comments this PR, please ping me back in 2 weeks, and I will just merge it.

@xdegaye: What's your plan for Python 3.7 and 3.8? I'm fine with only modifying Python 3.9 (master branch).

[xdegaye]

I will ping this PR in 2 weeks if it has remained silent during this interval.

@vstinner I also think it is ok to only bring this change to 3.9.

Interestingly with android devices providing more RAM and more powerful processors, people have been compiling natively on the android devices. However termux issue termux/termux-packages#3012 shows that it is difficult to compile a large application such as Python Panda with clang and that the clang developpers do not support the android platform to help fixing build problems (so they had to resort to using gcc).

[pmp-p]

Since on 3.7.x cross compiling means using internals ( like _PYTHON_SYSCONFIGDATA_NAME ) it would be nice to introduce "good practice" as soon as 3.8. it's not so easy to patch master when targeting API < 24 / ndk20.

Also in android cross-compilation case, i've got some questions about how you load those third-party extension modules at runtime in a real apk ? Their final location could be a matter of setuptools/pip. But do their names start by "lib" and do they have correct soname ? i think that may be in the scope of that PR.
( cf : https://android.googlesource.com/platform/bionic/+/master/android-changes-for-ndk-developers.md )

so they had to resort to using gcc

but afaik only clang is supported by newer ndk does that mean their gcc is old ?.

[vstinner]

@xdegaye: Cross-compilation to ARM is kind of supported on Windows: https://bugs.python.org/issue33125

cc @zooba @paulmon

[xdegaye]

@pmp-p

how you load those third-party extension modules at runtime in a real apk ?

This PR is not android specific. As explained in the PR whatsnew changes, use pip to install the extension module on the target platform.

but afaik only clang is supported by newer ndk does that mean their gcc is old ?.

The NDK is not used, please do read termux/termux-packages#3012,

[xdegaye]

@vstinner
Is not https://bugs.python.org/issue33125 a native build of Python on Windows 10 ARM64 ?

[pmp-p]

@xdegaye

use pip to install the extension module on the target platform.

interesting, i did not know pip could handle android platform apk distribution (jni cpython, not termux). It would be nice to demonstrate that here https://discuss.python.org/t/packaging-and-python-for-android/2036/23

The NDK is not used, please do read termux/termux-packages#3012

So that would be a custom unsupported gcc (7) compiler, hosted by a third party, like termux or crystax ?

This PR is not android specific.

Sure android is only one amongst possible (un)supported targets, but it is picky enough on shared objects names to demonstrate that both setuptools and pip lack a way to set lib prefix, lib suffix, lib soname and maybe namespace/folders requirements . ( Stdlib "batteries" C module builder lack that too but i've no idea if it is the same module builder ) cf pmp-p/pydk#6

Maybe i'm very wrong but i'd say as long as cross-compilation need setting internals it can stay an unsupported process that may need manual tweaking and lower user expectations, while PYTHON_PROJECT_BASE makes it like "cross compilation is -now- officially supported" without specifying any supported targets, any target specific options and imply potential call for support ( posix arm-mips-*/win-posix layers/android/wasm cross compilation )

[xdegaye]

@pmp-p

Maybe i'm very wrong but i'd say as long as cross-compilation need setting internals it can stay an unsupported process that may need manual tweaking and lower user expectations, while PYTHON_PROJECT_BASE makes it like "cross compilation is -now- officially supported" without specifying any supported targets, any target specific options and imply potential call for support ( posix arm-mips-*/win-posix layers/android/wasm cross compilation )

Your rant is not welcome. If you think that documentation on the cross-compilation process on Android is lacking then please contribute to issue https://bugs.python.org/issue28542 that I mentionned above in an answer to @vstinner. If you are aware of a problem with pip on Android then please enter an issue and nosy the distutils developer (not sure that you are aware that your rant on pip above is barely comprehensible). You seem to miss entirely the point that Python is an open source project and that Python developers whether they are core developers or not, do that on their own free time.

[vstinner]

@xdegaye "Your rant is not welcome. If you think that documentation on the cross-compilation process on Android is lacking then please contribute (...) You seem to miss entirely the point that Python is an open source project and that Python developers whether they are core developers or not, do that on their own free time."

Honestly, Xavier, I don't think that @pmp-p deserves this kind of comment. This behavior goes against the code of conduct and doesn't make the discussion productive in any way. @pmp-p is trying to help. There are very few people around the world who care of having CPython working on Android, so we should all be extra welcoming and nice each others to make sure that this port can succeed.

I don't understand how people are supposed to help if there is no wiki, no documentation. How is somehow supposed to know how to build CPython for Android? There are many options: cross-compilation, build on Android, BeeWare VOC (Java), etc. There are also many options like the targeted NDK version. This topic is really complex and requires more documentation.

Cross-compilation have more users than Android: there is also cross-compilation on Ubuntu for "multiarch" (32 and 64 bits), cross-compilation on Windows to ARM, etc. It would help to have more than zero documentation for such complex build systems.

My outdated notes on Python on Android: https://pythondev.readthedocs.io/android.html

[pmp-p]

@xdegaye
well i think you got me wrong : i'm not ranting, because apart from unsupported platforms "fixing cross compilation" is maybe not needed or could break existing workflow ( multiarch ) as stated by concerned people.
But despite the risks, i'm a supporter of your PR because it steps in the good direction ( not using internals).

i'm working (a bit alone ?) since 3.7.0b to understand how cross compilation can be fixed to fully support both aosp ("android open source project") and wasm platform ( wasm is now a w3c standard real platform https://www.w3.org/2019/12/pressrelease-wasm-rec.html.en ).

concerning android, Termux is certainly a great tool for testing and p.o.c., but it's not a platform to build real world applications ( @rdb of Panda3D can correct me if i'm wrong ) and it will not support api < 21 ( official position iirc see their issues).

@vstinner

There are very few people around the world who care of having CPython working on Android

Indeed, that's why i suggested to backport to 3.8. because i don't think python-for-android( @inclement ... ), Beeware ( @freakboy3742 ... ) and others i don't know (but would like to) are interested in deploying real work ( targeting million devices ! ) on early 3.9 or get out of free time trying to run patching behind master ( that for me and pydk).

This topic is really complex and requires more documentation.

Yes, and i'll go further and assert cross-compilation efforts can't actually target anything else other than stable codebases, 3.7 and possibly 3.8. So If you want anyone to help and support then please at least target current 3.8 instead of planning backward incompatible changes.

if 3.8 is targeted, available as a simple diff to apply or better a experimental repo with both PR/issues open , potentially allowing android/wasm as a supported platforms, i'm willing to document android requirements for dynamic linking part and seek for experts/core dev to bake any required changes for that matter.

But only when it's real because i'm not good at documenting things i can't test ( i use real standalone apk on real devices : i don't trust anything else, videos of a running emulator are not valid proof for - out of the lab - real world decisions).

[xdegaye]

pmp-p comments are making the discussion productive in this PR and "trying to help" ?

@pmp-p wrote:

interesting, i did not know pip could handle android platform apk distribution

The PR does not mention apk and is not specific to android. If pmp-p wants to use apk to distribute an android application this should be discussed elsewhere. It is the second time in this PR that pmp-p attempts to bring the discussion on apk. Do you think @vstinner that this form of irony used here "interesting, i did not know..." is helpful or productive or even nice to me (since you claim we should be nice to each other) especially as I never wrote anywhere that pip could handle apk distributions ? BTW what is the status of the CoC regarding this form of rethoric ?

@pmp-p wrote:

android [...] is picky enough on shared objects names to demonstrate that both setuptools and pip lack a way to set lib prefix, lib suffix, lib soname and maybe namespace/folders requirements

If pmp-p can do this kind of demonstration, do you think @vstinner that it is "trying to help" the Python dev community to mention that here instead of entering an issue in bpo. And anyway how is this relevant to this PR ?

@pmp-p wrote:

PYTHON_PROJECT_BASE makes it like "cross compilation is -now- officially supported" without specifying any supported targets, any target specific options and imply potential call for support ( posix arm-mips-*/win-posix layers/android/wasm cross compilation )

This PR does two things:

• Fix in distutils some incorrect usage of the native interpreter configuration variables when cross-compiling extension modules.
• Avoid the error-prone usage of private environment variables defined in configure.ac to do this cross-compilation.

What are those target specific options ? Is it productive or helpful to bring those target specific options here when this is not related to the problem the PR is trying to solve and to consider that this PR must be the PR that solves all problems. If you consider @vstinner with pmp-p that this PR must solve all cross-compilation problems and all distribution problems of those cross-builds for all platforms in one shot then I will happily close this PR, the corresponding issue and remove my branch.

[xdegaye]

@vstinner
I realise now that pmp-p is using this PR as a forum and so I must admit that I was wrong in writing about a rant :(

I do not think it is right to hijack a PR as a forum. I will close the PR and stop contributing to Python on cross-compilation matters to avoid this problem in the future.

[vstinner]

Well, in a conflict, everyone has their share of responsibility. It's sad that this issue ends like that, but you're free to close your own PR @xdegaye ;-)

If someone wants to work on the Android port and/or cross compatibility, I suggest to start by writing down use cases and list the different options. Here it seems like @xdegaye and @pmp-p have different use cases and different solutions. And that's ok. There is no perfect solution which solves all use case.

[vstinner]

I read again all comments of this PR. As I already wrote, @pmp-p seems to have a very different approach than @xdegaye to run Python on Android. I don't think that there is a one way which covers all use cases. But it seems that you don't understand each others. That's why I asked multiple times to write down your use cases and document as much things as possible.

I have no idea how Python is used on Android. Should pip be supported or not? Should we only compile C extensions and build a APK package? Should we use Android NDK? I have no idea. Why CPython rather than BeeWare VOC (Java)? What are drawbacks and advantages of each option? I don't see why anyone would spend time on BeeWare VOC (Java) if CPython on Android would be perfect. So I expect many drawbacks, for cross-compiled CPython and CPython compiled directly on Android.

It's not the first time that the CPython port to Android is stalled like that. Each time I'm asking for more documentation. Each time, no one comes with concrete documentation. Then the project stops. It's sad.

I don't know anything about Android but it seems like my notes https://pythondev.readthedocs.io/android.html are the most complete resource for CPython on Android...

Note: @pmp-p asks mentioned WASM which is yet another approach which can also be valid for specific use cases.

[freakboy3742]

Since I was flagged on this one, I figured I'd address @vstinner's queries from perspective of the BeeWare project.

I can confirm that we (i.e., BeeWare) see this manifestations of this problem on both Android and Apple platforms (iOS most notably, but notionally on tvOS, watchOS as well).

Supporting mobile platforms essentially faces four problems:

1. Building Python for the platform architecture (or architectures)
2. Building binary modules for the platform architecture (or architectures)
3. Installing modules for a platform other that the development platform.
4. Using those modules at runtime, on the target device.

At present, (1) is supported reasonably well (although some external patching and tooling is required), but (2) and (3) are not supported due to some baked in assumptions about how Python will be invoked and used (in particular, that the host platform and target platform are assumed to always be the same). The mechanisms supporting (4) vary depending on the platform.

BeeWare uses the manufacturer provided developer toolchains (Xcode et al for Apple; Android Studio and Android NDK for Android/WearOS). Historically, we've used VOC to cross compile to Java bytecode on Android. We've put that approach on hold recently; our recent PSF Education grant is based on pursuing an embedded CPython approach. However, that decision doesn't alter many of the cross-platform targeting problems we're facing.

The Xcode and Android Studio toolchains both work based on cross-compilation. The manufacturer-provided tooling does not support running compilers on device. Any source code is compiled on an x86_64 desktop machine, for a target architecture (and usually multiple target architectures).

This means:

1. CPython itself needs to be compiled for multiple ARM architectures, using x86_64 as the host architecture.
2. python setup.py needs to able to generate binary modules and wheels for multiple ARM architectures, using x86_64 is the host architecture.
3. pip install needs to be able to be told to install an iOS/Android wheel, when the host platform is macOS/Linux/Windows.

(1) is supported reasonably well at present (with some patching and external tooling required). The problem comes with (2) and (3). This patch appears to be addressing (2).

From BeeWare's perspective, the issue is building and distributing wheels, not APK packages. APKs are designed to distribute entire, self contained applications, not libraries. If I'm building an application, and I need to use a third party package with a C module, I need to be able to install that module into my application's site_packages. My application will be distributed as an APK, but an APK is not a helpful or useful medium of exchange for a third party Python module.

APK packaging is required to distribute final applications. However, BeeWare has come to the position that setuptools isn't well suited to application packaging; we're in the process of converting Briefcase into being a standalone PEP518 compliant build tool that can be used to produce application installers for distribution through traditional App Store channels.

To date, BeeWare has taken the strategy of avoiding problems of binary modules, only claiming to support pure Python modules. Limited support for binary modules has been achieved by embedding the code for those modules into the application code itself (i.e., embedding the libraries into the binary of the native application). However, this isn't a generalized solution. We'd much rather be able to pip install an Android/iOS wheel from PyPI, or at least cross-compile source to produce the relevant binary modules when building the native application for distribution.

I am aware that some Android Python users/developers are using the device as a development platform (so they have GCC, pip etc hosted on device); however, this use case is not an option for BeeWare. App stores (especially iOS App Store) strictly control the ability of distributed applications to download external dependencies that aren't distributed with the application; on-device development processes violate those requirements.

I'm happy to provide any any further clarifications or details if anyone wants them.