Python: fix cross-compilation of extension modules

[FRidh]

Issue description

The changes in #53123 improve the cross-compilation situation of Python. It's now possible to produce functioning cross-compiled Python packages.

Neither the interpreter nor the packages will depend on the build machine. It does involve in both the interpreter and wrapPythonPrograms an ugly hack to rewrite shebangs from build Python to host Python. This is a general issue discussed in #33956. In this case I think we should somehow instruct distutils of our host Python.

Cross-compiling extension modules still fails. In the interpreter build we've nuked references to the compiler, and that seems to be an issue. When not cross-compiling, all goes fine though.

Steps to reproduce

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

[FRidh]

PYTHON_FOR_BUILD should supposedly always be set when cross-compiling. First, let's see whether we can set strictDeps = true; in buildPythonPackage and fix the outfall of that.

[Mic92]

cc @matthewbauer @Ericson2314

[FRidh]

Useful changeset SynoCommunity/spksrc@6a071f8

[FRidh]

We need to keep _sysconfigdata*.py when cross-compiling because it's needed for extension modules. Probably best put it in a separate output so we don't have Python depending on dev outputs.

Using export PYTHON_FOR_BUILD=${getBin pythonForBuild.interpreter} seemed to break cross-compilation of Python itself.

[danbst]

Seems like we need a special Python for building extension modules (talking about cross-compiling numpy).

platform.machine() should return target platform

sysconfig.get_config_var('LDSHARED') should use target compiler.

[danbst]

Oh yeah, I've made numpy crosscompile, but with hacks. Given an overlay entry:

                python3 = super.python3.override {
                    packageOverrides = pythonself: pythonsuper: {
                        numpy = pythonsuper.numpy.overrideAttrs (old: {
                            _PYTHON_HOST_PLATFORM = pkgs.lib.removeSuffix "-" self.stdenv.cc.targetPrefix;
                            buildPython = self.buildPackages.python3;
                            patches = (old.patches or []) ++ [
                                (self.writeText "numpy-patch-cross-extensions.diff" ''
diff --git a/numpy/distutils/ccompiler.py b/numpy/distutils/ccompiler.py
index 14451fa..39dbf8f 100644
--- a/numpy/distutils/ccompiler.py
+++ b/numpy/distutils/ccompiler.py
@@ -140,7 +140,7 @@ def CCompiler_spawn(self, cmd, display=None):
             display = ' '.join(list(display))
     log.info(display)
     try:
-        subprocess.check_output(cmd)
+        subprocess.check_output([ (cmd[0] if cmd[0].startswith("${self.stdenv.cc.targetPrefix}") else "${self.stdenv.cc.targetPrefix}" + cmd[0]) ] + [ (arg.replace(os.environ['buildPython'], "${self.python3}")) for arg in cmd[1:]])
     except subprocess.CalledProcessError as exc:
         o = exc.output
         s = exc.returncode
diff --git a/numpy/random/setup.py b/numpy/random/setup.py
index a820d32..fa1d536 100644
--- a/numpy/random/setup.py
+++ b/numpy/random/setup.py
@@ -51,9 +51,9 @@ def configuration(parent_package=''', top_path=None):
         EXTRA_COMPILE_ARGS += ['-std=c99']
         INTEL_LIKE = any(arch in platform.machine()
                          for arch in ('x86', 'i686', 'i386', 'amd64'))
-        if INTEL_LIKE:
-            # Assumes GCC or GCC-like compiler
-            EXTRA_COMPILE_ARGS += ['-msse2']
+        #if INTEL_LIKE:
+        #    # Assumes GCC or GCC-like compiler
+        #    EXTRA_COMPILE_ARGS += ['-msse2']

     # Use legacy integer variable sizes
     LEGACY_DEFS = [('NP_RANDOM_LEGACY', '1')]
                                '')
                            ];
                        });
                    };
                    self = self.python3;
                };

it builds numpy without references to build python. @FRidh What is nice, your suggestion about nuked references wasn't an issue at all (e.g. it compiles even with refs nuked).

PS: I still haven't run this, so don't know if it works.

[FRidh]

What is nice, your suggestion about nuked references wasn't an issue at all (e.g. it compiles even with refs nuked).

That's convenient, though we of course need to check that for more packages.

A fix: buildPython = self.python.pythonForBuild
Maybe we could add to the setup hook _PYTHON_HOST_PLATFORM if it is generic enough.

[Ericson2314]

It sounds like the basic issue is that configure-script-like-things are not aware of cross compilation at all.

[FRidh]

#91178 was a generic fix. I am not aware of any further issues, but then again, I don't use cross-compilation myself.

[FRidh]

Splicing of Python packages in #104201.

[FRidh]

I consider this fixed with #98915 and #104135. Let's see until we find more needs to be done 😄

Note pybind11 does not yet cross-compile, other than that all the packages I tried that have extension modules build.