Fix finding headers when cross compiling

[lopsided98]

When cross-compiling extension modules using build_ext, get_python_inc() may be called to return the path to Python's headers. However, it uses the sys.prefix or sys.exec_prefix of the build Python, which returns incorrect paths when cross-compiling (paths pointing to build system headers).

To fix this, we use the INCLUDEPY and CONFINCLUDEPY conf variables, which can be configured to point at host Python by setting _PYTHON_SYSCONFIGDATA_NAME. The existing behavior is maintained on non-POSIX platforms or if a prefix is manually specified.

This was originally submitted to stdlib distutils as python/cpython#22440, but unfortunately they are no longer accepting patches to it even though setuptools still uses stdlib distutils by default.

I tested this with the netifaces package, which compiles an extension module and fails with the following error if it is cross-compiled from x86_64 to armv7l without this patch, due to the word size difference:

  In file included from /nix/store/c84d97qz2x7yfaijcjwapvrnbwlkljs0-python3-3.8.7/include/python3.8/Python.h:63,
                   from netifaces.c:1:
  /nix/store/c84d97qz2x7yfaijcjwapvrnbwlkljs0-python3-3.8.7/include/python3.8/pyport.h:726:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
    726 | #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
        |  ^~~~~
  error: command 'armv6l-unknown-linux-gnueabihf-gcc' failed with exit status 1

After applying this patch and setting _PYTHON_SYSCONFIGDATA_NAME and SETUPTOOLS_USE_DISTUTILS appropriately, the build succeeds.

The stdlib distutils version of this patch is currently applied in nixpkgs (NixOS/nixpkgs#98915) and we have not observed any breakage.

[jaraco]

Thanks for the contrib.

I don't have a lot of experience with the sysconfig module. Right now, there's a lot of work going on right now to refactor sysconfig to enable better configurability by different platforms, so it's possible this change will conflict with some of those efforts.

Is there a test that could be added to this PR that would help capture the missed expectation (and thus fail before and pass after the patch)?

[lopsided98]

Would you be willing to take another look at this patch? I have rebased it and reworked it a bit (not visible unless the PR is reopened). I will also explain more thoroughly how we (nixpkgs) use this patch to enable cross-compilation of Python packages with extension modules, as well as the challenges involved in developing an automated test.

We use the following techniques to make cross-compilation work:

• Apply this patch to either stdlib distutils (before setuptools 60) or the bundled distutils in setuptools.
• Add the directory containing the _sysconfigdata module for the host platform Python installation to PYTHONPATH, so that it can be imported from build platform Python.
• Set _PYTHON_HOST_PLATFORM to the correct value for the host platform (e.g. linux-armv7l)
• Set _PYTHON_SYSCONFIGDATA_NAME to the appropriate value for the host platform (e.g. _sysconfigdata__linux_arm-linux-gnueabihf).

setup.py is executed using build platform Python (by necessity, since the build machine can't normally run host platform binaries), but the sysconfig module returns data from the host platform Python installation, due to _PYTHON_SYSCONFIGDATA_NAME.

If the package contains extension modules, eventually distutils.sysconfig.get_python_inc() will be called to get the location of the Python headers. Without this patch, the build platform headers will be returned, since get_python_inc() returns paths relative to sys.base_prefix or sys.base_exec_prefix. This causes problems when the build platform word length doesn't match the host platform, resulting in errors like the following:

In file included from /nix/store/01kia41csjia67pry1rv828i9pvnnqfq-python3-3.9.12/include/python3.9/Python.h:50,
                 from btrfsutilpy.h:27,
                 from error.c:20:
/nix/store/01kia41csjia67pry1rv828i9pvnnqfq-python3-3.9.12/include/python3.9/pyport.h:741:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  741 | #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
      |  ^~~~~

This patch avoids this problem by relying on sysconfig to provide the include paths. With _PYTHON_SYSCONFIGDATA_NAME set appropriately, the include directories will be returned from host platform Python.

The latest version of the patch will always use sysconfig on posix if the relevant config variables are defined, otherwise it will fall back to the current behavior.

I have been having trouble coming up with an effective and simple automated test due to the amount of infrastructure required. I think you would need multiple Python installations in order to demonstrate how _PYTHON_SYSCONFIGDATA_NAME can affect the return value of get_python_inc(). IMO, the most important thing is ensuring that existing (non-cross) use cases are not affected by this change.

[jaraco]

Yes. I'd be happy to revisit the patch. Thanks for all the extra detail. I'm currently unable to reopen the pull request due to the branch having been forced pushed or recreated:

If you wish to push the old code, I can re-open the PR and then you can force push again... or you can open a new PR.

[lopsided98]

I opened #145