Add PyTorch macosx-arm64 cross-compiled build

[sbrunk]

Fixes #1461

Add PyTorch cross-compilation macosx-x86_64 -> macosx-arm64. The idea is to use the host Python interpreter to run setup.py build See #1461 for more details about this. Cross-compilation settings are derived from the PyTorch builder.

After some experimentation, I figured out that we can use a Maven profile to to use the host Python interpreter for cross-compilation in CI without affecting the existing native builds.

Since we're building just the C++ part of PyTorch (LibTorch), I think it should be OK to add the host Python interpreter like this. We might need to adapt a little bit once we have CPython and NumPy for arm, if it requires running the host Python but also linking against target libpython but I'm not sure (see #1461).

Perhaps a somewhat cleaner approach could be trying to keep $CPYTHON_PATH set to the $PLATFORM cpython and get the host interpreter out another way in the build script but that would need some more experimentations.

@saudet what do you think?

BTW I also tried cross-compiling cpython as well before realizing that we need the host interpreter for running setup.py but couldn't get it working as it's not really supported currently. See Support cross compilation on macOS #90905. Perhaps GitHub will provide free native macosx-arm64 runners at some point which should make this easier.

#1069

[saudet]

I've already been doing something like that for NumPy and SciPy, so could you bring in the host version of CPython the same way as those?
https://github.com/bytedeco/javacpp-presets/blob/master/numpy/cppbuild.sh
https://github.com/bytedeco/javacpp-presets/blob/master/numpy/pom.xml
https://github.com/bytedeco/javacpp-presets/blob/master/scipy/cppbuild.sh
https://github.com/bytedeco/javacpp-presets/blob/master/scipy/pom.xml

[saudet]

Please remove the if [[ $PLATFORM == $PLATFORM_HOST ]]; then and do something like this instead:

if ! $PYTHON_BIN_PATH -m pip install --target=$PYTHON_LIB_PATH $TOOLS; then
    chmod +x "$CPYTHON_HOST_PATH/bin/python3.12"
    export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$CPYTHON_HOST_PATH/lib/:$CPYTHON_HOST_PATH"
    chmod +x $CPYTHON_HOST_PATH/lib/python3.12/bin/*
    export PATH="$CPYTHON_HOST_PATH/lib/python3.12/bin/:$PATH"
    export PYTHON_BIN_PATH="python"
fi

[sbrunk]

if ! $PYTHON_BIN_PATH -m pip install --target=$PYTHON_LIB_PATH $TOOLS; then

fails (stopping the script) in the cross-compilation setting since PYTHON_LIB_PATH is not set at all in that case right now without arm cpython, that's why I put the whole block in a conditional.

The same is true i.e. for

export PYTHONPATH="$PYTHON_INSTALL_PATH:$NUMPY_PATH/python/"

since the PYTHON_INSTALL_PATH is not set.

But I think see the issue, it would break if we wanted to use both host and target cpython right?

What if we test for PYTHON_BIN_PATH to exist before running pip install?

[saudet]

Right, so instead of all that, let's simply put CPYTHON_PATH=CPYTHON_HOST_PATH before
if [[ -f "$CPYTHON_PATH/include/python3.12/Python.h" ]]; then

[saudet]

I suppose we need to do something similar in platform/pom.xml:

2024-01-24T00:31:14.1394220Z [ERROR] Failed to execute goal on project pytorch-platform: Could not resolve dependencies for project org.bytedeco:pytorch-platform:jar:2.1.2-1.5.10-SNAPSHOT: Could not find artifact org.bytedeco:cpython:jar:macosx-arm64:3.12.1-1.5.10-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]
2024-01-24T00:31:14.1403910Z org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project pytorch-platform: Could not resolve dependencies for project org.bytedeco:pytorch-platform:jar:2.1.2-1.5.10-SNAPSHOT: Could not find artifact org.bytedeco:cpython:jar:macosx-arm64:3.12.1-1.5.10-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots)

[saudet]

Actually, I don't think we need to include those optional dependencies at all...

[sbrunk]

Ah that's much simpler. Thanks @saudet. I just ran cross-compilation locally from your latest commit bda4475 and then some smoke tests on an arm64 machine sucessfully. 🎉