Compilation failed with M1 macOS, PyO3 0.12, Python 3

[LER0ever]

🌍 Environment

• Your operating system and version: macOS Big Sur 11.1
• Your python version: Python 3.8 (system), Python 3.9 (homebrew), Python 3.9 (homebrew, Rosetta)
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: brew install, no.
• Your Rust version (rustc --version): rustc 1.49.0-beta.4 (877c7cbe1 2020-12-10)
• Your PyO3 version: 0.12
• Have you tried using latest PyO3 master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: no

💥 Reproducing

This happens on my own project, but I can reproduce it with maturin's test-crate

• install rust beta with rustup (stable does not yet have darwin aarch64 support)
• install python3.9 aarch64 with native homebrew

1. Compile with System python 3.8

git clone https://github.com/PyO3/maturin.git
cd maturin/test-crates/pyo3-pure
cargo build

would produce the following error:

  = note: Undefined symbols for architecture arm64:
            "_PyExc_ValueError", referenced from:
                _$LT$pyo3..exceptions..PyValueError$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h059c73bcf950bd88 in libpyo3-a087e80261990d6e.rlib(pyo3-a087e80261990d6e.pyo3.1al8rrrd-cgu.0.rcgu.o)
            "_PyExc_BaseException", referenced from:
                _$LT$pyo3..exceptions..PyBaseException$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h82828fb355186050 in libpyo3-a087e80261990d6e.rlib(pyo3-a087e80261990d6e.pyo3.1al8rrrd-cgu.0.rcgu.o)
            "_PyList_Append", referenced from:
                pyo3::types::list::PyList::append::_$u7b$$u7b$closure$u7d$$u7d$::h5630ce7dc966791a in libpyo3-a087e80261990d6e.rlib(pyo3-a087e80261990d6e.pyo3.1al8rrrd-cgu.4.rcgu.o)
            "_PyExc_SystemError", referenced from:
                _$LT$pyo3..exceptions..PySystemError$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h76492aefdb4928b0 in libpyo3-a087e80261990d6e.rlib(pyo3-a087e80261990d6e.pyo3.1al8rrrd-cgu.0.rcgu.o)
            "_PyList_New", referenced from:
                pyo3::types::list::PyList::empty::h3e9f2e6039a3ef3a in libpyo3-a087e80261990d6e.rlib(pyo3-a087e80261990d6e.pyo3.1al8rrrd-cgu.4.rcgu.o)
            "_PyErr_PrintEx", referenced from:
                pyo3::err::PyErr::print::h12dca2eb6fa69d90 in libpyo3-a087e80261990d6e.rlib(pyo3-a087e80261990d6e.pyo3.1al8rrrd-cgu.7.rcgu.o)
            "_PyObject_SetAttr", referenced from:

...
...

The python3 binary is from Apple, in universal format with both x86_64 and arm64. file $(which python3) gives /usr/bin/python3: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]

2. Compile with HomeBrew Python 3.9

basically the same process and same error

git clone https://github.com/PyO3/maturin.git
cd maturin/test-crates/pyo3-pure
PYO3_PYTHON=python3.9 cargo build

file $(which python3.9) output: /opt/homebrew/bin/python3.9: Mach-O 64-bit executable arm64

3. Compile with Rosetta under x86 works fine

[davidhewitt]

As you're using macOS - you would need cargo rustc --release -- -C link-arg=-undefined -C link-arg=dynamic_lookup?

Or otherwise you could try building with maturin develop, which will include these flags for you.

[LER0ever]

@davidhewitt Thanks for the reply, that is indeed the problem.

I copied my cargo config ~/.cargo/config which contains the fix for x86_64:

[target.x86_64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

Changing to [target.aarch64-apple-darwin] fixes the problem. Perhaps the documentation @ https://pyo3.rs/master/index.html#using-rust-from-python can be updated to include aarch64 for anyone else encountering this issue?

[LER0ever]

@davidhewitt Oh and maturin develop currently does not work because of platforms crate cannot correctly detect the triplet aarch64-apple-darwin.

$ maturin develop
💥 maturin failed
  Caused by: Could guess the current platform

Ref: https://github.com/RustSec/platforms-crate/blob/master/src/platform.rs and https://github.com/RustSec/platforms-crate/issues/31

Will open a separate issue to either maturin or platforms.