Install Failures on Termux

[xloem]

Bug Description

I was unable to install a pyo3 pip package on termux without manually providing a config file. For some reason the library parameters were not set. From #1800 (comment)

Steps to Reproduce

1. Install termux on android
2. In termux, install python and rust: pkg install python rust
3. In termux, install based58: pip3 install based58 or manually from https://github.com/kevinheavey/based58 (pkg install git)

Backtrace

$ PYO3_PRINT_CONFIG=1 maturin build
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
🐍 Not using a specific python interpreter (With abi3, an interpreter is only required on windows)
📦 Built source distribution to /data/data/com.termux/files/home/srctmp/based58/target/wheels/based58-0.1.0.tar.gz
   Compiling pyo3 v0.15.1
   Compiling pyo3-macros v0.15.1
error: failed to run custom build command for `pyo3 v0.15.1`

Caused by:
  process didn't exit successfully: `/data/data/com.termux/files/home/srctmp/based58/target/debug/build/pyo3-c92d6bef7e401530/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG

  -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.7
  shared=true
  abi3=true
  build_flags=WITH_THREAD
  suppress_build_script_link_lines=false
warning: build failed, waiting for other jobs to finish...
error: build failed
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `cargo rustc --manifest-path Cargo.toml --message-format json --lib --`

Your operating system and version

TERMUX_APK_RELEASE=F_DROID TERMUX_APP_PID=16828 TERMUX_IS_DEBUGGABLE_BUILD=0 TERMUX_MAIN_PACKAGE_FORMAT=debian TERMUX_VERSION=0.118.0 Packages CPU architecture: aarch64 Subscribed repositories: # sources.list deb https://termux.mentality.rip/termux-main/ stable main # x11-repo (sources.list.d/x11.list) deb https://packages.termux.org/apt/termux-x11 x11 main # sources.list.d/pointless.list deb https://its-pointless.github.io/files/24 termux extras Updatable packages: termux-keyring/stable 3.3 all [upgradable from: 3.2] Android version: 8.1.0 Kernel build information: Linux localhost 3.18.71-perf-gc7467a0 #1 SMP PREEMPT Thu Jan 24 02:00:44 EST 2019 aarch64 Android Device manufacturer: BlackBerry Device model: BBB100-1

Your Python version (python --version)

Python 3.10.4

Your Rust version (rustc --version)

rustc 1.60.0

Your PyO3 version

pyo3 = { version = "0.15.1", features = ["extension-module", "abi3-py37"] }

How did you install python? Did you use a virtualenv?

apt

Additional Info

This might be a bug with based58 . I'm not sure.

I tried mutating the Cargo.toml line and encountered different sets of errors. Providing a manual config based on the output of manually running cargo let me install the package successfully, as described in the linked origin issue.

The way Termux works, the root system and all libraries are installed relative to a prefix local to the application data folder. This path is stored in $PREFIX and is almost always /data/data/com.termux/files/usr . In order to run conventional shell scripts with #!/usr/bin/foo lines, Termux keeps LD_PRELOAD filled with a dynamic library called termux-exec that mutates this behavior when the scripts are executed, so as to find their launching processes relative to the prefix. Everything in Termux is installed without root. These are the biggest quirks of Termux that I am aware of.

[davidhewitt]

Thanks @xloem. Can you share the custom config file you needed to create as your workaround?

[xloem]

Sorry for leaving that out.

$ PYO3_PRINT_CONFIG=1 cargo build

implementation=CPython
version=3.7
shared=true
abi3=true
lib_name=python3.10
lib_dir=/data/data/com.termux/files/usr/lib
executable=/data/data/com.termux/files/usr/bin/python
pointer_width=64
build_flags=WITH_THREAD
suppress_build_script_link_lines=false

[davidhewitt]

Oops fat finger error!

@xloem can you try the latest maturin beta (0.12.13b1)? I think with recent changes credit to @ravenexp and @messense this might actually just work.

[davidhewitt]

(to clarify latest maturin beta, without your config file. There was a change that allows PyO3 to look for Python in more cases and it might have happened to fix this case.)

[ravenexp]

@xloem can you try the latest maturin beta (0.12.13b1)? I think with recent changes credit to @ravenexp and @messense this might actually just work.

That will only work if the upstream package upgrades from PyO3 v0.15.1 to v0.16.x

[ravenexp]

implementation=CPython
version=3.7
shared=true
abi3=true
lib_name=python3.10
lib_dir=/data/data/com.termux/files/usr/lib

Can we even support abi3 on Android given that unlike other Unixes, the libpython DSO must be linked explicitly by the extension modules, but nothing like python3.dll exists for Unix-like platforms?

[davidhewitt]

That will only work if the upstream package upgrades from PyO3 v0.15.1 to v0.16.x

Ah darn yes, I missed that.

Can we even support abi3 on Android given that unlike other Unixes, the libpython DSO must be linked explicitly by the extension modules, but nothing like python3.dll exists for Unix-like platforms?

I think so? My understanding is that as long as the wheel links against libpython.so or libpython3.so or similar which doesn't have a version-specific name, then at dynamic load time we should be able to load the abi3 symbols from any library with a matching name. I guess we might need to introduce some logic in pyo3-build-config to ensure this?

[ravenexp]

Oh, I was wrong, there is libpython3.so, which is not a symlink to libpython3.y.so, as I though before:

Dynamic Section:
  NEEDED               libpython3.10.so.1.0
  NEEDED               libc.so.6
  SONAME               libpython3.so

and it's even mentioned in PEP 384.

So yes, the build script logic can be altered to link to libpython3.so instead.

[xloem]

• upgrading from 1.15.1 to 1.16.4 moves the error from pyo3 to pyo3-ffi (which also shows 1.16.4)
• upgrading maturin to 0.12.13-beta.1 without upgrading pyo3 does not appear to change the error
• upgrading both eliminates the lib_name error. the compilation fails due to a different error. I'm not sure it's always occurring:
  $ maturin build

    Finished dev [unoptimized + debuginfo] target(s) in 1m 37s
💥 maturin failed
  Caused by: Cannot repair wheel, because required library libm.so could not be located.

$ maturin pep517 build-wheel

💥 maturin failed
  Caused by: Cannot repair wheel, because required library libdl.so could not be located.

[messense]

I guess Termux environment isn't manylinux compatible, so you should run maturin build --skip-auditwheel.

[xloem]

I'm sorry, I'm having trouble testing this, and I'm kinda frustrated and confused (phone keeps running out of space which disables internet, reinstalled rust repeated ...). The errors haven't quite seemed deterministic for me. I did get a non-wheel build to succeed repeatedly using the upgraded libraries. I haven't reproduced this today.

However, performing both upgrades appears to resolve the lib_name error.

[davidhewitt]

No problem, thanks very much for testing & sharing all this information with us.

Hopefully what this concludes is that once the ecosystem has had a chance to update, this class of problems is resolved. For now the custom config file will work for you at least 👍

[davidhewitt]

I guess Termux environment isn't manylinux compatible, so you should run maturin build --skip-auditwheel.

@messense is that true of all builds to Android? (Should maturin turn off wheel auditing if the android target is detected?)

[messense]

is that true of all builds to Android?

Well I don't really know.

Should maturin turn off wheel auditing if the android target is detected?

It was already disabled when invoking with pip wheel which should work for the normal pip install process. I don't think people should run maturin build on deployment target unless they're debugging though.

[xloem]

Arright, everything works with --skip-auditwheel. I had to update both the rust buildfile to reference pyo 0.16.4 and the python buildfiles to reference maturin 0.12.13-beta.1 .