Add supported platforms to the build workflow for GitHub Actions

[kitao]

For Pyxel 1.8.0, I've updated the build.yml to build Python wheels for each platform automatically:
https://github.com/kitao/pyxel/blob/main/.github/workflows/build.yml

But for now it only supports x86_64 Windows, x86_64 Linux, aarch64 Mac and x86_64 Mac.

I would like to support other platforms such as i686 Windows and aarch64 Linux, but that would require cross-compiling for Rust-SDL2 (references SDL2) and PyO3 (references Python) and I haven't been able to do that yet.

I look forward to any information or advice on how to do that.

[kitao]

I released Pyxel 1.8.0. But I would like to add wheels for other platforms through the development of the 1.8 series.
I look forward to your information and support!

[messense]

The current x86_64 Linux wheel is not portable, it will only work on Ubuntu 20.04 and later. (In other words it requires glibc 2.31.)

You might want to check out the manylinux project to build more portable Linux wheels if you care to support Ubuntu 18.04 or other Linux distros with older glibc.

[kitao]

@messense Thank you for your advice.
It seems that if I use manylinux project, I can also support aarch64.
Is there any good example uses it in GitHub Actions?

[messense]

Well, it's just Docker images, you can use them on GitHub Actions via the container option for x86_64: https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container

aarch64 requires QEMU so you'd need to pair https://github.com/docker/setup-qemu-action with https://github.com/addnab/docker-run-action, it's going to be very slow though.

[kitao]

@messense Thank you for the information.
Actually I have an another question. If I use ubuntu-18.04 with Python3.7+ instead of ubuntu-latest (just like I did in the previous version of Pyxel), glibc-related issue will be addressed?

[messense]

Well for the time being, yes: https://github.blog/changelog/2022-08-09-github-actions-the-ubuntu-18-04-actions-runner-image-is-being-deprecated-and-will-be-removed-by-12-1-22/

[merwok]

Here is the tool that many people use, and works in github actions: https://github.com/pypa/cibuildwheel

[kitao]

@merwok I've already checked the action but I couldn't find the way to use Makefile which includes pre-process and post-process instead of normal setup.py build.
Is there any way to use make command with it?
And Rust-SDL2 also build SDL2 with CMake in the build process. It should also be taken care.

[merwok]

In that case, you would need to call cibuildwheel from the makefile, between your pre- and post-process steps.

[kitao]

@messense Though I'm not familiar with zig, is it realistic to cross-compile with zig? If it can cover both SDL2 with CMake and Rust with Maturin, it is ideal.

[messense]

cmake-rs does not have good support for cross compiling right now: rust-lang/cmake-rs#158

[messense]

The good news is that with my cmake-rs patch, it compiles with maturin and zig, and runs fine:

$ maturin build --target aarch64-unknown-linux-gnu --release --zig -o dist
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
⚠️  Warning: skipped unavailable python interpreter 'python3.8' from pyenv
🐍 Not using a specific python interpreter
📡 Using build options manifest-path from pyproject.toml
   Compiling cmake v0.1.48 (https://github.com/messense/cmake-rs.git?branch=cross-compile#15ab68ba)
   Compiling pyo3-ffi v0.16.5
   Compiling pyo3 v0.16.5
   Compiling jpeg-decoder v0.1.22
   Compiling jpeg-decoder v0.2.6
   Compiling sysinfo v0.23.13
   Compiling sdl2-sys v0.35.2
   Compiling tiff v0.6.1
   Compiling tiff v0.7.3
   Compiling image v0.23.14
   Compiling image v0.24.3
   Compiling noise v0.7.0
   Compiling sdl2 v0.35.2
   Compiling pyxel-engine v1.8.0 (/Users/messense/Projects/pyxel/crates/pyxel-engine)
   Compiling pyxel-wrapper v1.8.0 (/Users/messense/Projects/pyxel/crates/pyxel-wrapper)
    Finished release [optimized] target(s) in 30.23s
📦 Built wheel for abi3 Python ≥ 3.7 to dist/pyxel-1.8.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

$ docker run --rm -it -v $(pwd):/io -w /io ubuntu:22.04
root@a49eb76a6a8d:/io# apt update && apt install -y python3-pip
root@a49eb76a6a8d:/io# pip3 install dist/pyxel-1.8.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Processing ./dist/pyxel-1.8.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Installing collected packages: pyxel
Successfully installed pyxel-1.8.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@a49eb76a6a8d:/io# python3
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyxel
>>> pyxel
<module 'pyxel' from '/usr/local/lib/python3.10/dist-packages/pyxel/__init__.py'>
>>> pyxel.pyxel_wrapper
<module 'pyxel.pyxel_wrapper' from '/usr/local/lib/python3.10/dist-packages/pyxel/pyxel_wrapper.abi3.so'>

The patch is

diff --git a/crates/pyxel-wrapper/Cargo.toml b/crates/pyxel-wrapper/Cargo.toml
index ee2eb477..b7766e6a 100644
--- a/crates/pyxel-wrapper/Cargo.toml
+++ b/crates/pyxel-wrapper/Cargo.toml
@@ -6,7 +6,7 @@ edition = "2021"
 description = "Python wrapper for Pyxel, a retro game engine for Python"
 repository = "https://github.com/kitao/pyxel"
 license = "MIT"
-readme = "../../docs/README-abspath.md"
+readme = "../../README.md"
 categories = ["game-engines", "graphics", "multimedia"]
 keywords = ["game", "gamedev", "python"]
 
@@ -21,5 +21,8 @@ pyxel-engine = { path = "../pyxel-engine", version = "1.8.0" }
 [target.'cfg(not(target_os = "emscripten"))'.dependencies]
 sysinfo = "0.23"
 
+[patch.crates-io]
+cmake = { git = "https://github.com/messense/cmake-rs.git", branch = "cross-compile" }
+
 [package.metadata.maturin]
 name = "pyxel.pyxel_wrapper"
diff --git a/pyproject.toml b/pyproject.toml
index 47c5ff79..756f3932 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -6,7 +6,7 @@ build-backend = "maturin"
 name = "pyxel"
 version = "1.8.0"
 description = "A retro game engine for Python"
-readme = "../../docs/README-abspath.md"
+readme = "README.md"
 requires-python = ">=3.7"
 license = { file = "../../LICENSE" }
 authors = [{ name = "Takashi Kitao", email = "[email protected]" }]

Had to patch readme location locally otherwise it does not build.

[kitao]

@messense Great news! Thank you for making a patch for CMake.

There are two reasons why I use Makefile now. One is to make a README without relative link for PyPI according to the latest README (in pre-process), the other is to copy the built SDL2.dll from release folder to the python/pyxel folder (in post-process).
The former can be reduced by changing the time to make README-abspath. Regarding the latter, if there is a way to do it, I can use pure Maturin without Makefile and it means I can also use your maturin-action.

Do you know whether it is possible to copy some built artifact (e.g. SDL2.dll) while compiling a wheel with Maturin?

[messense]

Do you know whether it is possible to copy some built artifact (e.g. SDL2.dll) while compiling a wheel with Maturin?

No, it's not possible. I think It might be better to statically link SDL2 on Windows, then you don't need to copy anything.

[kitao]

@messense
Yeah, it's what I'd really like to realize.
But it seems that it is impossible to link SDL2.lib to other dll statically on MSVC.
On the other hand, it seems that it is impossible to make ABI3 Python dll on MinGW.
If I misunderstood the situation, please let me know.

[messense]

But it seems that it is impossible to link SDL2.lib to other dll statically on MSVC.

I'm not sure about this, upstream rust-sdl2 has trouble building on Windows with bundled and static-link features, see https://github.com/Rust-SDL2/rust-sdl2/runs/7975840847?check_suite_focus=true

And this thread says it possible to link statically on Windows: http://forums.libsdl.org/viewtopic.php?p=39311#39311 , and this one: libsdl-org/SDL#846

[messense]

But it seems that it is impossible to link SDL2.lib to other dll statically on MSVC.

Fixed in Rust-SDL2/rust-sdl2#1252

[kitao]

I am amazed at the speed of your actions.
So I can try it with your Rust-SDL2 fix-windows-static branch for now?

[messense]

So I can try it with your Rust-SDL2 fix-windows-static branch for now?

Yes please, note that maturin support cross compiling to Windows, so you can just use maturin build --target x86_64-pc-windows-msvc --release -o dist on macOS/Linux to test it out on your computer.

[kitao]

@messense According to your advice, I'm modifying Pyxel's build.yml to build wheels with manylinux container.
For aarch64 platform with QEMU, is there any good way to get source code and install rustup? I'd like to use command if possible.

[messense]

Now that you've patched cmake-rs, you can just compile aarch64 linux wheels with --zig, install zig is easy: pip install ziglang

pyxel/crates/pyxel-wrapper/Cargo.toml

Lines 24 to 26 in 535ea15

	[patch.crates-io]
	cmake = { git = "https://github.com/messense/cmake-rs.git", branch = "cross-compile" }
	sdl2 = { git = "https://github.com/messense/rust-sdl2.git", branch = "fix-windows-static" }

Note that my patches to rust-sdl2 are merged in master, consider to change it to upstream master branch.

[kitao]

Thank you, @messense .
I’ll change to use the latest rust-sdl2.
When I cross-compile with —zig option for aarch64, should I keep to use manylinux container? Sorry for many questions!

[messense]

When I cross-compile with —zig option for aarch64, should I keep to use manylinux container? Sorry for many questions!

No need to use manylinux container when using --zig because zig can target any glibc version, we set it to the version specified by --compatibility option which defaults to manylinux2014 for aarch64 Linux.

[kitao]

In the latest build.yml of Pyxel, I'm using --zig option for cross-copilation, but 'bits/sys_errlist.h' file not found error occurs.
@messense Do you know that I'm missing?

[messense]

I'm not sure, I haven't tried compile it on Linux with zig yet, but it worked for me on macOS before.

[messense]

I don't have the time to investigate recently, meanwhile you can also try my messense/manylinux2014-cross:aarch64 docker image for cross compiling linux aarch64 wheels without --zig.

[kitao]

@messense
Though --zig option is not used, thanks to your container, it seems that every build for seven platforms succeeded.
As soon as I have time, I will do an operation check test.

[kitao]

It's not easy... I checked this artifacts built with the latest build.yml
pyxel-wheels.zip

Platform	Result
x86_64 Windows	Worked
i686 Windows	Not checked yet
aarch64 macOS	ImportError: dlopen(/Users/takashi/Downloads/venv/lib/python3.10/site-packages/pyxel/pyxel_wrapper.abi3.so, 0x0002): symbol not found in flat namespace (___isPlatformVersionAtLeast)
x86_64 macOS	Worked
aarch64 Ubuntu	Python can't find pyxel module after installing pyxel wheel. (No module named pyxel error)
x86_64 Ubuntu	Not checked yet

It seems that cross-compiled version wheel is missing something.
And I'm wondering why aarch64 macOS version doesn't work because it worked in the previous version's build.yml.

[kitao]

Thank you.
The version of maturin is updated on manylinux and now I can import Pyxel after installing the wheel.

Then, now I'm facing "No available video device error" while initializing SDL2.
I've seen this error before and I changed the rust-sdl2 setting for Linux to only unsafe-texture with installing libsdl2-dev at that time. (To be honest, I didn't know why it worked. Perhaps because cmake's setting was changed?)

Now I'm thinking about how to handle it for manylinux.

[messense]

Then, now I'm facing "No available video device error" while initializing SDL2.

Sounds like compiling from source did not enable X11 support?

[kitao]

Ah yes, I think so.
I'm wondering there's some way to enable it with static build.

[kitao]

If SDL2 is built from source code, it seems that it depends on the current environment and dummy display driver is set to be used. So I removed bundled and static features from Linux environment and installed SDL2 package separately for dynamic link.
For installing SDL2 package, I just used sudo apt install -y libsdl2-dev but compilation failed because the compiler couldn't find libSDL2. I think it should be installed in a way that suits cross-compilation on messense/manylinux2014-cross.
Could you suggest proper way to install external packages for cross-compilation?

[messense]

The Docker image contains a working GCC cross compiler, so I think you could compile and install SDL2 to it's sysroot before building wheel.

There is a zlib build example: https://github.com/messense/manylinux-cross/blob/6ebafe2c05ea608e688dc5744b48ee594af35c2c/manylinux2014/aarch64/Dockerfile#L98-L108

[kitao]

According to this instructions, I arranged the following commands:

git clone https://github.com/libsdl-org/SDL
cd SDL
mkdir build
cd build
../configure
make
sudo make install

to

- name: Build and install SDL2
  run: |
    export CC=$TARGET_CC && \
    export AR=$TARGET_AR && \
    export RANLIB=$TARGET_RANLIB && \
    git clone --depth 1 https://github.com/libsdl-org/SDL.git && \
    cd SDL && \
    mkdir build && \
    cd build && \
    CFLAGS="-O3 -fPIC" ../configure --prefix=/usr/${{ matrix.target }}/ && \
    make -j4 && \
    sudo make -j4 install && \
    cd ../.. && \
    rm -rf SDL

In the case of x86_64-unknown-linux-gnu, it says ranlib command not found.

mkdir -p -- /usr/x86_64-unknown-linux-gnu/lib/cmake/SDL2
/usr/bin/install -c -m 644 sdl2-config.cmake /usr/x86_64-unknown-linux-gnu/lib/cmake/SDL2
/usr/bin/install -c -m 644 sdl2-config-version.cmake /usr/x86_64-unknown-linux-gnu/lib/cmake/SDL2
./libtool: line 1737: x86_64-unknown-linux-gnu-ranlib: command not found
make: *** [Makefile:189: install-lib] Error 127

Regarding aarch64-unknown-linux-gnu, configure fails due to cross compiling.

configure: error: in `/__w/pyxel/pyxel/SDL/build':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
checking whether we are cross compiling... 
Error: Process completed with exit code 77.

I'll continue to check the cross-compilation way for SDL2.

P.S.
It seems that I missed some parameters for cross compile.
https://forum.clockworkpi.com/t/tutorial-cross-compile-sdl2-c-applications-from-linux-to-clockworkos/3211

[kitao]

I specified --build and --host for cross compilation, then compilation progressed.
Now the same error occurs on both aarch64 and x86_64.

./libtool: line 1737: aarch64-unknown-linux-gnu-ranlib: command not found

[messense]

It's really odd, ranlib does exist:

root@bc3c11800dac:~/SDL# aarch64-unknown-linux-gnu-ranlib
Usage: aarch64-unknown-linux-gnu-ranlib [options] archive
 Generate an index to speed access to archives
 The options are:
  @<file>                      Read options from <file>
  --plugin <name>              Load the specified plugin
  -D                           Use zero for symbol map timestamp (default)
  -U                           Use an actual symbol map timestamp
  -t                           Update the archive's symbol map timestamp
  -h --help                    Print this help message
  -v --version                 Print version information
aarch64-unknown-linux-gnu-ranlib: supported targets: elf64-littleaarch64 elf64-bigaarch64 elf32-littleaarch64 elf32-bigaarch64 elf32-littlearm elf32-bigarm elf64-little elf64-big elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex

[messense]

Perhaps you can force it use absolute path: export RANLIB=$(which $TARGET_RANLIB)

[kitao]

Thank you.
I read this article: https://stackoverflow.com/questions/23078282/ranlib-not-found
And removed sudo, then it worked now.

But other error occurred while linking.

"/github/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-c21be34a5cae8449.rlib" "-Wl,-Bdynamic" "-lSDL2" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/github/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/__w/pyxel/pyxel/crates/pyxel-wrapper/target/x86_64-unknown-linux-gnu/release/deps/libpyxel_wrapper.so" "-Wl,--gc-sections" "-shared" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
  = note: /usr/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/4.8.5/../../../../x86_64-unknown-linux-gnu/bin/ld: cannot find -lSDL2

Library path should be modified...?

[messense]

You can try export RUSTFLAGS="-L/usr/${{ matrix.target }}/lib" and I think you might need to enable the static-link feature, otherwise it will try to link to libSDL2 dynamically.

[kitao]

If I use static feature, the SDL2 which uses dummy display device instead of X11 is linked and it doesn't work on actual Linux environment. (It is the same result with bundled and static features).

On the other hand, if I don't specify static feature, making wheel failed.
So I think I need to find out how to build SDL2 which can be adapted to users' video environment dynamically.

[kitao]

In Pyxel 1.7.2. I built Linux binary only with static feature of rust-sdl2 and it worked in cooperate with SDL2 installed by users.
I'm wondering why the current build process failed due to the lack of .so file. Maturin tries to include it?

[kitao]

I'll try SDL2 configure with --enable-x11-shared and --enabled-wayland-shared

[messense]

I'm wondering why the current build process failed due to the lack of .so file. Maturin tries to include it?

For manylinux compliance you need to bundle everything except a list of allowed shared libraries. You can use --skip-auditwheel to skip it.

[kitao]

Thanks to your advices. Finally cross-compiled aarch64 wheel worked on my aarch64 Linux VM!
I also found that the wheel name changed from manylinux to linux and I see the reason is against manylinux compliance.
In that case, is it more proper to use --manylinux off option instead of --skip-auditwheel?

[messense]

In that case, is it more proper to use --manylinux off option instead of --skip-auditwheel?

Actually you should use --manylinux 2014 along with --skip-auditwheel to pretend to be a manylinux wheel, otherwise linux wheel can not be published to PyPI.

[kitao]

I understood. I added --manylinux 2014 option as well. Thank you!

[kitao]

I've released Pyxel 1.8.1 includes new platforms. Thank you for your cooperation!

[merwok]

along with --skip-auditwheel to pretend to be a manylinux wheel

It would be best to not publish non-complicant manylinux wheels!

IMO the goal should be to have proper manylinux wheels (with libs bundled), or a complete sdist with pyproject.toml that lets pip build from source.

[kitao]

@messense
Actually I'm still investigating the difference between Pygame and Pyxel from the perspective of SDL2 support.
It seems that the difference is pre-installed packages in docker container.

Here is the Pygame's Dockerfile:
https://github.com/pygame/pygame/blob/main/buildconfig/manylinux-build/docker_base/Dockerfile-aarch64#L14-L22

Is it possible I can install same packages when running your container or I should make my own Dockerfile?
I tried sudo apt install -y libx11-dev-like things, but it seems that SDL2 couldn't find it while configuring.

I would appreciate it if you could give me the information. Thank you.

[kitao]

I checked the SDL2's makefile and found that there's no easy way to make SDL2 recognize libX11 on cross-compile environment.
I close this issue. (I may try it again in the future...)