[Bug] Cannot build Python wheels for macOS

[BewareMyPower]

Search before asking

• I searched in the issues and found nothing similar.

Version

• OS: macOS Monterey 12.3.1
• CMake version: 3.21.1
• Clang++ version: Apple clang version 13.1.6 (clang-1316.0.21.2.5)
• Python version: 3.10.6
• Pulsar version: v2.10.1

Minimal reproduce step

Run the following command.

pulsar-client-cpp/python/build-mac-wheels.sh

What did you expect to see?

The Python wheels for macOS should be generated under pulsar-client-cpp/python/pkg/osx/ directoyr.

What did you see instead?

The error logs:

DYLD_LIBRARY_PATH=/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13 ./python.exe -E ./setup.py install \
	   	--prefix=/Users/xuyunze/.pulsar-mac-wheels-cache/py-3.7 \
		--install-scripts=/Users/xuyunze/.pulsar-mac-wheels-cache/py-3.7/bin \
		--install-platlib=/Users/xuyunze/.pulsar-mac-wheels-cache/py-3.7/lib/python3.7/lib-dynload \
		--root=/
running install
running build
running build_ext
building '_lzma' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -fPIC -O3 -mmacosx-version-min=10.15 -I/Users/xuyunze/.pulsar-mac-wheels-cache/install/include -arch x86_64 -I/usr/local/opt/krb5/include -I./Include -I/Users/xuyunze/.pulsar-mac-wheels-cache/py-3.7/include -I. -I/usr/local/opt/krb5/include -I/usr/local/include -I/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13/Include -I/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13 -c /Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13/Modules/_lzmamodule.c -o build/temp.macosx-10.15-x86_64-3.7/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13/Modules/_lzmamodule.o
In file included from /Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13/Modules/_lzmamodule.c:17:
/usr/local/include/lzma.h:293:10: fatal error: 'lzma/version.h' file not found
#include "lzma/version.h"

Anything else?

This is a reason that the Python client 2.10.1 for macOS failed to release. @codelipenghui @tisonkun

I didn't look deeper into the script at the moment, but it looks like this script has some requirements for the OS version and Python version because it just builds the specific Python from source code in local env (not docker env). Maybe we need to enhance this script for more systems. /cc @merlimat

Are you willing to submit a PR?

• I'm willing to submit a PR!

[tisonkun]

@BewareMyPower I'm running the build-mac-wheels.sh script on macOS with Apple M1 silicon. It seems that old versions of Python don't support new arch.

After comment out the old version:

diff --git a/pulsar-client-cpp/python/build-mac-wheels.sh b/pulsar-client-cpp/python/build-mac-wheels.sh
index 6a4dae7117..21e37ca887 100755
--- a/pulsar-client-cpp/python/build-mac-wheels.sh
+++ b/pulsar-client-cpp/python/build-mac-wheels.sh
@@ -21,9 +21,9 @@
 set -e
 
 PYTHON_VERSIONS=(
-   '3.7  3.7.13'
-   '3.8  3.8.13'
-   '3.9  3.9.10'
+#   '3.7  3.7.13'
+#   '3.8  3.8.13'
+#   '3.9  3.9.10'
    '3.10 3.10.2'
 )

I build pulsar_client-2.11.0-cp310-cp310-macosx_10_15_universal2.whl successfully to pulsar-client-cpp/python/dist.

You may try to reproduce and I suggest we release more Python client as possible for resolvin #15477 & #16849. Perhaps no supports for old Python versions.

[tisonkun]

Confirm that Python support Apple M1 silicon arch from 3.9: https://docs.python.org/3/whatsnew/3.9.html#macos-11-0-big-sur-and-apple-silicon-mac-support

[BewareMyPower]

Thank you @tisonkun, I'm testing it with 3.10 now.

[BewareMyPower]

@tisonkun It still doesn't work.

dyld[46563]: missing symbol called
Traceback (most recent call last):
  File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/ensurepip/__init__.py", line 277, in _main
    return _bootstrap(
  File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/ensurepip/__init__.py", line 193, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
  File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/ensurepip/__init__.py", line 93, in _run_pip
    return subprocess.run([sys.executable, '-W', 'ignore::DeprecationWarning',
  File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/python.exe', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/var/folders/b5/5vrry5193db1v8r7xfg6r3880000gn/T/tmp9_b1n933/setuptools-58.1.0-py3-none-any.whl\', \'/var/folders/b5/5vrry5193db1v8r7xfg6r3880000gn/T/tmp9_b1n933/pip-21.2.4-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/var/folders/b5/5vrry5193db1v8r7xfg6r3880000gn/T/tmp9_b1n933\', \'--root\', \'/\', \'--upgrade\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' died with <Signals.SIGABRT: 6>.
make: *** [install] Error 1

[tisonkun]

@BewareMyPower can you attach the full log file and tell me what branch you're running against?

I can build 2.11.0 pulsar-client on master on macos 12.5 M1 with Python 3.9 & 3.10 & Apple Clang:

Apple clang version 14.0.0 (clang-1400.0.29.102)
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[BewareMyPower]

Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

• out.log
• err.log

The error logs are still the same as before. I think it's because some C dependencies must be installed when building Python from source. The script might require users to install some dependencies via brew install.

For example, the 'lzma/version.h' file not found error requires the lzma dependency, so I searched for the related headers in /usr directory.

$ find /usr -name "*.h" | grep lzma
find: /usr/sbin/authserver: Permission denied
/usr/local/include/lzma.h
/usr/local/Cellar/xz/5.2.6/include/lzma.h
/usr/local/Cellar/xz/5.2.6/include/lzma/index.h
/usr/local/Cellar/xz/5.2.6/include/lzma/version.h
/usr/local/Cellar/xz/5.2.6/include/lzma/index_hash.h
/usr/local/Cellar/xz/5.2.6/include/lzma/lzma12.h
/usr/local/Cellar/xz/5.2.6/include/lzma/container.h
/usr/local/Cellar/xz/5.2.6/include/lzma/delta.h
/usr/local/Cellar/xz/5.2.6/include/lzma/vli.h
/usr/local/Cellar/xz/5.2.6/include/lzma/check.h
/usr/local/Cellar/xz/5.2.6/include/lzma/bcj.h
/usr/local/Cellar/xz/5.2.6/include/lzma/stream_flags.h
/usr/local/Cellar/xz/5.2.6/include/lzma/block.h
/usr/local/Cellar/xz/5.2.6/include/lzma/hardware.h
/usr/local/Cellar/xz/5.2.6/include/lzma/filter.h
/usr/local/Cellar/xz/5.2.6/include/lzma/base.h

lzma is included in xz dependency in macOS. But the install path is /usr/local/Cellar/xz/<version>, the include directory and library directory are not added to the related environment variables.

[BewareMyPower]

The root cause is the --with-universal-archs option. When it's universal2 on a macOS with Intel based CPU architecture, make test would fail. But if it's Intel-64, the Python wheel might not support ARM architecture. Currently, I will change the option to Intel-64 to make the script work for Intel based macs.

[Jason918]

The root cause is the --with-universal-archs option. When it's universal2 on a macOS with Intel based CPU architecture, make test would fail. But if it's Intel-64, the Python wheel might not support ARM architecture. Currently, I will change the option to Intel-64 to make the script work for Intel based macs.

Same here when I am releasing 2.10.2.
Seems the best option now would be not supporting ARM until it's fixed.

[BewareMyPower]

@Jason918 It's already fixed in latest pulsar-client-python repo. The root cause is when building universal2 Python from source, it relies on the universal binary of zlib. So a universal binary of zlib must be built before building Python.

https://github.com/apache/pulsar-client-python/blob/main/pkg/mac/build-dependencies.sh

[Jason918]

@Jason918 It's already fixed in latest pulsar-client-python repo. The root cause is when building universal2 Python from source, it relies on the universal binary of zlib. So a universal binary of zlib must be built before building Python.

@BewareMyPower Great, I will try it on 2.10.2.

[Jason918]

So a universal binary of zlib must be built before building Python.

Got error when compiling zlib. See the log:
build-mac.log
Version
OS: MacOS 10.15.7
CMake version: 3.24.2
Clang version: Apple clang version 11.0.3 (clang-1103.0.32.59)
Pulsar version: v2.10.2

@BewareMyPower Can you help release python mac wheels for release 2.10.2?
From the "Existing branches" part in #17724, we need to keep release pulsar-client-python from main repo for active branches.

[BewareMyPower]

@Jason918 I might not have time recently. The 2.8.4, 2.10.1 released also missed the Python mac wheels. IMO, for macOS users, they should turn to the new repo.

[tisonkun]

@BewareMyPower is it fixed now in the separate repo?

[BewareMyPower]

@tisonkun Yes. Close this issue by #17468 (comment)