Publish a platform-specific wheel for Apple M1 (macOS ARM64) on PyPI

[oliverl-21]

SUMMARY

add Package for macos arm64

ISSUE TYPE

• Feature Idea

ADDITIONAL INFORMATION

N/A

[webknjaz]

It's an interesting idea but until GitHub adds respective VMs to GHA, there's nothing we can do about this. So for now the users will have to install ansible-pylibssh from the sdist.

Refs:

• Support for VMs on Apple M1 actions/runner-images#2187
• Add ARM + MacOS target to be able to use on self-hosted runners actions/runner#805

[radokristof]

Well for me even building it locally does not work.
Before Ventura I was able to build it and install it locally. Now this fails with an error related to clang:

* Installing packages in isolated environment... (wheel)
* Building wheel...
[1/1] Cythonizing /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.pyx
running build_ext
building 'pylibsshext._libssh_version' extension
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -DCYTHON_TRACE=1 -DCYTHON_TRACE_NOGIL=1 -I/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-env-yn3ws_5a/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.c -o /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.o
creating build
creating build/lib.macosx-13-arm64-3.10
creating build/lib.macosx-13-arm64-3.10/pylibsshext
clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -lssh -DCYTHON_TRACE=1 -DCYTHON_TRACE_NOGIL=1 /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.o -o build/lib.macosx-13-arm64-3.10/pylibsshext/_libssh_version.cpython-310-darwin.so
ld: library not found for -lssh
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang' failed with exit code 1

ERROR Backend subprocess exited when trying to invoke build_wheel
ERROR: InvocationError for command /Users/radokristof/Documents/git/pylibssh/.tox/build-dists/bin/python -m build --outdir /Users/radokristof/Documents/git/pylibssh/dist/ . (exited with code 1)

[l3ender]

Seeing the same error after installing libssh (via brew) and soft linking libssh into command line tools (ref). I see the comment to "install ansible-pylibssh from the sdist"; can anyone provide detail on it?

I am only trying to install ansible-pylibssh because it a dependency of the ansible.netcommon collection. :(

[dmacduff]

@l3ender @radokristof I was able to build locally with the correct compiler flags exported, no symlink required. Procedure.

[webknjaz]

@dmacduff you shouldn't need to go through the contributor path (using tox etc.)
If you have all the build deps setting the env vars should be enough.
I'd expect CFLAGS="-I $(brew --prefix)/include -I ext -L $(brew --prefix)/lib -lssh" pip install ansible-pylibssh to just work. If you want to produce a wheel artifact, change the command to pip wheel or python -m build --wheel.

[dmacduff]

@webknjaz Brilliant! I spun up a new venv, and once I pip cache purged, your solution worked like a charm, no git clone, tox, or locally built wheels required. Much simpler. Now back to S.O. to update my answers.

[andresfrancopt]

Hi @webknjaz,
Will the solution you provided still work if I'm using a virtual environment?
Thanks in advance.

[webknjaz]

@andresfrancopt the person commenting before you said it worked.

[samdoran]

There is actually a universal wheel available, but the dynamic libraries in the universal wheel are x86_64 only.

> pip download ansible-pylibssh
> unzip ansible_pylibssh-1.1.0-cp311-cp311-macosx_10_9_universal2.whl
> file pylibsshext/.dylibs/*
pylibsshext/.dylibs/libcrypto.1.1.dylib: Mach-O 64-bit dynamically linked shared library x86_64
pylibsshext/.dylibs/libssh.4.9.4.dylib:  Mach-O 64-bit dynamically linked shared library x86_64

The .so files in the wheel are universal.

So long as libssh is compiled and in the include and and linker paths (see command above), the --no-binary flag will force a build. This is essentially the same as building a wheel from source and installing it.

pip install --no-binary :all: ansible-pylibssh

[samdoran]

This was the root cause of [WARNING]: ansible-pylibssh not installed, falling back to paramiko being displayed even though ansible-pylibssh is installed.

The network connection plugin checks for pylibssh by running three imports. The second fails:

> python -c 'from pylibsshext.errors import LibsshSCPException, LibsshSessionException'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/sdoran/.pyenv/versions/debug/lib/python3.11/site-packages/pylibsshext/errors.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ssh_get_error'

[NilashishC]

Hi @ryanmerolle. Does the steps shared by @webknjaz or @samdoran their comment work for you?

[ryanmerolle]

Sorry I missed this update. Here is the error I get on my m1 MacBook:

$ pip install --no-binary :all: ansible-pylibssh
WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/cffi-1.16.0.dist-info due to invalid metadata entry 'name'
Collecting ansible-pylibssh
  Downloading ansible-pylibssh-1.1.0.tar.gz (106 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.2/106.2 kB 4.0 MB/s eta 0:00:00
  Installing build dependencies ... /
done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: ansible-pylibssh
  Building wheel for ansible-pylibssh (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for ansible-pylibssh (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      [1/1] Cythonizing /private/var/folders/ct/kd8qq0sd7y1218smxjw9fwsh0000gn/T/pip-install-gmxpv3cu/ansible-pylibssh_35f12a538de049858f6ca36822265f2e/src/pylibsshext/_libssh_version.pyx
      /private/var/folders/ct/kd8qq0sd7y1218smxjw9fwsh0000gn/T/pip-install-gmxpv3cu/ansible-pylibssh_35f12a538de049858f6ca36822265f2e/src/pylibsshext/_libssh_version.c:1201:10: fatal error: 'libssh/libssh.h' file not found
      #include "libssh/libssh.h"
               ^~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for ansible-pylibssh
Failed to build ansible-pylibssh

[samdoran]

That looks like you are missing the path to libssh in your LDFLAGS and CPPFLAGS.

[ryanmerolle]

It all makes sense to me, just not super intuitive on the mac/arm space.

If anyone stumbles on this, make sure to install libssh using brew if installing native to mac, or if using a linux container on mac install the relevant libssh-dev package.

Feel free to close this. Thanks for your help!

[webknjaz]

I'm labeling this as a "docs" issue since it seems to me that this information would be useful to be documented. Especially in the context of Python 3.12 which we don't currently ship wheels for.

[ssbarnea]

@webknjaz Cannot we really something to make this build out of the box? This bug is directly affecting devtools team and lightspeed work. Ideally shipping py312 arm64 wheel should even avoid the need to build it.

[webknjaz]

There's no usable way of having runners for this. Maybe, it's possible to have universal2 macOS wheels but that would be something that's not even verified to function. I'll get back to this once I complete my other packaging related activities that may provide me with extra insights.

[valiac]

just adding my experience with it here:
with libssh 0.10.6 & libssh2 1.11.0_1 installed, the pip package installed without issues, but ansible still didn't detect it, even after re-installing with the env vars provided here by @webknjaz

The only way i could get it to work was locally building it as per https://stackoverflow.com/questions/74048675/cannot-install-ansible-pylibssh-on-macos

Afterwards ansible picked it up just fine

[webknjaz]

libssh2 is separate from libssh and we don't use it so its presence doesn't matter.
libssh dev headers are needed to compile the C-extensions, when building from source. And so the env vars can tell the compiler where to find those headers. Once built, the wheel will be cached and the following pip installs will just unarchive it, provided that the Python version is the same.
It is possible that ansible-core didn't detect this project if you installed it into a different (virtual)env, not the one having Ansible installed. Different envs are isolated so the installed packages don't leak into one another.

[lastorel]

After checking multiple workarounds I find out - my ansible installed globally doesn't see any lib in venv like Святослав said
So at the moment if you have libssh installed by brew:

1. NOT in venv: CFLAGS="-I $(brew --prefix)/include -I ext -L $(brew --prefix)/lib -lssh" pip install ansible-pylibssh

done.