import gensim segmentation fault (macOS Big Sur, Apple M1/Apple Silicon/ARM)

[mkunib]

Problem description

I want to use gensim on a new Apple M1 machine but I can't import gensim because of a segmentation fault.

Steps/code/corpus to reproduce

brew install openblas

cd ~/Library/python_enviroments
rm -r test
virtualenv -p /opt/homebrew/bin/python3 test
source ~/Library/python_enviroments/test/bin/activate
pip cache purge
pip install --upgrade pip

OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python -m pip install cython --no-use-pep517
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python -m pip install numpy --no-use-pep517
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python -m pip install pybind11 --no-use-pep517
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python -m pip install scipy --no-use-pep517
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python -m pip install gensim --no-use-pep517

Now I am running a python shell and run this import command:

import gensim

Output:

Python 3.9.1 (default, Jan  8 2021, 12:11:08) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import gensim
zsh: segmentation fault  python

Note: import numpy and import scipy are working.

Versions

• macOS Big Sur 11.1 (20C69)
• openblas/0.3.13 (homebrew)
• Python 3.9.1 (homebrew)

Package    Version
---------- -------
Cython     0.29.21
gensim     3.8.3
numpy      1.20.0
pip        21.0.1
pybind11   2.6.2
scipy      1.6.0
setuptools 51.3.3
six        1.15.0
smart-open 4.1.2
wheel      0.36.2

>>> import platform; print(platform.platform())
macOS-11.1-arm64-arm-64bit
>>> import sys; print("Python", sys.version)
Python 3.9.1 (default, Jan  8 2021, 12:11:08) 
[Clang 12.0.0 (clang-1200.0.32.28)]
>>> import struct; print("Bits", 8 * struct.calcsize("P"))
Bits 64
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.20.0
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.6.0

[mpenkov]

It'd be good to know what exactly is causing the segfault.

Can you run the same code via gdb and show the output?
Something like:

gdb
r python -c "import gensim"

should probably do it.

[mkunib]

Sorry, I am not familiar with gdb and I can't install it.

Output of brew install gdb:

Error: gdb: no bottle available!
You can try to install from source with:
  brew install --build-from-source gdb
Please note building from source is unsupported. You will encounter build
failures with some formulae. If you experience any issues please create pull
requests instead of asking for help on Homebrew's GitHub, Twitter or any other
official channels.

Output of brew install --build-from-source gdb:

==> Downloading https://ftp.gnu.org/gnu/gdb/gdb-10.1.tar.xz
Already downloaded: /Users/matthias/Library/Caches/Homebrew/downloads/d6dfd55ca620ab89b3209b96be8cac6a9b85c636faf527f4c5e4bc7e7ff07cee--gdb-10.1.tar.xz
==> ../configure --enable-targets=all --prefix=/opt/homebrew/Cellar/gdb/10.1 --d
==> make
==> make install-gdb
Error: Empty installation

lldb (lldb-1200.0.44.2) is installed but I don't know how use it in a python virtualenv.

[mkunib]

I think it's related to this issue trouble installing on apple silicon M1 #13409 of scipy. If I run from scipy.integrate import quad I got also a segmentation fault.

My temporary workaround: I can use gensim in an emulated x86_64 environment.

arch -x86_64 /bin/bash
cd ~/Library/python_enviroments
virtualenv -p /usr/bin/python3 my_x86_64_env
source ~/Library/python_enviroments/my_x86_64_env/bin/activate
pip cache purge
pip install --upgrade pip
pip install gensim

[mpenkov]

May be worth asking the scipy guys about this. If the problem is inside scipy, it's unlikely that gensim can do anything about it.

[gojomo]

The suggestion in #3226, that a MacOS installation using pip install gensim --no-binary :all: fixed some compatibility mismatches, might be worth trying in other cases of problems on M1 systems. (I'm not sure what other prereqs that might need for local recompilation.) Please let us know if that helps make Gensim usable on any M1 Macs!

[piskvorky]

Also, please try with a recent version of Gensim (4.x). The old Gensim 3.8.3 is no longer supported.

[piskvorky]

I have a M1 MBP now!

@mpenkov @gojomo this means I'm able to test the next release there. Or help with wheel building on that platform, if necessary.

FWIW, pip install gensim installs fine, resulting in:

• Gensim 4.1.2
• Numpy 1.22.2
• Scipy 1.8.0 (and from scipy.integrate import quad doesn't segfault)
• Python 3.8.9

This is on OSX Monterey 12.2.1 (not Big Sur like the OP), and built-in openblas instead of from brew.

[piskvorky]

Cannot reproduce, closing. If this still happens with the latest version of Gensim & scipy, please reopen.