[Python] Illegal hardware instruction on pyarrow import #20848

asfimport · 2019-01-16T09:02:00Z

I can't import pyarrow, it crashes:

>>> import pyarrow as pa
[1]    31441 illegal hardware instruction (core dumped)  python3

Core dump is attached to issue, it can help you to understand what is the problem.

The environment is:

Python 3.6.7
PySpark 2.4.0
PyArrow: 0.11.1
Pandas: 0.23.4
NumPy: 1.15.4
OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Environment: Python 3.6.7
PySpark 2.4.0
PyArrow: 0.11.1
Pandas: 0.23.4
NumPy: 1.15.4
OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Elchin

Original Issue Attachments:

core

_{Note: This issue was originally created as ARROW-4272. Please see the migration documentation for further details.}

asfimport · 2019-01-16T09:37:08Z

Antoine Pitrou / @pitrou:
Without the exact same binaries and libraries as you, it will be difficult to do something with the core dump.
Could you run gdb python3.6 core ? Then under the gdb prompt type bt to get the C backtrace and post the result here.

asfimport · 2019-01-16T09:47:28Z

Elchin:
@pitrou of course:

#0  0x00007f445ba46070 in std::pair<std::__detail::_Node_iterator<std::pair<absl::string_view const, std::function<bool (tensorflow::Variant*)> >, false, true>, bool> std::_Hashtable<absl::string_view, std::pair<absl::string_view const, std::function<bool (tensorflow::Variant*)> >, std::allocator<std::pair<absl::string_view const, std::function<bool (tensorflow::Variant*)> > >, std::__detail::_Select1st, std::equal_to<absl::string_view>, tensorflow::hash<absl::string_view, void>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::pair<absl::string_view, std::function<bool (tensorflow::Variant*)> > >(std::integral_constant<bool, true>, std::pair<absl::string_view, std::function<bool (tensorflow::Variant*)> >&&) () from /home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so
#1  0x00007f445ba47f25 in tensorflow::UnaryVariantOpRegistry::RegisterDecodeFn(std::string const&, std::function<bool (tensorflow::Variant*)> const&) ()
   from /home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so
#2  0x00007f445ba2467c in tensorflow::variant_op_registry_fn_registration::UnaryVariantDecodeRegistration<tensorflow::Tensor>::UnaryVariantDecodeRegistration(std::string const&) () from /home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so
#3  0x00007f445b991885 in _GLOBAL__sub_I_tensor.cc () from /home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so
#4  0x00007f446e520733 in call_init (env=0xc109f0, argv=0x7ffc89f96088, argc=1, l=<optimized out>) at dl-init.c:72
#5  _dl_init (main_map=main_map@entry=0x14a1090, argc=1, argv=0x7ffc89f96088, env=0xc109f0) at dl-init.c:119
#6  0x00007f446e5251ff in dl_open_worker (a=a@entry=0x7ffc89f941b0) at dl-open.c:522
#7  0x00007f446e2862df in __GI__dl_catch_exception (exception=exception@entry=0x7ffc89f94190, operate=operate@entry=0x7f446e524dc0 <dl_open_worker>,
    args=args@entry=0x7ffc89f941b0) at dl-error-skeleton.c:196
#8  0x00007f446e5247ca in _dl_open (file=0x7f445c6ed8c0 "/home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so", mode=-2147483646,
    caller_dlopen=0x7f4466aa4cda <py_dl_open+90>, nsid=<optimized out>, argc=1, argv=<optimized out>, env=0xc109f0) at dl-open.c:605
#9  0x00007f446dcfcf96 in dlopen_doit (a=a@entry=0x7ffc89f943e0) at dlopen.c:66
#10 0x00007f446e2862df in __GI__dl_catch_exception (exception=exception@entry=0x7ffc89f94380, operate=operate@entry=0x7f446dcfcf40 <dlopen_doit>,
    args=args@entry=0x7ffc89f943e0) at dl-error-skeleton.c:196
#11 0x00007f446e28636f in __GI__dl_catch_error (objname=objname@entry=0xafbf20, errstring=errstring@entry=0xafbf28, mallocedp=mallocedp@entry=0xafbf18,
    operate=operate@entry=0x7f446dcfcf40 <dlopen_doit>, args=args@entry=0x7ffc89f943e0) at dl-error-skeleton.c:215
#12 0x00007f446dcfd735 in _dlerror_run (operate=operate@entry=0x7f446dcfcf40 <dlopen_doit>, args=args@entry=0x7ffc89f943e0) at dlerror.c:162
#13 0x00007f446dcfd051 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#14 0x00007f4466aa4cda in py_dl_open () at ./Modules/_ctypes/callproc.c:1348
#15 0x0000000000502d6f in _PyCFunction_FastCallDict (kwargs=<optimized out>, nargs=<optimized out>, args=<optimized out>,
    func_obj=<built-in method dlopen of module object at remote 0x7f4466ceb188>) at ../Objects/methodobject.c:234
#16 _PyCFunction_FastCallKeywords (kwnames=<optimized out>, nargs=<optimized out>, stack=<optimized out>, func=<optimized out>) at ../Objects/methodobject.c:294
#17 call_function.lto_priv () at ../Python/ceval.c:4837
#18 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#19 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446c756448, for file /usr/lib/python3.6/ctypes/__init__.py, line 348, in __init__ (name='/home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so', mode=0, handle=None, use_errno=False, use_last_error=False, _FuncPtr=<_ctypes.PyCFuncPtrType at remote 0x14a0cf8>)) at ../Python/ceval.c:4166
#20 _PyEval_EvalCodeWithName.lto_priv.1761 () at ../Python/ceval.c:4166
#21 0x0000000000501ba7 in _PyFunction_FastCallDict () at ../Python/ceval.c:5070
#22 0x0000000000591461 in _PyObject_FastCallDict (kwargs=0x0, nargs=2, args=0x7ffc89f947c0, func=<function at remote 0x7f4466ceabf8>) at ../Objects/abstract.c:2310
#23 _PyObject_Call_Prepend (kwargs=0x0, args=<optimized out>, obj=<optimized out>, func=<function at remote 0x7f4466ceabf8>) at ../Objects/abstract.c:2373
#24 method_call.lto_priv () at ../Objects/classobject.c:317
#25 0x000000000054b813 in PyObject_Call (kwargs=0x0, args=('/home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so',),
    func=<method at remote 0x7f446e61fb88>) at ../Objects/abstract.c:2261
#26 slot_tp_init () at ../Objects/typeobject.c:6420
#27 0x0000000000555421 in type_call.lto_priv () at ../Objects/typeobject.c:915
#28 0x00000000005a730c in _PyObject_FastCallDict (kwargs=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<type at remote 0xc56148>)
    at ../Objects/tupleobject.c:131
#29 _PyObject_FastCallKeywords () at ../Objects/abstract.c:2496
#30 0x0000000000503073 in call_function.lto_priv () at ../Python/ceval.c:4861
#31 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#32 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x14c2858, for file /home/elch10/.local/lib/python3.6/site-packages/pyarrow/compat.py, line 229, in import_tensorflow_extension (os=<module at remote 0x7f446e5ce4a8>, tensorflow_loaded=False, path='/home/elch10/.local/lib/python3.6/site-packages/tensorflow', ext='/home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so', ctypes=<module at remote 0x7f44690d9548>)) at ../Python/ceval.c:754
#33 _PyFunction_FastCall (globals=<optimized out>, nargs=21768280, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#34 fast_function.lto_priv () at ../Python/ceval.c:4954
#35 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#36 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#37 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0xb77f08, for file /home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py, line 50, in <module> ()) at ../Python/ceval.c:4166
#38 _PyEval_EvalCodeWithName.lto_priv.1761 () at ../Python/ceval.c:4166
#39 0x0000000000511d78 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0,
    locals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated),
    globals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated), _co=<code at remote 0x7f446ce3fc90>) at ../Python/ceval.c:4187
#40 PyEval_EvalCode (
    locals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated),
    globals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated), co=<code at remote 0x7f446ce3fc90>) at ../Python/ceval.c:731
#41 builtin_exec_impl (module=<optimized out>,
    locals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated),
    globals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated), source=<code at remote 0x7f446ce3fc90>) at ../Python/bltinmodule.c:983
#42 builtin_exec.lto_priv () at ../Python/clinic/bltinmodule.c.h:283
#43 0x000000000056617e in PyCFunction_Call () at ../Objects/methodobject.c:112
#44 0x000000000050bb66 in do_call_core (kwdict={},
    callargs=(<code at remote 0x7f446ce3fc90>, {'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsi...(truncated), func=<built-in method exec of module object at remote 0x7f446e704638>) at ../Python/ceval.c:5102
#45 _PyEval_EvalFrameDefault () at ../Python/ceval.c:3404
#46 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446e588930, for file <frozen importlib._bootstrap>, line 219, in _call_with_frames_removed (f=<built-in method exec of module object at remote 0x7f446e704638>, args=(<code at remote 0x7f446ce3fc90>, {'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__i...(truncated)) at ../Python/ceval.c:4166
#47 _PyEval_EvalCodeWithName.lto_priv.1761 () at ../Python/ceval.c:4166
#48 0x0000000000502540 in fast_function.lto_priv () at ../Python/ceval.c:4978
#49 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#50 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#51 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446e5dfcf8, for file <frozen importlib._bootstrap_external>, line 678, in exec_module (self=<SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, module=<module at remote 0x7f446ce5b318>, code=<code at remote 0x7f446ce3fc90>))
    at ../Python/ceval.c:754
#52 _PyFunction_FastCall (globals=<optimized out>, nargs=139931886157048, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#53 fast_function.lto_priv () at ../Python/ceval.c:4954
#54 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#55 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#56 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446ced2448, for file <frozen importlib._bootstrap>, line 665, in _load_unlocked (spec=<ModuleSpec(name='pyarrow', loader=<SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/
#56 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446ced2448, for file <frozen importlib._bootstrap>, line 665, in _load_unlocked (spec=<ModuleSpec(name='pyarrow', loader=<SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/---Type <return> to continue, or q <return> to quit---
__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, module=<module at remote 0x7f446ce5b318>))
    at ../Python/ceval.c:754
#57 _PyFunction_FastCall (globals=<optimized out>, nargs=139931861984328, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#58 fast_function.lto_priv () at ../Python/ceval.c:4954
#59 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#60 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#61 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0xb78328, for file <frozen importlib._bootstrap>, line 955, in _find_and_load_unlocked (name='pyarrow', import_=<built-in method __import__ of module object at remote 0x7f446e704638>, path=None, parent='', spec=<ModuleSpec(name='pyarrow', loader=<SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>)) at ../Python/ceval.c:754
#62 _PyFunction_FastCall (globals=<optimized out>, nargs=12026664, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#63 fast_function.lto_priv () at ../Python/ceval.c:4954
#64 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#65 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#66 0x0000000000501945 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0xb1edb8, for file <frozen importlib._bootstrap>, line 971, in _find_and_load (name='pyarrow', import_=<built-in method __import__ of module object at remote 0x7f446e704638>, module=<object at remote 0x7f446e6c6030>)) at ../Python/ceval.c:754
#67 _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#68 _PyFunction_FastCallDict () at ../Python/ceval.c:5021
#69 0x00000000005a36f1 in _PyObject_FastCallDict () at ../Objects/abstract.c:2310
#70 0x000000000059662e in _PyObject_CallMethodIdObjArgs () at ../Objects/abstract.c:2796
#71 0x00000000004ee84d in PyImport_ImportModuleLevelObject () at ../Python/import.c:1578
#72 0x000000000050896c in import_name (level=0, fromlist=None, name='pyarrow', f=Frame 0x7f446e5df788, for file <stdin>, line 1, in <module> ())
    at ../Python/ceval.c:5231
#73 _PyEval_EvalFrameDefault () at ../Python/ceval.c:2899
#74 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f446e5df788, for file <stdin>, line 1, in <module> ()) at ../Python/ceval.c:4166
#75 _PyEval_EvalCodeWithName.lto_priv.1761 () at ../Python/ceval.c:4166
#76 0x0000000000506393 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<optimized out>,
    globals=<optimized out>, _co=<optimized out>) at ../Python/ceval.c:4187
#77 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:731
#78 0x0000000000634d52 in run_mod () at ../Python/pythonrun.c:1025
#79 0x00000000004a38c5 in PyRun_InteractiveOneObjectEx (fp=fp@entry=0x7f446e50aa00 <_IO_2_1_stdin_>, filename=filename@entry='<stdin>',
    flags=flags@entry=0x7ffc89f95e8c) at ../Python/pythonrun.c:246
#80 0x00000000004a5cd5 in PyRun_InteractiveLoopFlags (fp=fp@entry=0x7f446e50aa00 <_IO_2_1_stdin_>, filename_str=filename_str@entry=0x7541e0 "<stdin>",
    flags=flags@entry=0x7ffc89f95e8c) at ../Python/pythonrun.c:114
#81 0x00000000006387b3 in PyRun_AnyFileExFlags () at ../Python/pythonrun.c:75
#82 0x000000000063915a in run_file (p_cf=0x7ffc89f95e8c, filename=<optimized out>, fp=<optimized out>) at ../Modules/main.c:340
#83 Py_Main () at ../Modules/main.c:810
#84 0x00000000004a6f10 in main (argc=1, argv=0x7ffc89f96088) at ../Programs/python.c:69

asfimport · 2019-01-16T09:54:07Z

Antoine Pitrou / @pitrou:
Ah, you're using Tensorflow. I take it you installed both Tensorflow and PyArrow using pip?

Tensorflow publishes incompatible Python packages (wheels) that can make other libraries such as Arrow crash (#). I'm not 100% sure it's the problem here but that is quite likely.

Can you try using conda instead and install both libraries using conda?

Also, I think it would be helpful if you reported this problem to the Tensorflow project, due to the issue I mentioned above.

cc @pcmoritz

(#) More explanation here: https://groups.google.com/a/tensorflow.org/forum/#!topic/developers/TMqRaT-H2bI

asfimport · 2019-01-16T19:17:59Z

Elchin:
@pitrou of course, with conda it works fine.

abcbarryn · 2023-02-20T14:55:11Z

I have this same issue and I do not have tensorflow installed.
Steps to reproduce:

pip install pyarrow

$ python
import pyarrow
illegal instruction

CPU and system information:

vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5160  @ 3.00GHz
stepping	: 6
microcode	: 0xd2
cpu MHz		: 3000.116
cache size	: 4096 KB

Linux static1 3.16.7-48-desktop #1 SMP PREEMPT Wed Oct 26 18:09:22 UTC 2016 (8c027f6) x86_64 GNU/Linux

glibc-2.18-4.44.1.x86_64

OpenSuSE linux

kou · 2023-02-20T21:10:52Z

Could you open a new issue for your case with your Python version and full log of pip install pyarrow?

abcbarryn · 2023-02-20T22:36:42Z

Python versions tried: 3.9.4, 3.9.16, 3.9.13, 3.11
pip install pyarrow doesn't throw any errors, how do I view the log? It builds and install cleanly, but when I try to use it I get illegal instruction. I will open a new case shortly.

lahwaacz · 2024-05-08T08:30:32Z

It appears that pyarrow requires a CPU with SSE 4.1 due to the PINSRQ instruction:

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install pyarrow
Collecting pyarrow
  Using cached pyarrow-16.0.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Collecting numpy>=1.16.6 (from pyarrow)
  Using cached numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Using cached pyarrow-16.0.0-cp312-cp312-manylinux_2_28_x86_64.whl (40.8 MB)
Using cached numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.0 MB)
Installing collected packages: numpy, pyarrow
Successfully installed numpy-1.26.4 pyarrow-16.0.0
$ objdump -S .venv/lib/python3.12/site-packages/pyarrow/lib.cpython-312-x86_64-linux-gnu.so | grep pinsrq
   6e8b2:	66 48 0f 3a 22 c2 01 	pinsrq $0x1,%rdx,%xmm0
   6e8cd:	66 48 0f 3a 22 c2 01 	pinsrq $0x1,%rdx,%xmm0
   6e8eb:	66 48 0f 3a 22 c2 01 	pinsrq $0x1,%rdx,%xmm0
   6e90d:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
   6e93a:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
   6e9df:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
   6e9fd:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
   6ea1b:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
...

The first PINSRQ instruction gets called right at the module import.

HanabishiRecca · 2024-05-08T12:10:54Z

https://arrow.apache.org/docs/cpp/env_vars.html#envvar-ARROW_USER_SIMD_LEVEL

by default, SSE4.2 optimizations are enabled on x86 builds: therefore, with this default setting, Arrow C++ does not work at all on a CPU without support for SSE4.2. This setting can be changed using the ARROW_SIMD_LEVEL CMake variable so as to either raise or lower the optimization level.

asfimport closed this as completed Feb 8, 2019

levicki mentioned this issue Sep 7, 2023

python3 server.py --chat Ilegal instruction ('core' generated) oobabooga/text-generation-webui#2163

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Python] Illegal hardware instruction on pyarrow import #20848

[Python] Illegal hardware instruction on pyarrow import #20848

asfimport commented Jan 16, 2019

asfimport commented Jan 16, 2019

asfimport commented Jan 16, 2019

asfimport commented Jan 16, 2019

asfimport commented Jan 16, 2019

abcbarryn commented Feb 20, 2023 •

edited by kou

Loading

kou commented Feb 20, 2023

abcbarryn commented Feb 20, 2023

lahwaacz commented May 8, 2024

HanabishiRecca commented May 8, 2024

[Python] Illegal hardware instruction on pyarrow import #20848

[Python] Illegal hardware instruction on pyarrow import #20848

Comments

asfimport commented Jan 16, 2019

Original Issue Attachments:

asfimport commented Jan 16, 2019

asfimport commented Jan 16, 2019

asfimport commented Jan 16, 2019

asfimport commented Jan 16, 2019

abcbarryn commented Feb 20, 2023 • edited by kou Loading

kou commented Feb 20, 2023

abcbarryn commented Feb 20, 2023

lahwaacz commented May 8, 2024

HanabishiRecca commented May 8, 2024

abcbarryn commented Feb 20, 2023 •

edited by kou

Loading