libassistant_embedder.so segfaults with glibc 2.29

[ofuangka]

Apr 24 19:25:41 puter kernel: googlesamples-a[3843]: segfault at 57a38b90 ip 00007fab5a6c0e3a sp 00007ffea35e9c18 error 4 >
Apr 24 19:25:41 puter kernel: Code: f3 0f 1e fa 66 0f ef c0 66 0f ef c9 66 0f ef d2 66 0f ef db 48 89 f8 48 89 f9 48 81 e1>
Apr 24 19:25:41 puter kernel: audit: type=1701 audit(1556151941.138:209): auid=4294967295 uid=973 gid=973 ses=4294967295 p>
Apr 24 19:25:41 puter systemd[1]: Started Process Core Dump (PID 3847/UID 0).
Apr 24 19:25:41 puter audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@18-384>
Apr 24 19:25:41 puter kernel: audit: type=1130 audit(1556151941.148:210): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='>
Apr 24 19:25:41 puter systemd-coredump[3848]: Removed old coredump core.googlesamples-a.973.a6d463f3f2f54c5ea22af9f6991d53>
Apr 24 19:25:41 puter systemd[1]: googleassistant.service: Main process exited, code=dumped, status=11/SEGV
Apr 24 19:25:41 puter systemd[1]: googleassistant.service: Failed with result 'core-dump'.
Apr 24 19:25:41 puter audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=googleassistant comm="sy>
Apr 24 19:25:41 puter kernel: audit: type=1131 audit(1556151941.758:211): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='>
Apr 24 19:25:41 puter systemd-coredump[3848]: Process 3843 (googlesamples-a) of user 973 dumped core.
                                                  
                                                  Stack trace of thread 3843:
                                                  #0  0x00007fab5a6c0e3a __GI___strlen_sse2 (libc.so.6)
                                                  #1  0x00007fab5734d54d n/a (/var/lib/googleassistant/.venv/lib/python3.7/sit>
                                                  #2  0x00007fab57349253 n/a (/var/lib/googleassistant/.venv/lib/python3.7/sit>
                                                  #3  0x00007fab59b246d0 ffi_call_unix64 (libffi.so.6)
                                                  #4  0x00007fab59b240a0 ffi_call (libffi.so.6)
                                                  #5  0x00007fab59d25625 _ctypes_callproc (_ctypes.cpython-37m-x86_64-linux-gn>
                                                  #6  0x00007fab59d25fb0 n/a (_ctypes.cpython-37m-x86_64-linux-gnu.so)
                                                  #7  0x00007fab5a43162c _PyObject_FastCallKeywords (libpython3.7m.so.1.0)
                                                  #8  0x00007fab5a4760db _PyEval_EvalFrameDefault (libpython3.7m.so.1.0)
                                                  #9  0x00007fab5a3bedbb _PyFunction_FastCallDict (libpython3.7m.so.1.0)
                                                  #10 0x00007fab5a3ce818 _PyObject_Call_Prepend (libpython3.7m.so.1.0)
                                                  #11 0x00007fab5a41d0e3 n/a (libpython3.7m.so.1.0)
                                                  #12 0x00007fab5a43139c _PyObject_FastCallKeywords (libpython3.7m.so.1.0)
                                                  #13 0x00007fab5a475987 _PyEval_EvalFrameDefault (libpython3.7m.so.1.0)
                                                  #14 0x00007fab5a4046db _PyFunction_FastCallKeywords (libpython3.7m.so.1.0)
                                                  #15 0x00007fab5a47122d _PyEval_EvalFrameDefault (libpython3.7m.so.1.0)
                                                  #16 0x00007fab5a3bdd09 _PyEval_EvalCodeWithName (libpython3.7m.so.1.0)
                                                  #17 0x00007fab5a3bec64 PyEval_EvalCodeEx (libpython3.7m.so.1.0)
                                                  #18 0x00007fab5a3bec8c PyEval_EvalCode (libpython3.7m.so.1.0)
                                                  #19 0x00007fab5a4e7694 n/a (libpython3.7m.so.1.0)
                                                  #20 0x00007fab5a4e8b6e PyRun_FileExFlags (libpython3.7m.so.1.0)
                                                  #21 0x00007fab5a4ec035 PyRun_SimpleFileExFlags (libpython3.7m.so.1.0)
                                                  #22 0x00007fab5a4ee2a7 n/a (libpython3.7m.so.1.0)
                                                  #23 0x00007fab5a4ee4ec _Py_UnixMain (libpython3.7m.so.1.0)
                                                  #24 0x00007fab5a64bce3 __libc_start_main (libc.so.6)
                                                  #25 0x0000560ccadaf05e _start (python3.7)

This is using ArchLinux glibc-2.29-1. Reverting to glibc-2.28-6 resolves the issue, but causes problems elsewhere.

Seems like it could be related to this: https://www.phoronix.com/scan.php?page=news_item&px=GCC-MMX-Intrinsics-With-SSE

If I get time I'll try to figure out how to run in 32-bit mode.

[ofuangka]

It seemed like a pain to compile all the necessary libraries for 32-bit, but I was able to work around this by recompiling glibc 2.29 and turning off stack protection. So it seems like it could be a memory allocation issue.

[polyc]

I was able to work around this by recompiling glibc 2.29 and turning off stack protection
Could you point me to a good resource that explains how to compile and install glibc without stack protection?

[ofuangka]

@polyc I'm using ArchLinux, so I cloned the glibc-git project from AUR and modified the PKGBUILD:

git clone https://aur.archlinux.org/glibc-git
cd glibc-git
sed -i "s/enable-stack-protector=strong/enable-stack-protector=no/" PKGBUILD
makepkg -si

The relevant line changes to --enable-stack-protector=no. Once you've made that change, you can run makepkg -si from the directory that PKGBUILD is in. More information can (and should) be found at https://wiki.archlinux.org/index.php/Arch_Build_System

Note that this is an incredibly dangerous workaround--my admittedly limited understanding is that the stack protector is normally enabled to prevent stack overflow bugs or even malicious attacks, and disabling it like this would affect anything that uses glibc, not just Google Assistant. The real fix needs to be done within libassistant_embedder.so, which is not open source and probably won't be for a long time, if ever. If Google ever acknowledges and fixes this issue, it would be best to revert back to the stable version of glibc.

[polyc]

@ofuangka thank you very much for your help. I know that could be very dangerous but i only need to try it, certainly not for production purposes.

[jrgleason]

How would this work for Raspberry pi? Asking for the above question