Memory allocated by ASAN is freed by glibc

[dang]

I have a case where memory allocated by ASAN (verified by stepping through in the debugger) is freed by glibc (again, verified by stepping through in the debugger). This trips glibc's invalid pointer check in munmap_chunk(), causing the program to abort. It seems similar to issue #70 but is on Fedora 24, not on OSX. What can I do to debug and (hopefully) fix this, as it's blocking my continued use of ASAN.

I have:
Fedora 24
gcc: 6.3.1-1.fc24
glibc: 2.23.1-11.fc24
libasan: 6.3.1-1.fc24

The code in question is OSS (it's in-progress work for nfs-ganesha) but not posted anywhere yet. This code runs correctly under valgrind. It's built with -O0 -g -gdwarf-2 -fsanitize=address -fno-omit-frame-pointer Note that other branches of nfs-ganesha run fine with ASAN, so something is different.

[kcc]

GCC uses dynamic asan library by default, and this is more prone to problems like this.
Please either try clang (which uses static linking by default) or try gcc with -static-libasan to check if the
dynamic linking is at guilt here.

[yugr]

@dang if this is GCC-specific I'd be curious to see symbolized backtraces.

[dang]

Okay, I've done some testing, and it still happens with -static-libasan and with clang. Clang version is 3.8.1-1.fc24. Below is my backtrace. I'm going to try again on a fedora 25 box (so newer everything) and see if it hits there.

#0 0x00007ffff526a765 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff526c36a in __GI_abort () at abort.c:89
#2 0x00007ffff52ab710 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff53c2200 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff52b7568 in malloc_printerr (ar_ptr=0x0, ptr=, str=0x7ffff53c2228 "munmap_chunk(): invalid pointer", action=) at malloc.c:5004
#4 munmap_chunk (p=) at malloc.c:2840
#5 GI___libc_free (mem=) at malloc.c:2961
#6 0x00007fffefdcb640 in gsh_free (p=0x60600000e960, file=0x7fffefdcc960 "/home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/FSAL/FSAL_MEM/memfs_export.c", line=79, function=0x7fffefdcca38 <func.21917> "memfs_release") at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/include/abstract_mem.h:274
#7 0x00007fffefdcb76d in memfs_release (exp_hdl=0xa4b010) at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/FSAL/FSAL_MEM/memfs_export.c:79
#8 0x000000000066b6b5 in mdcache_exp_release (exp_hdl=0x616000068480) at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_export.c:180
#9 0x000000000062194e in free_export_resources (export=0x61500000fd48) at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/support/exports.c:2087
#10 0x0000000000642db3 in free_export (export=0x61500000fd48) at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/support/export_mgr.c:252
#11 0x0000000000645203 in put_gsh_export (export=0x61500000fd48) at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/support/export_mgr.c:632
#12 0x0000000000645f65 in remove_all_exports () at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/support/export_mgr.c:762
#13 0x000000000049e5c7 in do_shutdown () at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/MainNFSD/nfs_admin_thread.c:433
#14 0x000000000049eb62 in admin_thread (UnusedArg=0x0) at /home/dang/src/ganesha/next/mdcache/nfs-ganesha/src/MainNFSD/nfs_admin_thread.c:466
#15 0x00007ffff5c6e5ca in start_thread (arg=0x7fff65a71700) at pthread_create.c:333
#16 0x00007ffff53390ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[chefmax]

Could you share the code of gsh_free function? Does it call free regularly, or maybe through pointer?

[dang]

https://github.com/dang/nfs-ganesha/blob/FSAL_MEM/src/include/abstract_mem.h#L271

So it just prints and calls free()

[yugr]

Is the failing library dlopened in https://github.com/dang/nfs-ganesha/blob/FSAL_MEM/src/FSAL/fsal_manager.c#L230 ? I think we had problems with RTLD_DEEPBIND in the past (see #611) but I'm not sure about the reasons.

[dang]

Yes, the failed lib is dlopened. Each FSAL is dlopened when it's needed. However, I run with FSAL_VFS (the FSAL that uses a local filesystem, which is also dlopened) with ASAN all the time, and have not seen this issue with it.

That said, issue #611 appears to be my exact issue. I'll try without DEEPBIND and report back.

[chefmax]

but I'm not sure about the reasons.

Perhaps because GCC does not add DT_NEEDED for libasan to shared libraries? Thus RTLD searches in Glibc first and uses __libc_free.

[dang]

Yep, that did it. I'll modify our build to disable DEEPBIND when using ASAN. Is this something that is likely to get fixed in the future?

[chefmax]

Is this something that is likely to get fixed in the future?

Nope, but we can probably complain and exit in this case (see @kcc first comment in #611). For now we can mention this issue in FAQ though.

[yugr]

we can probably complain and exit in this case

I think we should, people keep hitting this again and again.

[yugr]

@dang Could you resolve this as dup?

[dang]

Sure.

[dang]

Duplicate of #166

[Bu11etmagnet]

I think you mean #611

[dang]

Yes, sorry, #611