asan: triggered deadley signal for clang-14.0.6 built application

[koteswararao18]

passed “-fsanitize= address” to clang-14.0.6 and built an application and tried to launch it but it failed with the below error.
{{
ASAN_OPTIONS=‘stack_trace_format=“[frame=%n, function=%f, location=%S]”’ proxy

AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.
}}
Tried to gdb it but failed with SIGSEGV immediately.
{{
(gdb) r
Starting program: /bin/proxy
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib64/libthread_db.so.1”.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7f090f3 in ?? () from /lib64/libjemalloc.so.2
gdb) bt
#0 0x00007ffff7f090f3 in ?? () from /lib64/libjemalloc.so.2
#1 0x00007ffff7f097d8 in ?? () from /lib64/libjemalloc.so.2
#2 0x00007ffff7f0af69 in ?? () from /lib64/libjemalloc.so.2
#3 0x00007ffff7eaf7ca in sdallocx () from /lib64/libjemalloc.so.2
#4 0x00007ffff2fc0677 in std::__new_allocator<std::__detail::_State >::deallocate(std::__detail::_State, unsigned long) () from /lib64/libjsonschema.so
#5 0x00007ffff2fbb41e in std::allocator_traits<std::allocator<std::__detail::_State > >::deallocate(std::allocator<std::__detail::_State >&, std::__detail::_State, unsigned long) ()
from /lib64/libjsonschema.so
#6 0x00007ffff2fb7b1e in std::_Vector_base<std::__detail::_State, std::allocator<std::__detail::_State > >::_M_deallocate(std::__detail::_State, unsigned long) () from /lib64/libjsonschema.so
#7 0x00007ffff2fb48bb in void std::vector<std::__detail::_State, std::allocator<std::__detail::_State > >::_M_realloc_insert<std::__detail::_State >(__gnu_cxx::__normal_iterator<std::__detail::_State, std::vector<std::__detail::_State, std::allocator<std::__detail::_State > > >, std::__detail::_State&&) () from /lib64/libjsonschema.so
#8 0x00007ffff2fb22dc in std::__detail::_State& std::vector<std::__detail::_State, std::allocator<std::__detail::_State > >::emplace_back<std::__detail::_State >(std::__detail::_State&&) ()
from /lib64/libjsonschema.so
#9 0x00007ffff2fb071e in std::vector<std::__detail::_State, std::allocator<std::__detail::_State > >::push_back(std::__detail::_State&&) () from /lib64/libjsonschema.so
#10 0x00007ffff2fae036 in std::__detail::_NFA<std::__cxx11::regex_traits >::_M_insert_state(std::__detail::_State) () from /lib64/libjsonschema.so
#11 0x00007ffff2fb49b2 in std::__detail::_NFA<std::__cxx11::regex_traits >::_M_insert_line_begin() ()
--Type for more, q to quit, c to continue without paging-- c
from /lib64/libjsonschema.so
#12 0x00007ffff2fb2364 in std::__detail::_Compiler<std::__cxx11::regex_traits >::_M_assertion() () from /lib64/libjsonschema.so
#13 0x00007ffff2fb0804 in std::__detail::_Compiler<std::__cxx11::regex_traits >::_M_term() () from /lib64/libjsonschema.so
#14 0x00007ffff2fae188 in std::__detail::_Compiler<std::__cxx11::regex_traits >::_M_alternative() () from /lib64/libjsonschema.so
#15 0x00007ffff2faa92e in std::__detail::_Compiler<std::__cxx11::regex_traits >::_M_disjunction() () from /lib64/libjsonschema.so
#16 0x00007ffff2fa74ad in std::__detail::_Compiler<std::__cxx11::regex_traits >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type) () from /lib64/libjsonschema.so
#17 0x00007ffff2fa1c4e in std::__cxx11::basic_regex<char, std::__cxx11::regex_traits >::_M_compile(char const*, char const*, std::regex_constants::syntax_option_type) () from /lib64/libjsonschema.so
#18 0x00007ffff229d089 in std::__cxx11::basic_regex<char, std::__cxx11::regex_traits >::basic_regex<std::char_traits, std::allocator >(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::regex_constants::syntax_option_type) () from /lib64/libjsonschema2021.so
#19 0x00007ffff228facb in ?? () from /lib64/libjsonschema2021.so
#20 0x00007ffff228fbbd in ?? () from /lib64/libjsonschema2021.so
#21 0x00007ffff7fd2f9a in ?? () from /lib64/ld-linux-x86-64.so.2
#22 0x00007ffff7fd3089 in ?? () from /lib64/ld-linux-x86-64.so.2
#23 0x00007ffff7fe4faa in ?? () from /lib64/ld-linux-x86-64.so.2
#24 0x0000000000000001 in ?? ()
#25 0x00007fffffffef88 in ?? ()
#26 0x0000000000000000 in ?? ()

}}
it seems address sanitizer library is not started completely and application trying to use sanitizer for address validation is causing this crash? Above crash is observed on application built on clang-14.0.6 with glibc updated to 2.35 in toolchain. Prior to it, 2.31 glibc included clang-14.0.6 version not reported above issue.
Below issue discussed about it long ago and proposed patch as well for older clang version which can't be adopted now for clang-14.0.6 version.
#858
any insights for further debugging, please let me know?

rgds
koti

[Enna1]

Hi, asan is not compatible with jemalloc. when enable asan, please do not link or preload jemalloc.

[koteswararao18]

Dear enna1,

asan is not compatible with jemalloc. when enable asan, please do not link or preload jemalloc.

If application compiled with glibc-2.31 and clang-14.0.6 by passing "fsanitize=address" (asan option) by loading and linking with jemalloc, then issue is not observed .
Similarly, if compiled application using gcc-12.1.0 by preloading and linking with "-ljemalloc" , along with asan options then issue is not observed.
Issue is observed with clang built application only using glibc-2.35 and clang-14.0.6 with asan options.

Rgds
Koti

[Enna1]

Can you check the needed library of your application?

• objdump -p your-application | grep NEEDED
• ldd your-application

[koteswararao18]

Dear Enna
i updated jemalloc library from 5.0.1 to 5.3.0 version and used libjemalloc.so.2 built in 5.3.0 , with it, the application is loaded successfully through gdb. DEADLEY signal and gdb running proxy process is successful.
but still observed proxy application launching failed and repeating crash of application . I noticed even main() of proxy application too is not triggered . This is observed only if '-fsanitize=address' is passed during proxy application build using clang-14.0.6 with glibc-2.35 library. Attached needed.txt library for application for reference.

Rgds
koti

[koteswararao18]

Hi please find the attached needed library for application. rgds koti

On Mon, 7 Aug 2023 at 15:30, Enna1 ***@***.***> wrote: Can you check the needed library of your application? - objdump -p your-application | grep NEEDED - ldd your-application — Reply to this email directly, view it on GitHub <#1673 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHEWYK7A5F67ACINOKFR2Q3XUC4DVANCNFSM6AAAAAA3FYKO2E> . You are receiving this because you authored the thread.Message ID: ***@***.***>

/bin/x86_64-linux-gnu-objdump -p proxyd | grep NEEDED NEEDED libunwind.so.8 NEEDED libunwind-x86_64.so.8 NEEDED libjemalloc.so.2 NEEDED libcmf.so NEEDED libcmfquery.so NEEDED libfcrypto.so NEEDED libfdisk.so NEEDED libpthread.so.0 NEEDED libprotutil.so NEEDED libnetutil.so NEEDED libstdext.so NEEDED libsandbox_client.so NEEDED liboffstack.so NEEDED libfgtutil.so NEEDED libavdb.so NEEDED libwafutil.so NEEDED libsgml.so NEEDED libauth.so NEEDED libkrb_auth.so NEEDED libsaml_auth.so NEEDED libconf_op.so NEEDED libncfg.so NEEDED libpolicy.so NEEDED libxml2.so.2 NEEDED libwhitelist.so NEEDED libcaptcha.so NEEDED liblogbase.so NEEDED libunicode.so NEEDED liblangutil.so NEEDED libflgrptout.so NEEDED libpng16.so.16 NEEDED libgd.so.3 NEEDED libfortichart.so NEEDED lib_signaturedb.so NEEDED libbonet.so NEEDED libfpoll.so NEEDED libmulpat.so NEEDED libfortiprotect.so NEEDED libsqlite.so NEEDED libsqlite3.so.0 NEEDED libknownenginesxml.so NEEDED libmaxminddb.so NEEDED libcmfcore.so NEEDED libsysutil.so NEEDED libconfilter.so NEEDED libcmdb_plugin.so NEEDED libwassd.so NEEDED libpcap.so.1 NEEDED libpcre.so.1 NEEDED libm.so.6 NEEDED libz.so.1 NEEDED libdl.so.2 NEEDED libgssapi_krb5.so.2 NEEDED libfipsext.so NEEDED libfips.so NEEDED librnd.so NEEDED libcommoncriteria.so NEEDED libhealth_chk.so NEEDED libl7sync.so NEEDED libdns.so NEEDED libcurl.so.4 NEEDED libfjson.so NEEDED libjson.so NEEDED libebtree.so NEEDED libdpdkport.so NEEDED libslz.so NEEDED libsqlast.so NEEDED libupdatedb.so NEEDED libcert.so NEEDED libuuid.so.1 NEEDED libmariadb.so.3 NEEDED libhwssl.so NEEDED libcmime.so.0.2 NEEDED libwbxml2.so.0 NEEDED liburcu.so NEEDED liburcu-cds.so NEEDED lib_malloc.so NEEDED libirdb_cli.so NEEDED libbrotlicommon.so.1.0.1 NEEDED libbrotlidec.so.1.0.1 NEEDED libbrotlienc.so.1.0.1 NEEDED libml_common_db.so NEEDED libmathmodel.so NEEDED libml_event.so NEEDED libml_gui_op.so NEEDED libml_daemon_op.so NEEDED libiconv.so.2 NEEDED libsvm.so NEEDED libstdc++.so.6 NEEDED libenvinit.so NEEDED libhs.so.4 NEEDED libhs_runtime.so.4 NEEDED libxmlsoap.so NEEDED libjsonschema.so NEEDED libxmlsec1.so.1 NEEDED libml_robot_redis.so NEEDED libml_robot_svm_redis.so NEEDED libfuzzy.so.2 NEEDED libadfscomm.so NEEDED libadfsproxyclient.so NEEDED libmiggui.so NEEDED libhiredis.so.1.0.0 NEEDED libhiredis_ssl.so.1.0.0 NEEDED libml_common_redis.so NEEDED libcrypto.so.1.1 NEEDED libjwt.so.0 NEEDED libicapclient.so NEEDED libsysha.so NEEDED libcloudutil.so NEEDED libhcr.so NEEDED liburlenc.so NEEDED libpagespeed.so NEEDED libarchive.so.13 NEEDED liblzma.so.5 NEEDED libxssdetection.so NEEDED libwaf_redis.so NEEDED libsys_confd_sync.so NEEDED libacme.so NEEDED libgmime-3.0.so.0 NEEDED libgio-2.0.so.0 NEEDED libgmodule-2.0.so.0 NEEDED libgobject-2.0.so.0 NEEDED libglib-2.0.so.0 NEEDED libgthread-2.0.so.0 NEEDED libffi.so.7 NEEDED libjansson.so.4 NEEDED libftime.so NEEDED libcjson.so.1 NEEDED libjsonschema2021.so NEEDED libfgd_query.so NEEDED libhlck.so NEEDED libec.so NEEDED libmagic.so.1 NEEDED libclustering.so NEEDED libcares.so.2 NEEDED libpython3.10.so.1.0 NEEDED libmlapi_category.so NEEDED libcjose.so.0 NEEDED libssl.so.1.1 NEEDED libfgbk_cipher.so NEEDED libc.so.6 NEEDED ld-linux-x86-64.so.2

[koteswararao18]

Errors mentioned in previous comments are fixed after using the latest built libjemalloc version .
But still the application is failing to start successfully and glibc-2.35 libc.so.6 is not triggering the main() function of the application which means there is issue in glibc-2.35 libc_start_main() function as it is responsible for triggering main() of the application.
it seems clang-14.0.6 + fsanitize=address + glibc-2.35 has some incompatibility leading this failure in calling application main() . Any insights plesae let me know..

[koteswararao18]

hi,
i noticed example application built by passing fsanitize=address opitons to clang-14 version compiler is failing with below error. this is observed if glibc>=2.34 version is present in toolchain .
{{
clang-14.0.6 leak.c -o leak -fsanitize=address -O1 -fno-omit-frame-pointer -g

./leak

==3812418==ERROR: AddressSanitizer failed to allocate 0x0 (0) bytes of SetAlternateSignalStack (error code: 22)
==3812418==Process memory map follows:
0x000000400000-0x00000041f000 /leak
}}
But in clang compiler-rt/ , below patch is proposed as fix for above error, but still noticed issue as clang-14 contains below fix.
{{
commit 82150606fb11d28813ae6da1101f5bda638165fe
Author: Vitaly Buka [email protected]
Date: Fri Apr 16 09:50:24 2021 -0700

Sanitizer built against glibc 2.34 doesn't work

As mentioned in https://gcc.gnu.org/PR100114 , glibc starting with the
https://sourceware.org/git/?p=glibc.git;a=commit;h=6c57d320484988e87e446e2e60ce42816bf51d53
change doesn't define SIGSTKSZ and MINSIGSTKSZ macros to constants, but to sysconf function call.
sanitizer_posix_libcdep.cpp has
static const uptr kAltStackSize = SIGSTKSZ * 4;  // SIGSTKSZ is not enough.
which is generally fine, just means that when SIGSTKSZ is not a compile time constant will be initialized later.
The problem is that kAltStackSize is used in SetAlternateSignalStack which is called very early, from .preinit_array
initialization, i.e. far before file scope variables are constructed, which means it is not initialized and
mmapping 0 will fail:
==145==ERROR: AddressSanitizer failed to allocate 0x0 (0) bytes of SetAlternateSignalStack (error code: 22)

Here is one possible fix, another one could be to make kAltStackSize a preprocessor macro if _SG_SIGSTKSZ is defined
(but perhaps with having an automatic const variable initialized to it so that sysconf isn't at least called twice
during SetAlternateSignalStack.

Reviewed By: vitalybuka

Differential Revision: https://reviews.llvm.org/D100645

}}
Any insights in to the issue?

[koteswararao18]

update jemallocversion.