Segmentation fault with -thread_privat option

[Desperado1985]

Segmentation fault running dynamorio / drcov on an Aarch64 Yocto Linux when -thread_private option is enabled.

DynamoRIO-AArch64-Linux-8.0.0-1

/media/mp000/DynamoRIO-AArch64-Linux-8.0.0-1/bin64/drrun -root /media/mp000/DynamoRIO-AArch64-Linux-8.0.0-1 -thread_private -t drcov -logdir /media/mp000/ -dump_binary -- /bin/ls
<Application /bin/busybox (4023). DrCov internal crash at PC 0x00000000710c6694. Please report this at http://dynamorio.org/issues. Program aborted.
Received SIGSEGV at pc 0x00000000710c6694 in thread 4023
Base: 0x0000000071000000
Registers: eflags=0x0000000040000000
version 8.0.0, build 1
-no_dynamic_options -client_lib '/media/mp000/DynamoRIO-AArch64-Linux-8.0.0-1/tools/lib64/release/libdrcov.so;0;"-logdir" "/media/mp000/" "-dump_binary"' -code_api -stack_size 56K -signal_stack_size 32K -nop_initial_bblock -max_elide_jmp 0 -max_elide_call 0 -thread_private -no_atomic_inlined_linking -inline_trace_ibl -no
0x0000007f2f6a3950 0x00000000710b5eb0
0x0000007f2f6a39a0 0x00000000710b8340
0x0000007f2f6a3a20 0x00000000710545e4
0x0000007f2f6a3b80 0x000000007108df70
0x0000007f2f6a3bb0 0x000000007104f850
0x0000007f2f6a3bd0 0x000000007104176c
0x0000007f2f6a3c40 0x00000000710593c0
0x0000007f2f6a3cb0 0x000000007110232c
0x0000007f2f6a3d70 0x0000000071057668
0x0000007f2f6a3f20 0x00000000711102cc
0x0000007fc6807480 0x00000000711107e8>

Testapp is standard /bin/ls.

[derekbruening]

I'm not sure -thread_private was ever implemented for AArch64? It was not for ARM: #1884. If AArch64 is the same, this can be marked as a dup of #1884.

We'd be happy to look at pull requests contributing the support, or at least a friendly failure message instead of a crash -- or is that already there in -debug?

[derekbruening]

@AssadHashmi -- is -thread_private support implemented for AArch64?

[AssadHashmi]

@AssadHashmi -- is -thread_private support implemented for AArch64?

No

Duplicate of #1884

[Desperado1985]

It seems to be the same with -no_shared_bbs. Is that one supported on ARM/Aarch64?:

/media/mp000/DynamoRIO-AArch64-Linux-8.0.0-1/bin64/drrun -root /media/mp000/DynamoRIO-AArch64-Linux-8.0.0-1 -no_shared_bbs -t drcov -logdir /media/mp000/ -dump_binary -- /bin/ls
<Application /bin/busybox (2642). DrCov internal crash at PC 0x0000000031009498. Please report this at http://dynamorio.org/issues. Program aborted.
Received SIGSEGV at generated pc 0x0000000031009498 in thread 2642
Base: 0x0000000071000000
Registers: eflags=0x0000000080000000
version 8.0.0, build 1
-no_dynamic_options -client_lib '/media/mp000/DynamoRIO-AArch64-Linux-8.0.0-1/tools/lib64/release/libdrcov.so;0;"-logdir" "/media/mp000/" "-dump_binary"' -code_api -stack_size 56K -signal_stack_size 32K -nop_initial_bblock -max_elide_jmp 0 -max_elide_call 0 -no_shared_bbs -early_inject -emulate_brk -no_inline_ignored_sys
0x0000007fc3123240 0x0000007f8b4dc698
0x0000007fc3123280 0x0000007f8b4dbd08>
[root@vw-infotainment-917902:~]# /media/mp000/DynamoRIO-AArch64-Linux-8.0.0-1/bin64/drrun -root /media/mp000/DynamoRIO-AArch64-Linux-8.0.0-1 -t drcov -logdir /media/mp000/ -dump_binary -- /bin/ls

[Desperado1985]

Btw the reason why I care about this option is because there is a strong burden on this read write lock when adding fragments which slows my program down to almost a halt and I hoped this option could be a remedy:

The readers ( dozens ):
Thread 42 (LWP 2650):
#0 0x000000007112fd08 in dynamorio_syscall ()
#1 0x000000007105b2a8 in d_r_read_lock (rw=0x7f0fefe5ac) at /home/travis/build/DynamoRIO/dynamorio/core/utils.c:1185
#2 0x0000000071048730 in fragment_lookup_type (lookup_flags=15, tag=0x7f0c28431c "\342\003", dcontext=0x7f110859e0) at /home/travis/build/DynamoRIO/dynamorio/core/fragment.c:2738
#3 fragment_lookup (tag=0x7f0c28431c "\342\003", dcontext=0x7f110859e0) at /home/travis/build/DynamoRIO/dynamorio/core/fragment.c:2756
#4 fragment_lookup_fine_and_coarse (dcontext=dcontext@entry=0x7f110859e0, tag=0x7f0c28431c "\342\003", wrapper=wrapper@entry=0x7f10c44fc8, last_exit=0x7f10e09a48) at /home/travis/build/DynamoRIO/dynamorio/core/fragment.c:8109
#5 0x0000000071057d80 in dispatch_exit_fcache_stats (dcontext=0x7f110859e0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:1416
#6 dispatch_enter_dynamorio (dcontext=0x7f110859e0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:856
#7 d_r_dispatch (dcontext=0x7f110859e0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:164
#8 0x0000007f0c2842fc in ?? ()

The writer who slows everything down:

#0 0x000000007104257c in hashtable_fragment_add (dcontext=dcontext@entry=0x7f14cc9900, e=0x7f130e9e48, table=table@entry=0x7f0fefe568) at /home/travis/build/DynamoRIO/dynamorio/core/hashtablex.h:782
782 /home/travis/build/DynamoRIO/dynamorio/core/hashtablex.h: No such file or directory.
(gdb) bt
#0 0x000000007104257c in hashtable_fragment_add (dcontext=dcontext@entry=0x7f14cc9900, e=0x7f130e9e48, table=table@entry=0x7f0fefe568) at /home/travis/build/DynamoRIO/dynamorio/core/hashtablex.h:782
#1 0x00000000710423d0 in hashtable_fragment_add (table=0x7f0fefe568, e=, dcontext=0x7f14cc9900) at /home/travis/build/DynamoRIO/dynamorio/core/hashtablex.h:768
#2 hashtable_fragment_check_size (add_now=1, add_later=0, table=0x7f0fefe568, dcontext=0x7f14cc9900, dcontext@entry=0x7115a000 <reg_names+544>) at /home/travis/build/DynamoRIO/dynamorio/core/hashtablex.h:1102
#3 hashtable_fragment_add (dcontext=dcontext@entry=0x7f14cc9900, e=0x7f159b2860, table=table@entry=0x7f0fefe568) at /home/travis/build/DynamoRIO/dynamorio/core/hashtablex.h:780
#4 0x0000000071043148 in hashtable_fragment_add (table=0x7f0fefe568, e=, dcontext=0x7f14cc9900) at /home/travis/build/DynamoRIO/dynamorio/core/hashtablex.h:768
#5 fragment_add_to_hashtable (table=0x7f0fefe568, e=, dcontext=0x7f14cc9900) at /home/travis/build/DynamoRIO/dynamorio/core/fragment.c:872
#6 fragment_add (dcontext=0x7f14cc9900, f=) at /home/travis/build/DynamoRIO/dynamorio/core/fragment.c:2975
#7 0x0000000071059efc in emit_fragment_common (dcontext=dcontext@entry=0x7f14cc9900, tag=0x200000001 <error: Cannot access memory at address 0x200000001>, tag@entry=0x3028a30 "\202\n@\371C", ilist=0x7115a000 <reg_names+544>, flags=, vmlist=0x3028a30, link_fragment=false, link_fragment@entry=255, add_to_htable=false, add_to_htable@entry=true, replace_fragment=0x7f14cc9900, replace_fragment@entry=0x0) at /home/travis/build/DynamoRIO/dynamorio/core/emit.c:810
#8 0x0000000071059f48 in emit_fragment_ex (dcontext=dcontext@entry=0x7f14cc9900, tag=tag@entry=0x3028a30 "\202\n@\371C", ilist=, flags=, vmlist=, link=link@entry=255, visible=visible@entry=true) at /home/travis/build/DynamoRIO/dynamorio/core/emit.c:864
#9 0x000000007110232c in build_basic_block_fragment (dcontext=dcontext@entry=0x7f14cc9900, start=0x3028a30 "\202\n@\371C", initial_flags=initial_flags@entry=0, link=255, link@entry=true, visible=visible@entry=true, for_trace=for_trace@entry=false, unmangled_ilist=unmangled_ilist@entry=0x0) at /home/travis/build/DynamoRIO/dynamorio/core/arch/interp.c:5167
#10 0x0000000071057668 in d_r_dispatch (dcontext=0x7f14cc9900) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:214
#11 0x0000000003028a30 in ?? ()

[derekbruening]

It seems to be the same with -no_shared_bbs. Is that one supported on ARM/Aarch64?:

That would be the same as -thread_private (just with only bbs being private, and traces shared) so #1884.

If you could file the lock performance problem as a separate issue we can continue discussion of it there, and we'll close this in favor of #1884 for -thread_private. Scaling up AArch64 to bigger programs recently has revealed a number of issues, including with locks, some of which were fixed (#3956). Please include some performance numbers in the new issue and how many blocks are being created here over what kind of time span.