CRASH (1.4.0 libcbr.so) Internal DynamoRIO Error when running thread private cache on 64-bit Linux

[derekbruening]

From [email protected] on June 24, 2009 17:20:23

What steps will reproduce the problem? 1. build libcbr.so from the download packages samples/
2. run with thread private code cache option
qin_zhao@cagnode10:~/Workspace/DynamoRIO/DynamoRIO-Linux-1.4.0-20/build$
../bin64/drdeploy -client
~/Workspace/DynamoRIO/DynamoRIO-Linux-1.4.0-20/build/bin/libcbr.so 0 ""
-debug -ops -thread_private ls

What do you see?

<Starting application ls (6657)>
<Application ls (6657). Internal Error Internal DynamoRIO Error:
/work/dr/tot/opensource/core/x86/instr.c:4750
version 1.4.0, build 20
-client_lib
'/afs/csail.mit.edu/u/q/qin_zhao/Workspace/DynamoRIO/DynamoRIO-Linux-1.4.0-20/build/bin/libcbr.so;0;'
-code_api -max_elide_jmp 0 -max_elide_call 0 -no_shared_bbs -no_shared_trace
0x0000000048085440 0x000000007109fe23
0x0000000048085590 0x00000000711445f9
0x00000000480855f0 0x0000000071144789
0x0000000048085620 0x000000007114a698
0x0000000048085680 0x0000000071167b40
0x0000000048085820 0x000000007116feae
0x0000000048085860 0x000000007116ffd5
0x0000000048085a30 0x000000007200183e
0x0000000048085b50 0x00000000711735c7
0x0000000048085bf0 0x0000000071150cb2
0x0000000048085c80 0x0000000071158dc1>

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=162

[derekbruening]

From [email protected] on June 24, 2009 14:58:18

I examined the execute. The reason is cause from dr_insert_clean_call.
dr_insert_clean_call insert code perform several things including switch stack
pointer, save registers, and push parameters.

When insert clean call, on save/restore application stack, DR does not implement the
absolute address yet, (x86/instr.c:4764 IF_X64(ASSERT_NOT_IMPLEMENTED(!absolute));)

[derekbruening]

From [email protected] on June 24, 2009 15:49:07

It is interesting, I think the context switch code for dr_insert_clean_call should be
similar to the code on entering and exiting the code cache. Why it works there but
not for clean call.

[derekbruening]

From [email protected] on June 24, 2009 15:53:35

Can you paste in the DR call stack when the assert is triggered

[derekbruening]

From [email protected] on June 24, 2009 19:24:11

(gdb) bt
#0 external_error (
file=0x711abdd0 "/.../Workspace/DynamoRIO/dynamorio-dev/core/x86/encode.c",
line=1932, msg=0x711ac690 "instr_encode error: no encoding found")
at /.../Workspace/DynamoRIO/dynamorio-dev/core/utils.c:199
#1 0x0000000071132403 in instr_encode_common (dcontext=0x77846060, instr=0x7787b430,
pc=, check_reachable=1, assert_reachable=0)
at /.../Workspace/DynamoRIO/dynamorio-dev/core/x86/encode.c:1932
#2 0x0000000071132c19 in instr_encode_check_reachability (dcontext=0x711abdd0,
instr=0x78c,
pc=0x711ac690 "instr_encode error: no encoding found")
at /.../Workspace/DynamoRIO/dynamorio-dev/core/x86/encode.c:2237
#3 0x00000000711479ea in private_instr_encode (dcontext=0x77846060, instr=0x7787b430,
always_cache=0)
at /.../Workspace/DynamoRIO/dynamorio-dev/core/x86/instr.c:1792
#4 0x0000000071148d15 in instr_length (dcontext=0x77846060, instr=0x7787b430)
at /.../Workspace/DynamoRIO/dynamorio-dev/core/x86/instr.c:2716
#5 0x000000007109cb41 in emit_fragment_common (dcontext=0x77846060,
tag=0x2ab2f8c50b0b "H\213\223\b\001", ilist=0x77866df0, flags=536872576,
vmlist=0x7786e918, link_fragment=1, add_to_htable=1, replace_fragment=0x0)
at /.../Workspace/DynamoRIO/dynamorio-dev/core/emit.c:468
#6 0x00000000710a00d0 in emit_fragment_ex (dcontext=0x711abdd0,
tag=0x78c <Address 0x78c out of bounds>, ilist=0x711ac690, flags=1897581321,
vmlist=0x711b1b3c, link=1, visible=1)
at /.../Workspace/DynamoRIO/dynamorio-dev/core/emit.c:1009
#7 0x000000007115c580 in build_basic_block_fragment (dcontext=0x77846060,
start=0x2ab2f8c50b0b "H\213\223\b\001", initial_flags=0, link=1, visible=1,
for_trace=0,
unmangled_ilist=0x0)
at /.../Workspace/DynamoRIO/dynamorio-dev/core/x86/interp.c:4065
#8 0x000000007109b176 in dispatch (dcontext=0x77846060)
at /.../Workspace/DynamoRIO/dynamorio-dev/core/dispatch.c:184
#9 0x0000000077826d59 in ?? ()
#10 0x0000000000000000 in ?? ()

[derekbruening]

From [email protected] on June 24, 2009 19:34:50

The instr cause the decoding failure has following content:
opcode = 55, num_dsts = 1 '\001', num_srcs = 1 '\001',
src0 = {kind = 9, size = 146 '\222', seg = 0,
value = {pc = 0x778461f0,
base_disp = {disp = 2005164528, base_reg = 0, index_reg = 0,
scale = 0, encode_zero_disp = 0, force_full_disp = 0,
disp_short_addr = 0}, addr = 0x778461f0}
}
dsts[0] = {kind = 5, size = 214, seg = {far_pc_seg_selector = 30597,
segment = 133 '\205'}, value = {immed_int = 1897157137, immed_float = 7.34447898e+29,
pc = 0x71145211 "\216", instr = 0x71145211, reg = 17 '\021', base_disp = {
disp = 1897157137, base_reg = 0 '\0', index_reg = 0 '\0', scale = 0 '\0',
encode_zero_disp = 0 '\0', force_full_disp = 0 '\0', disp_short_addr = 0 '\0'},
addr = 0x71145211}}

[derekbruening]

From [email protected] on June 25, 2009 09:45:46

Your call stack is for an encoding assert, but you opened the case with an
assert-not-implemented?

[derekbruening]

From [email protected] on June 25, 2009 10:10:26

Sorry, that's was my mistake.
My source code is newer that the compiled code, so mis-aligned code reading.
But still, I do not understand why similar code works fine for code cache entry/exit,
but have encoding problem for dr_insert_clean_call.

[derekbruening]

From [email protected] on June 25, 2009 12:28:16

for x64, with a lack of absolute addressing able to reach the whole address space, we
decided to go with a unified TLS scheme whether thread-shared or thread-private.
this looks like a piece that was overlooked (all-thread-private on x64 wasn't high
priority: only support for internally-thread-private was considered necessary).

I know you can't access these but listing for my own xref purposes. I have a bunch
of design notes w/ alternatives for these:

PR 215396: x64 scratch space

PR 244737: [x64] scratch space for thread-private fragments (which are used
even in thread-shared DR)

Maybe best to assign to me so I can revisit our decisions back then and look at
what else we may have missed.

Owner: derek.bruening

[derekbruening]

From [email protected] on July 28, 2009 14:33:47

There are several bugs in dr_insert_clean_call and its callee:

in core/x86/instr.c:instr_create_restore_dynamo_stack:4902
instr_create_restore_from_dcontext(dcontext, REG_ESP, DSTACK_OFFSET); should be
instr_create_restore_from_dcontext(dcontext, REG_XSP, DSTACK_OFFSET);

in core/x86/mangle.c:prepare_for_clean_call:551
instr_create_restore_from_dcontext(dcontext, REG_EAX, ERRNO_OFFSET)); should be
instr_create_restore_from_dcontext(dcontext, REG_XAX, ERRNO_OFFSET)); because
ERRNO_OFFSET's size is OPSZ_PTR

Actually, this bug is more tricky, because the errno and app_errno field in
dcontext_t is int, I think it is better to change it to reg_t.

[derekbruening]

From [email protected] on July 28, 2009 15:27:31

long-term we don't want to limit ourselves to the 2GB reachability we're using to
solve some issues today: we don't want to add to the list of things that currently
requires that, to make it easier to eventually get away from it.
DR itself uses a unified TLS scheme as mentioned above and that's what I would
prefer to see here, so the code you're referring to would never be executed
for x64. (even if so, for errno int is the right data type, and REG_EAX is the right
target: the creation routine there should get the size from the register like
instr_create_restore_from_dc_via_reg() does)

[derekbruening]

From [email protected] on July 28, 2009 23:06:46

fixed in r190 we'll open a new Issue if we want to provide rip-rel access when reachable: for now
we'll go w/ all-tls

Status: Fixed

[derekbruening]

From [email protected] on July 28, 2009 23:07:21

can you verify

Owner: qin.zhao