Build and Test failure on Ubuntu 12.04

[derekbruening]

From [email protected] on May 02, 2012 15:37:30

What version of DynamoRIO are you using? URL: http://dynamorio.googlecode.com/svn/trunk Revision: 1354 What operating system version are you running on? Linux 3.2.0-24-generic-pae #37-Ubuntu SMP i686 i686 i386 GNU/Linux

Copy-pasted commands from HowToBuild (except for ia32-libs part), got the following:

[ 14%] Building C object core/CMakeFiles/dynamorio.dir/linux/os.c.o
/tmp/dynamorio/core/linux/os.c: In function ‘dr_create_client_thread’:
/tmp/dynamorio/core/linux/os.c:2935:18: error: ‘CLONE_VM’ undeclared (first use in this function)

Seems like now it is hard requirement that one should define _GNU_SOURCE in order to use CLONE_VM. Just defining _GNU_SOURCE at the top of linux/os.c does not work, since it brings in REG_ZZZ defines that conflict with the ones from dynamorio. Attached patch solves the problem for me.

Attachment: dr-use-gnu.patch

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=762

[derekbruening]

From [email protected] on May 12, 2012 08:49:53

Thanks for the report and patch:

CLONE_VM is defined in /usr/include/bits/sched.h under macro __USE_GNU in Ubuuntu-12.04.
__USE_GNU is defined under macro _GNU_SOURCE in /usr/include/features.h

There are several test failure to be fixed after patch:

  Start 47: code_api|pthreads.pthreads

47/98 Test #47: code_api|pthreads.pthreads ............................._Failed Required regular expression not found.Regex=[^] 0.00 sec
Start 48: code_api|pthreads.pthreads_exit
48/98 Test #48: code_api|pthreads.pthreads_exit ........................_Failed Required regular expression not found.Regex=[^] 0.00 sec
Start 49: code_api|pthreads.ptsig_FLAKY
49/98 Test #49: code_api|pthreads.ptsig_FLAKY .........................._Failed Required regular expression not found.Regex=[^] 0.00 sec
Start 50: code_api|security-linux.trampoline
50/98 Test #50: code_api|security-linux.trampoline ....................._Failed Required regular expression not found.Regex=[^] 0.00 sec

Summary: Build and Test failure on Ubuntu 12.04
Status: Started
Owner: [email protected]
Labels: -Priority-Medium Priority-High OpSys-Linux

[derekbruening]

From [email protected] on May 14, 2012 21:56:55

It seems there is a deadlock.

#0 0x00007fc2adf551bb in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fc2aded9cb1 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fc2aded7a37 in malloc () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fc2adead66f in _IO_vfscanf () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fc2adec6360 in vsscanf () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00000000711f910e in our_sscanf (str=0x715bfec0 "00400000-00419000 r-xp 00000000 08:01 917582", ' ' <repeats 29 times>, "/bin/ls",
fmt=0x713443a8 "%016lx-%016lx %s %016lx %*s %lu %4096s") at /home/dynamorio/Workspace/DynamoRIO/dynamorio.svn/core/io.c:123
#6 0x00000000712aa9de in maps_iterator_next (iter=0x41b927b0) at /home/dynamorio/Workspace/DynamoRIO/dynamorio.svn/core/linux/os.c:6716
...
#12 0x0000000071239a2c in check_new_page_start (dcontext=0x41b33a00, bb=0x41b92dd0) at /home/dynamorio/Workspace/DynamoRIO/dynamorio.svn/core/x86/interp.c:598
#13 0x00000000712405bc in build_bb_ilist (dcontext=0x41b33a00, bb=0x41b92dd0) at /home/dynamorio/Workspace/DynamoRIO/dynamorio.svn/core/x86/interp.c:2740
#14 0x00000000712479d8 in build_basic_block_fragment (dcontext=0x41b33a00,
start=0x7fc2aded7a37 "H\211\323H\211\356H\211\337\350\333\322\377\377H\205\300I\211\305ts\203=\200\227\063", initial_flags=0, link=true, visible=true,
for_trace=false, unmangled_ilist=0x0) at /home/dynamorio/Workspace/DynamoRIO/dynamorio.svn/core/x86/interp.c:4245
#15 0x00000000710f9c37 in dispatch (dcontext=0x41b33a00) at /home/dynamorio/Workspace/DynamoRIO/dynamorio.svn/core/dispatch.c:186

The app is executing malloc code around
0x7fc2aded7a22 <malloc+82>: lock cmpxchg %esi,(%rdx)
0x7fc2aded7a26 <malloc+86>: jne 0x7fc2aded9ca2
0x7fc2aded7a2c <malloc+92>: jmp 0x7fc2aded7a37 <malloc+103>
0x7fc2aded7a2e <malloc+94>: cmpxchg %esi,(%rdx)
0x7fc2aded7a31 <malloc+97>: jne 0x7fc2aded9ca2
0x7fc2aded7a37 <malloc+103>: mov %rdx,%rbx
0x7fc2aded7a3a <malloc+106>: mov %rbp,%rsi
0x7fc2aded7a3d <malloc+109>: mov %rbx,%rdi
0x7fc2aded7a40 <malloc+112>: callq 0x7fc2aded4d20

While the DR is building code for app, it calls vfscanf to check memory map, which calls to app's malloc, and causes deadlock.

0x7fc2adf551a0: push % r10 0x7fc2adf551a2: push %rdx
0x7fc2adf551a3: xor % r10 ,% r10 0x7fc2adf551a6: mov $0x2,%edx
0x7fc2adf551ab: mov $0x80,%esi
0x7fc2adf551b0: cmp %edx,%eax
0x7fc2adf551b2: jne 0x7fc2adf551bb
0x7fc2adf551b4: mov $0xca,%eax // futex system call
0x7fc2adf551b9: syscall
0x7fc2adf551bb: mov %edx,%eax

Labels: Bug-Hang

[derekbruening]

From [email protected] on May 15, 2012 05:35:39

This happens despite our private loader and segment mangling, or did you turn those off to debug? I guess we have to figure out how it finds the lock, %rdx in the above code snippet.

[derekbruening]

From [email protected] on May 15, 2012 07:22:04

Even using private loader, DR is still using app's libc. So I am not surprised there would be a deadlock. Since we are using scanf, we should either write our own scanf or let DR use a separate libc.

[derekbruening]

From [email protected] on May 15, 2012 08:00:33

I guess sscanf is just not as safe as we assumed. We really can't call it if it calls malloc.

I found a BSD stand alone implementation of sscanf in an ancient revision of Kerberos and I have it in my ptrace injector branch, along with a bunch of other libc independence code ( issue #46 ). I'll split sscanf out as a separate diff and send it along, once I've made sure that the code works. It appears to modify the input string, which I don't like.

It would be interesting to try and find the glibc change between versions of Ubuntu that introduced this malloc call.

[derekbruening]

From [email protected] on May 15, 2012 08:09:38

From the source code of _IO_vfscanf_internal @ stdio-common/vfscanf.c, it clearly uses realloc, which might call malloc.

[derekbruening]

From [email protected] on May 15, 2012 09:20:25

Right, I'm curious though if this is something new in the glibc that ships with Ubuntu 12.04 vs. 11.08, or if we just never hit it before. We'd have to dig into glibc revision history. It would be interesting to know, but not super important.

[derekbruening]

From [email protected] on May 22, 2012 07:40:06

We can also fix the CLONE_* build errors by including linux/sched.h directly. I actually prefer this because I've never liked having to define GNU_SOURCE before header inclusions, and it sidesteps the REG* naming conflicts. clone is really a linux specific concept, so I have no qualms about including linux/sched.h.

I also have a new, minimal sscanf implementation prepared.

Owner: [email protected]

[derekbruening]

From [email protected] on June 04, 2012 21:29:27

Status: Fixed