"make test" shows failures not present with "ctest" on Linux x64

[derekbruening]

From [email protected] on June 05, 2011 17:18:39

Ubuntu 11.04 32-bit

72% tests passed, 26 tests failed out of 92

Total Test time (real) = 50.29 sec

The following tests FAILED:
64 - code_api|client.abort (Failed)
65 - code_api|client.alloc (Failed)
66 - code_api|client.call-retarget (Failed)
67 - code_api|client.cleancall (Failed)
68 - code_api|client.count-ctis (Failed)
69 - code_api|client.count-bbs (Failed)
70 - code_api|client.syscall (Failed)
71 - code_api|client.modules (Failed)
72 - code_api|client.events (Failed)
73 - code_api|client.signal (Failed)
74 - code_api|client.syscall-mod (Failed)
75 - code_api|client.nudge_test (Failed)
76 - code_api|client.timer (Failed)
77 - code_api|client.file_io (Failed)
78 - code_api|client.flush (Failed)
79 - code_api|client.strace (Failed)
80 - code_api|client.thread (Failed)
81 - code_api|client.truncate (Failed)
82 - code_api|client.unregister (Failed)
83 - code_api|client.inline (Failed)
84 - code_api|client.drmgr-test (Failed)
85 - code_api|client.drwrap-test (Failed)
89 - code_api,opt_memory|client.events (Failed)
90 - code_api,thread_private|client.events (Failed)
91 - code_api,disable_traces|client.events (Failed)
92 - code_api,thread_private,disable_traces|client.events (Failed)
Errors while running CTest
make: *** [test] Error 8

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=480

[derekbruening]

From [email protected] on June 05, 2011 14:22:57

In 32-bit Unbuntu 11.04, libc is in /lib/i386-linux-gnu, by adding it to the search path, only 8 test cases fail now.

91% tests passed, 8 tests failed out of 92

Total Test time (real) = 58.20 sec

The following tests FAILED:
70 - code_api|client.syscall (Failed)
77 - code_api|client.file_io (Failed)
78 - code_api|client.flush (Failed)
79 - code_api|client.strace (Failed)
80 - code_api|client.thread (Failed)
83 - code_api|client.inline (Failed)
84 - code_api|client.drmgr-test (Failed)
85 - code_api|client.drwrap-test (Failed)
Errors while running CTest
make: *** [test] Error 8

[derekbruening]

From [email protected] on June 05, 2011 14:38:00

For test 70, client failed to find module client.syscall because its module name is set NULL:

*data = {{start = 0x8048000 "\177ELF\001\001\001", handle = 0x8048000}, end = 0x804b000 <Address 0x804b000 out of bounds>,
entry_point = 0x8048510 "1\355^\211\341\203\344\360PTRhp\206\004\bh\020\206\004\bQVh�\004\b\350\277\377\377\377\364\220\220\220\220\220\220\220\220\220\220\220\220\220\220U\211\345S\203\354\004\2
00=\030\240\004\b", flags = 0, names = {module_name = 0x0, file_name = 0x18f2ea50 "client.syscall", inode = 0},
full_path = 0x18f2e9f0 "/home/zhaoqin/Workspace/DynamoRIO/dynamorio-read/build/suite/tests/bin/client.syscall", contiguous = 1, num_segments = 2, segments = 0x18f2ea68}

[derekbruening]

From [email protected] on June 06, 2011 20:07:11

For test 70, dr_module_preferred_name works correctly, the problem happens strcmp returnning a wrong result.

[derekbruening]

From [email protected] on June 07, 2011 02:44:53

Qin, on Ubuntu 10.04 w/ the private loader we hit a SIGSEGV here:

#0 0x00007f0bc1200b28 in __strcasecmp (s1=0x72000f43 "libstdc++.so.6", s2=0x2fe229a92f <Address 0x2fe229a92f out of bounds>) at strcasecmp.c:65
#1 0x00000000711dd1bf in privload_lookup (name=0x72000f43 "libstdc++.so.6") at /home/username/DR/core/loader_shared.c:273
#2 0x00000000712a15a8 in privload_process_imports (mod=0x4157fb38) at /home/username/DR/core/linux/loader.c:462
#3 0x00000000711ddd2e in privload_load_finalize (privmod=0x4157fb38) at /home/username/DR/core/loader_shared.c:499
#4 0x00000000711dcddc in loader_init () at /home/username/DR/core/loader_shared.c:119
#5 0x00000000710805ab in dynamorio_app_init () at /home/username/DR/core/dynamo.c:496
#6 0x00007f0bc20d0b00 in _init () at /home/username/DR/core/linux/preload.c:186
#7 0x00007f0bc22e5d25 in call_init (main_map=0x7f0bc24f8128, argc=1, argv=0x7fffe229ba78, env=0x7fffe229ba88) at dl-init.c:70
#8 _dl_init (main_map=0x7f0bc24f8128, argc=1, argv=0x7fffe229ba78, env=0x7fffe229ba88) at dl-init.c:134
#9 0x00007f0bc22d7b2a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#10 0x0000000000000001 in ?? ()

have you already hit this? do you need more information? should this be a separate Issue or is it quite related to the test failures?

basically we've had to revert to pre-private-loader to get this C++ client to work

[derekbruening]

From [email protected] on June 07, 2011 09:24:07

No, I did hit this, can you provide more information?

[derekbruening]

From [email protected] on June 07, 2011 11:45:42

split comment 4 as issue #489

[derekbruening]

From [email protected] on June 14, 2011 20:43:59

on Fedora 12 I'm seeing these tests fail. they all pass with -no_private_loader:

debug-internal-32: 90 tests passed, **** 3 tests failed: ****
code_api|client.flush
code_api|client.drwrap-test
debug-internal-64: 85 tests passed, **** 4 tests failed: ****
code_api|client.syscall
code_api|client.flush
code_api|client.drwrap-test
code_api|client.drmgr-test

flush:
has the use_unlink id's: how??

syscall:
ERROR: did not find start/stop markers

drwrap:
cannot find lib export

drmgr:
failed to hit clean call

[derekbruening]

From [email protected] on July 04, 2011 02:11:52

For syscall test, it fails on strcmp.
The code for strcmp in glibc-2.12.1 is shown below:
sysdeps/x86_64/multiarch/strcmp.S:
.text
ENTRY(STRCMP)
.type STRCMP, @gnu_indirect_function
cmpl $0, __cpu_features+KIND_OFFSET(%rip)
jne 1f
call __init_cpu_features
1:
leaq STRCMP_SSE42(%rip), %rax
testl $bit_SSE4_2, __cpu_features+CPUID_OFFSET+index_SSE4_2(%rip)
jnz 2f
leaq STRCMP_SSSE3(%rip), %rax
testl $bit_SSSE3, __cpu_features+CPUID_OFFSET+index_SSSE3(%rip)
jnz 2f
leaq STRCMP_SSE2(%rip), %rax
2: ret
END(STRCMP)

It does not execute any version of real strcmp code but put it into the rax and returns.

[derekbruening]

From [email protected] on July 04, 2011 15:38:59

is this just the new indirect export type first encountered in issue #248 ?

[derekbruening]

From [email protected] on July 05, 2011 08:58:29

split comment 8 to issue #505

[derekbruening]

From [email protected] on July 08, 2011 02:41:33

Now r861 has passed most test cases.
However there are several non-deterministic test case failure as below:

zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build_suite$ ctest -S ../suite/runsuite.cmake
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!

RESULTS

debug-external-32: build successful; no tests for this build
debug-external-64: build successful; no tests for this build
debug-internal-32: 91 tests passed, **** 2 tests failed: ****
code_api|linux.persist
code_api|linux.freeze
debug-internal-64: all 89 tests passed
release-external-32: build successful; no tests for this build
release-external-64: build successful; no tests for this build
vmsafe-debug-internal-32: build successful; no tests for this build
vps-debug-internal-32: build successful; no tests for this build

Error in read script: /home/zhaoqin/Workspace/DynamoRIO/dynamorio/suite/runsuite.cmake
zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build_suite$ ctest -S ../suite/runsuite.cmake
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!

RESULTS

debug-external-32: build successful; no tests for this build
debug-external-64: build successful; no tests for this build
debug-internal-32: 92 tests passed, **** 1 tests failed: ****
code_api|security-common.jmp_from_trace => Application security-common.jmp_from_trace (5327). Internal Error Internal DynamoRIO Error: /home/zhaoqin/Workspace/DynamoRIO/dynamorio/core/linux/signal.c:2271 false
debug-internal-64: 88 tests passed, **** 1 tests failed: ****
code_api|security-common.decode-bad-stack
release-external-32: build successful; no tests for this build
release-external-64: build successful; no tests for this build
vmsafe-debug-internal-32: build successful; no tests for this build
vps-debug-internal-32: build successful; no tests for this build

Error in read script: /home/zhaoqin/Workspace/DynamoRIO/dynamorio/suite/runsuite.cmake
zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build_suite$ ctest -S ../suite/runsuite.cmake
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!
No tests were found!!!

RESULTS

debug-external-32: build successful; no tests for this build
debug-external-64: build successful; no tests for this build
debug-internal-32: 91 tests passed, **** 2 tests failed: ****
code_api|security-common.jmp_from_trace => Application security-common.jmp_from_trace (23741). Internal Error Internal DynamoRIO Error: /home/zhaoqin/Workspace/DynamoRIO/dynamorio/core/linux/signal.c:2271 false
code_api|linux.persist
debug-internal-64: 88 tests passed, **** 1 tests failed: ****
code_api|linux.threadexit2
release-external-32: build successful; no tests for this build
release-external-64: build successful; no tests for this build
vmsafe-debug-internal-32: build successful; no tests for this build
vps-debug-internal-32: build successful; no tests for this build

Error in read script: /home/zhaoqin/Workspace/DynamoRIO/dynamorio/suite/runsuite.cmake

[derekbruening]

From [email protected] on July 08, 2011 02:52:44

In addition, different ways of running tests cause different results:
In Ubuntu-11.04 64-bit debug build:
Using make test

Total Test time (real) = 830.44 sec

The following tests FAILED:
9 - code_api|linux.execve-null (Failed)
41 - code_api|linux.threadexit (Failed)
43 - code_api|linux.vfork (Failed)
46 - code_api|pthreads.pthreads (Failed)
47 - code_api|pthreads.ptsig (Failed)
76 - code_api|client.drmgr-test (Failed)
Errors while running CTest
make: *** [test] Error 8

However, using ctest

zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build$ ctest -I 9,9
Test project /home/zhaoqin/Workspace/DynamoRIO/dynamorio/build
Start 9: code_api|linux.execve-null
1/1 Test #9: code_api|linux.execve-null ....... Passed 0.46 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.61 sec
zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build$ ctest -I 41,41
Test project /home/zhaoqin/Workspace/DynamoRIO/dynamorio/build
Start 41: code_api|linux.threadexit
1/1 Test #41: code_api|linux.threadexit ........ Passed 0.72 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.92 sec
zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build$ ctest -I 43,43
Test project /home/zhaoqin/Workspace/DynamoRIO/dynamorio/build
Start 43: code_api|linux.vfork
1/1 Test #43: code_api|linux.vfork ............. Passed 0.56 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.71 sec
zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build$ ctest -I 46,46
Test project /home/zhaoqin/Workspace/DynamoRIO/dynamorio/build
Start 46: code_api|pthreads.pthreads
1/1 Test #46: code_api|pthreads.pthreads ....... Passed 0.40 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.55 sec

zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build$ ctest -I 47,47
Test project /home/zhaoqin/Workspace/DynamoRIO/dynamorio/build
Start 47: code_api|pthreads.ptsig
1/1 Test #47: code_api|pthreads.ptsig .......... Passed 0.37 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.52 sec
zhaoqin@ubuntu:~/Workspace/DynamoRIO/dynamorio/build$ ctest -I 76,76
Test project /home/zhaoqin/Workspace/DynamoRIO/dynamorio/build
Start 76: code_api|client.drmgr-test
1/1 Test #76: code_api|client.drmgr-test ....... Passed 0.47 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.62 sec

Adding --force-new-ctest-process does not change the ctest result

[derekbruening]

From [email protected] on July 11, 2011 07:59:41

if it were one or two tests, could disable from "make test" (xref issue #389 )
with that many need to investigate: why are they failing? just nondet output order? I've seen some tests fail in the runsuite b/c they're run in parallel and when I run them serially they all succeed.

[derekbruening]

From [email protected] on July 12, 2011 10:20:32

Those failed tests using make test are caused by TLS access violation in spawned threads. I have trouble to reproduce the failure on individual run, which makes it difficult to debug.

[derekbruening]

From [email protected] on October 26, 2011 06:47:28

Actually, this is the older bug tracking this "make test" vs "ctest" issue and it has more relevant information, so I think I'll merge the new one into the old one.

Summary: "make test" shows failures not present with "ctest" on Linux x64
Owner: [email protected]
Labels: -Priority-Critical Priority-Medium Component-Tests

[derekbruening]

From [email protected] on October 26, 2011 06:47:40

Issue 591 has been merged into this issue.

[derekbruening]

From [email protected] on December 10, 2012 07:13:27

Owner: ---

[derekbruening]

From [email protected] on December 19, 2012 09:29:19

A user pointed out that Debian make adjusts the stack size of child processes: https://groups.google.com/d/msg/dynamorio-users/jY3JfkZGLh8/FJz95TE4QcUJ http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=696222 That may have something to do with this.