runsuite return 0 when tests fail

[derekbruening]

From [email protected] on February 01, 2011 21:37:42

http://build.chromium.org/p/client.drmemory/builders/win-xp-drm/builds/820/steps/Build%20and%20test%20Dr.%20Memory/logs/stdio ==================================================
RESULTS

drmemory-dbg-32: 5 tests passed, **** 15 tests failed: ****
free
registers
malloc
cs2bug => stderr failed to match :::Dr.Memory::: 1 unique, 1 total uninitialized
access(es)?, found :::Dr.Memory::: 5 unique, 5 total uninitialized
access(es) instead
patterns => stderr failed to match :::Dr.Memory::: 0 unique, 0 total unaddressable
access(es)?, found :::Dr.Memory::: 1 unique, 1 total unaddressable
access(es) instead
float
leaks-only
suppress => stderr failed to match :::Dr.Memory::: 0 unique, 0 total unaddressable
access(es)?, found :::Dr.Memory::: 6 unique, 6 total unaddressable
access(es) instead
suppress-gensyms
suppress-genoffs
wincrt
wincrtdbg => stderr failed to match :::Dr.Memory::: 7 unique, 7 total unaddressable
access(es)?, found :::Dr.Memory::: 4 unique, 4 total unaddressable
access(es) instead
malloccrt => stderr failed to match :::Dr.Memory::: 1 unique, 20 total unaddressable
access(es)?, found :::Dr.Memory::: 1 unique, 1 total unaddressable
access(es) instead
nudge => stderr failed to match :::Dr.Memory::: 2 unique, 21 total, 3259 byte(s)
of leak(s)?, found :::Dr.Memory::: 4 unique, 23 total, 3919 byte(s) of
leak(s) instead
realloc => (1452). Internal Error Internal DynamoRIO Error:
drmemory-rel-32: build successful; no tests for this build
final package: build successful; no tests for this build

program finished with exit code 0

Original issue: http://code.google.com/p/drmemory/issues/detail?id=284

[derekbruening]

From [email protected] on February 02, 2011 05:40:08

ctest scripts are designed to work with data consumers like cdash, which parse the xml logs to determine success of each configure, build, and test, and don't normally use the return code to indicate success of the tests, but this can be added.

what's disturbing are two things: 1) all the failures: on my xp machine only nudge fails, and 2) why my mismatch info isn't showing up for the other failing tests. I'll need to either log on to this machine or get the results logs (recently added) to figure that out.

[derekbruening]

From [email protected] on February 02, 2011 06:24:51

split the no-mismatch test failures off as issue #285 and the discrepancies in number of errors as issue #286

[derekbruening]

From [email protected] on May 04, 2011 08:54:40

OK, required parsing is done on the buildbot in r280 .

Still, I think ctest should return non-zero code to be used in cmd line etc but the priority is much lower now.

Owner: [email protected]
Labels: -Priority-Medium Priority-Low Component-Tests

[derekbruening]

From [email protected] on May 04, 2011 08:59:58

xref ctest_test()'s RETURN_VALUE parameter