build: convert suite/ to cmake and ctest

[derekbruening]

From [email protected] on March 03, 2009 18:14:51

split from base cmake feature issue #19 we should convert our suite/tests/Makefile build rules to cmake, and our
suite/runregression and suite/tests/Makefile test harness to ctest. then
we can hook it up to Dash and have a dashboard from remote nightly
regressions (xref issue #11 )

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=65

[derekbruening]

From [email protected] on March 03, 2009 15:54:27

this would also address issue #12

[derekbruening]

From [email protected] on March 24, 2009 21:15:51

r106 has a good first step but not quite all the way there

Major features in this commit:

• Basic build support for all 3 types of test: native, client, and
  standalone client. Required some work to do it all in single
  CMakeLists.txt while still using DynamoRIOUse.cmake and not have it
  scattered in subdirs.
• Support for a test either including configure.h or taking in
  platform defines on cmdline.
• Basic CTest test running support for all 3 types of test.
• CTest run support for most of the crazy things we did in tests/Makefile:
  • custom DR options per test
  • custom client options per test
  • "fake" tests that have no executable of their own
• Switch from gen_expect.pl to cpp for processing .template files
  to make it more intuitive for new users adding new tests and to
  help eliminate perl ( issue build: replace all uses of perl with cmake and ctest #68 ).
  I modified the custom .template syntax to use cpp syntax in all the
  .template files.
• Updated tools/runstats to have a representative exit value.
• Fixed bugs in tests/tools.h

[derekbruening]

From [email protected] on March 26, 2009 08:45:32

r112 has step 2:

Explored several alternatives but came up with the following plan:

• LONG vs SHORT is configure-time
• We build all tests at general build time: in future can try
  ctest --build-and-test
• "make test" runs the suite of tests for each of a set of runtime options
  for the current build. If not running a test suite of multiple builds
  it runs a full set of tests; otherwise it eliminates some overlapping
  tests on release or external builds that were run on debug internal.
• The set of runs is in a list form with qualifiers very similar to the
  list in suite/runregression, making it easy to change what tests are run
  and for which builds by each option set.
• Can set additional options at configure time via TEST_OPTIONS:
  these are added to every options run.
• We'll use ctest -S to build each target build config and run tests for each,
  and optionally submit to dashboard. Use w/o submit as "make runregression".

Fixed some issues:

• Must link tools.c and its embedded asm into tests
• Updated special SEC_VIO_* macros in .template files
• DynamoRIOUse.cmake should not clean ldscript in "make clean"

Still TODO (see all the FIXMEs in tests/CMakeLists.txt):

• run samples
• Windows x64 needs 32-bit drdeploy
• Windows support: .runall tests, internal drinject, build .dll.c,
  RUNREGRESSION_winversion in templates
• api/dis support: either run script or dis.c does its own diff
• several features: per-test time limit, pcache repeated run, run natively
• set up ctest -S script
• how handle execve multi-build test?
• update TestSuite .wiki

[derekbruening]

From [email protected] on March 26, 2009 10:52:41

r113 has step 3:

• Fix build errors on 32-bit linux when using 64-bit gcc
• Add single run in SHORT for several popular runtime options

[derekbruening]

From [email protected] on April 05, 2009 09:47:52

Issue 12 has been merged into this issue.

[derekbruening]

From [email protected] on April 12, 2009 12:57:22

Windows support for 32-bit is pretty much there, though more tests fail than did
before so a few wrinkles to iron out yet:

---

in r128 :
issue #65: initial Windows test support

• fix errors in Windows test templates
• fixed numerous issues raised by warnings now that tests are built /W3 /WX
  • doubles vs floats in INSTR_CREATE_ macros
  • no decls for asm routines
  • wrong type for _beginthreadex
  • unused vars
• added .rc resource files to Windows tests

---

in r134 :
issue #65: more Windows test support

• we require VS2005 SP1 for __nop() intrinsic in some CI tests
• added missing custom build rules:
  • /fixed
  • use bind.exe
  • no pdbs
• continued from r128 :
  fixed numerous issues raised by warnings now that tests are built /W3 /WX
  • several outright bugs in tests (uninitialized vars, etc.): we should
    have had warnings turned on earlier
  • disabled warnings (_CRT_SECURE_NO_WARNINGS, safe seh, fpe sig handler)
  • missing #includes (mostly stdio.h)
  • unused vars (a lot of these)
  • missing casts
• was using wrong linker flags for non-client dlls
• updated assert exemptions in the core for new test names "dirname.testname"
• fixed blank lines in template files wrt cpp and blank stripping
• copy dynamorio.dll to run dir for api.* tests

changes for tests that affect non-test usage:

• drinject.exe is now always EXTERNAL_INJECTOR and a new target
  drinjectx.exe is the internal version
  • added -ops parameter to drinjectx.exe as alternative to env var
• add /GS- to DynamoRIOUse.cmake as it can pull in kernel32 and libc

[derekbruening]

From [email protected] on April 12, 2009 21:14:38

in r135 :
issue #65: more Windows test support

• 64-bit test support: fixed some build errors; disabled other tests that
  require more work, covered by issue port all suite/tests/* to x64 #16
• fixed test failures for common.decode_, win32.dll, client.cbr_, client.flush

still TODO:
the major missing piece is the runall tests, separated as issue #120 also:

• run samples
• api/dis and TestMemProtChg w/ PROGSHEP
  make a script and compare perf on windows when using it for
  everything: if noticeable, use only for >1KLOC files
• missing features:
  • per-test time limit
  • pcache repeated run
  • run natively
• how handle execve multi-build test (xref r47 issue handle execve of different-architecture child: 32 to 64, 64 to 32 #20)?
• some tests don't build and/or run:
  see all the FIXMEs in tests/CMakeLists.txt

[derekbruening]

From [email protected] on May 25, 2009 14:32:04

r162 added execve multi-build test support

[derekbruening]

From [email protected] on February 27, 2010 21:48:02

in r280 :

Eliminated many Windows test failures.
Some tests needed multiple fixes and are listed multiple times below: Issue 122 : ASSERT synch.c:379 false (translation failure)

• win32.getthreadcontext: added handling for NtGetContextThread on self
• win32.tls.exe
• security-win32.except-execution

Added new feature .templatex where regex can be used in template:

• security-win32.except-execution
• client.syscall

Fixed test flakiness:

• security-win32.gbop-test

Updated the name of the dll/exe in the code + template:

• win32.delaybind
• win32.multisec
• win32.rebased
• win32.reload
• win32.reload-newaddr
• win32.reload-race
• win32.section-max
• security-win32.aslr-ind
• security-win32.secalign-fixed
• security-win32.sec-fixed
• security-win32.sec-xdata

Needed the NIGHTLY_REGRESSION define turned on:

• win32.reload
• win32.reload-newaddr
• win32.reload-race

Fixed minor errors in template:

• win32.partial_map

Disabled:

• win32.aslr-dll is flaky and not useful so made VPS only