WvTest: the dumbest cross-platform test framework that could possibly work
==========================================================================

I have a problem with your unit testing framework.  Yes, you.
The person reading this.  No, don't look away guiltily.  You
know your unit testing framework sucks.  You know it has a
million features you don't understand.  You know you hate it,
and it hates you.  Don't you?

Okay, fine.  Let's be honest.  Actually, I don't know who you
are or how you feel about your unit testing framework, but I've
tried a lot of them, and I don't like any of them.  WvTest is
the first one I don't hate, at least sort of.  That might be
because I'm crazy and I only like things I design, or it might
be because I'm crazy and therefore I'm the only one capable of
designing a likable unit testing framework.  Who am I to say?

Here are the fundamental design goals of WvTest:

 - Be the stupidest thing that can possibly work.  People are
   way, way too serious about their testing frameworks.  Some
   people build testing frameworks as their *full time job*.
   This is ridiculous.  A test framework, at its core, only does
   one thing: it runs a program that returns true or false.  If
   it's false, you lose.  If it's true, you win.  Everything
   after that is gravy.  And WvTest has only a minimal amount of
   gravy.

 - Be a protocol, not an API.  If you don't like my API, you can
   write your own, and it can still be WvTest and it can still
   integrate with other WvTest tools.  If you're stuck with
   JUnit or NUnit, you can just make your JUnit/NUnit test
   produce WvTest-compatible output if you want (although I've
   never done this, so you'll have to do it yourself).  I'll
   describe the protocol below.

 - Work with multiple languages on multiple operating systems.
   I'm a programmer who programs on Linux, MacOS, and Windows,
   to name just three, and I write in lots of programming
   languages, including C, C++, C#, Python, Perl, and others.
   And worse, some of my projects use *multiple* languages and I
   want to have unit tests for *all* of them.  I don't know of
   any unit testing framework - except maybe some horrendously
   overdesigned ones - that work with multiple languages at
   once.  WvTest does.

 - NO UNNECESSARY OBJECT ORIENTATION.  The big unit testing
   craze seems to have been started by JUnit in Java, which is
   object-oriented.  Now, that's not a misdesign in JUnit; it's
   a misdesign in Java.  You see, you can't *not* encapsulate
   absolutely everything in Java in a class, so it's perfectly
   normal for JUnit to require you to encapsulate everything in
   a class.  That's not true of almost any other language
   (except C#), and yet *every* clone of JUnit in *every*
   language seems to have copied its classes and objects.  Well,
   that's stupid.  WvTest is designed around the simple idea of
   test *functions*.  WvTest runs your function, it checks a
   bunch of stuff and it returns or else it dies horribly.  If
   your function wants to instantiate some objects while it does
   that, then that's great; WvTest doesn't care.  And yes, you
   can assert whether two variables are equal even if your
   function *isn't* in a particular class, just as God intended.

 - Don't make me name or describe my individual tests.  How many
   times have you seen this?

       assertTrue(thing.works(), "thing didn't work!");

   The reasoning there is that if the test fails, we want to be
   able to print a user-friendly error message that describes
   why.  Right?  NO!!  That is *awful*.  That just *doubled* the
   amount of work you have to do in order to write a test.
   Instead, WvTest auto-generates output including the line
   number of the test and the code on that line.  So you get a
   message like this:

       ! mytest.t.cc:431  thing.works()    FAILED

   and all you have to write is this:

       WVPASS(thing.works());

   (WVPASS is all-caps because it's a macro in C++, but also
    because you want your tests to stand out.  That's what
    you'll be looking for when it fails, after all.  And don't
    even get me started about the 'True' in assertTrue.  Come
    on, *obviously* you're going to assert that the condition is
    true!)

 - No setup() and teardown() functions or fixtures.  "Ouch!" you
   say.  "I'm going to have so much duplicated code!" No, only
   if you're an idiot.  You know what setup() and teardown() are
   code names for?  Constructor and destructor.  Create some
   objects and give them constructors and destructors, and I
   think you'll find that, like magic, you've just invented
   "test fixtures."  Nothing any test framework can possibly do
   will make that any easier.  In fact, everything test
   frameworks *try* to do with test fixtures just makes it
   harder to write, read, and understand.  Forget it.

 - Big long scary test functions.  Some test frameworks are
   insistent about the rule that "every function should test
   only one thing." Nobody ever really explains why.  I can't
   understand this; it just causes uncontrolled
   hormone-imbalance hypergrowth in your test files, and you
   have to type more stuff... and run test fixtures over and
   over.

   My personal theory for why people hate big long test
   functions: it's because their assertTrue() implementation
   doesn't say which test failed, so they'd like the *name of
   the function* to be the name of the failed test.  Well,
   that's a cute workaround to a problem you shouldn't have had
   in the first place.  With WvTest, WVPASS() actually tells you
   exactly what passed and what failed, so it's perfectly okay -
   and totally comprehensible - to have a sequence of five
   things in a row where only thing number five failed.


The WvTest Protocol
-------------------

WvTest is a protocol, not really an API.  As it happens, the
WvTest project includes several (currently five)
implementations of APIs that produce data in the WvTest format,
but it's super easy to add your own.

The format is really simple too.  It looks like this:

	Testing "my test function" in mytest.t.cc:
	! mytest.t.cc:432     thing.works()         ok
	This is just some crap that I printed while counting to 3.
	! mytest.t.cc.433     3 < 4                 FAILED

There are only four kinds of lines in WvTest, and each of the
lines above corresponds to one of them:

 - Test function header.  A line that starts with the word
   Testing (no leading whitespace) and then has a test function
   name in double quotes, then "in", then the filename, and then
   colon, marks the beginning of a test function.

 - A passing assertion.  Any line that starts with ! and ends with
   " ok" (whitespace, the word "ok", and a newline) indicates
   one assertion that passed.  The first "word" on that line is
   the "name" of that assertion (which can be anything, as long
   as it doesn't contain any whitespace).  Everything between the
   name and the ok is just some additional user-readable detail
   about the test that passed.

 - Random filler.  If it doesn't start with an ! and it doesn't
   look like a header, then it's completely ignored by anything
   using WvTest.  Your program can print all the debug output it
   wants, and WvTest won't care, except that you can retrieve it
   later in case you're wondering why a test failed.  Naturally,
   random filler *before* an assertion is considered to be
   associated with that assertion; the assertion itself is the
   last part of a test.

 - A failing assertion.  This is just like an 'ok' line, except
   augmented with extra detail.  A more advanced parser could choose to
   parse the extra string to count partial failures:

	xfail ok - test was marked as known to fail and failed
                (i.e. a known breakage)

	xpass ok - test was marked as known to fail and passed
                (i.e. previously known breakage is fixed)

	skip ok - test was skipped.

   and it could be something else instead, if you invent a new and improved way
   to fail.


Reading the WvTest Protocol: wvtestrun
--------------------------------------

WvTest provides a simple perl script called wvtestrun, which
runs a test program and parses its output.  It works like this:

	cd python
	../wvtestrun ./wvtest.py t/twvtest.py

(Why can't we just pipe the output to wvtestrun, instead of
 having wvtestrun run the test program?  Three reasons: first, a
 fancier version of wvtestrun could re-run the tests several
 times or give a GUI that lets you re-run the test when you push
 a button.  Second, it handles stdout and stderr separately.
 And third, it can kill the test program if it gets stuck
 without producing test output for too long.)

If we put the sample output from the previous section through
wvtestrun (and changed the FAILED to ok), it would produce this:

	$ ./wvtestrun cat sample-ok

	Testing "all" in cat sample-ok:
	! mytest.t.cc  my ok test function: ..... 0.010s ok

	WvTest: 5 tests, 0 failures, total time 0.010s.
	WvTest: 0 tests skipped, 0 known breakages, 0 fixed breakages.

	WvTest result code: 0

What happened here?  Well, wvtestrun took each test header (in
this case, there's just one, which said we're testing "my test
function" in mytest.t.cc) and turns it into a single test line.
Then it prints a dot for each assertion in that test function,
tells you the total time to run that function, and prints 'ok'
if the entire test function failed.

Note that the output of wvtestrun is *also* valid WvTest output.
That means you can use wvtestrun in your 'make test' target in a
subdirectory, and still use wvtestrun as the 'make test' runner
in the parent directory as well.  As long as your top-level
'make test' runs in wvtestrun, all the WvTest output will be
conveniently summarized into a *single* test output.

Now, what if the test had failed?  Then it would look like this:

	$ ./wvtestrun cat sample-error

	Testing "all" in cat sample-error:
	! mytest.t.cc  my error test function: .
	! mytest.t.cc:432     thing.works()                 ok
	This is just some crap that I printed while counting to 3.
	! mytest.t.cc.433     3 < 4	                    FAILED
	fXs 0.000s ok

	WvTest: 5 tests, 1 failure, total time 0.000s.
	WvTest: 1 test skipped, 1 known breakage, 1 fixed breakage.

	WvTest result code: 0

What happened there?  Well, because there were failed tests,
wvtestrun decided you'd probably want to see the detailed output
for that test function, so it expanded it out for you.  The line
with the dots is still there, but since it doesn't have an 'ok',
it's considered a failure too, just in case.

Watch what happens if we run a test with both the passing, and
then the failing, test functions:

	$ ./wvtestrun cat sample-ok sample-error

	Testing "all" in cat sample-ok sample-error:
	! mytest.t.cc  my ok test function: ..... 0.000s ok
	! mytest.t.cc  my error test function: .
	! mytest.t.cc:432     thing.works()                 ok
	This is just some crap that I printed while counting to 3.
	! mytest.t.cc.433     3 < 4                         FAILED
	fXs 0.000s ok

	WvTest: 10 tests, 1 failure, total time 0.000s.
	WvTest: 1 test skipped, 1 known breakage, 1 fixed breakage.

	WvTest result code: 0

Notice how the messages from sample-ok are condensed; only the
details from sample-error are expanded out, because only that
output is interesting.


How do I actually write WvTest tests?
-------------------------------------

Sample code is provided for these languages:

	C: try typing "cd c; make test"
	C++: try typing "cd cpp; make test"
	C# (mono): try typing "cd dotnet; make test"
	Python: try typing "cd python; make test"
	Shell: try typing "cd sh; make test"

There's no point explaining the syntax here, because it's really
simple.  Just look inside the cpp, dotnet, python, and sh
directories to learn how the tests are written.


How should I embed WvTest into my own program?
----------------------------------------------

The easiest way is to just copy the WvTest source files for your
favourite language into your project.  The WvTest protocol is
unlikely to ever change - at least not in a
backwards-incompatible way - so it's no big deal if you end up
using an "old" version of WvTest in your program.  It should
still work with updated versions of wvtestrun (or wvtestrun-like
programs).

Another way is to put the WvTest project in a subdirectory of
your project, for example, using 'svn:externals',
'git submodule', or 'git subtree'.


How do I run just certain tests?
--------------------------------

Unfortunately, the command-line syntax for running just *some*
of your tests varies depending which WvTest language you're using.
For C, C++ or C#, you link an executable with wvtestmain.c or
wvtestmain.cc or wvtestmain.cs, respectively, and then you can
provide strings on the command line.  Test functions will run only
if they have names that start with one of the provided strings:

	cd cpp/t
	../../wvtestrun ./wvtest myfunc otherfunc

With python, since there's no linker, you have to just tell it
which files to run:

	cd python
	../wvtestrun ./wvtest.py ...filenames...


What else can parse WvTest output?
----------------------------------

It's easy to parse WvTest output however you like; for example,
you could write a GUI program that does it.  We had a tcl/tk
program that did it once, but we threw it away since the
command-line wvtestrun is better anyway.

One other program that can parse WvTest output is gitbuilder
(http://github.com/apenwarr/gitbuilder/), an autobuilder tool
for git.  It reports a build failure automatically if there are
any WvTest-style failed tests in the build output.


Other Assorted Questions
------------------------


What does the "Wv" stand for?

	Either "Worldvisions" or "Weaver", both of which were part of the
	name of the Nitix operating system before it was called Nitix, and
	*long* before it was later purchased by IBM and renamed to Lotus
	Foundations.

	It does *not* stand for World Vision (sigh) or West Virginia.

Who owns the copyright?

	While I (Avery) wrote most of the WvTest framework in C++, C#, and
	Python, and I also wrote wvtestrunner, the actual code I wrote is
	owned by whichever company I wrote it for at the time.  For the most
	part, this means:

	C++: Net Integration Technologies, Inc. (now part of IBM)
	C#: Versabanq Innovations Inc.
	Python: EQL Data Inc.

What can I do with it?

	WvTest is distributed under the terms of the GNU LGPLv2.  See the
	file LICENSE for more information.

	Basically this means you can use it for whatever you want, but if
	you change it, you probably need to give your changes back to the
	world.  If you *use* it in your program (which is presumably a test
	program) you do *not* have to give out your program, only
	WvTest itself.  But read the LICENSE in detail to be sure.

Where did you get the awesome idea to use a protocol instead of an API?

	The perl source code (not to be confused with perlunit)
	did a similar trick for the perl interpreter's unit
	test, although in a less general way.  Naturally, you
	shouldn't blame them for how I mangled their ideas, but
	I never would have thought of it if it weren't for them.

Why are xfail/xpass/skip needed at all?

	Suppose you have a test which you expect to pass, but which is
	failing (in the usual sense).  You have two choices: 1) fix it right
	now, or 2) defer fixing.  Sometimes at the start we have lots of
	tests failing, and in order to make gradual progress, it makes sense
	to mark those presently-failing tests as "I know, it fails".

	Another use case is when tests always pass on e.g.  Linux, but some
	of them fail on Win32 due to differences in environment, and one
	does not want to concentrate on fixing win32 yet.

	So when you run tests again, you'd like to differentiate between
	failing tests marked as xfail (known to fail) and new failing tests.
	The latter should produce real FAILURE with details. xfail on the
	other hand should produce just a small warning/reminder (I'm failing
	here, please don't forget to fix me).

	That's the idea.

	So now what happens when a test marked as xfail passes anyway
	instead of the expected failure?  That's not a failure - quite
	differently, it _was_ a failure, and now it passes, so what should
	we do?  Right, we should tell the user that "hey, a test which used
	to fail now passes!  You probably would want to mark it back as
	PASS".

	This is how xfail/xpass works.

	With introduction of xfail tests are no longer strictly true or
	false, but you can ignore the additional information if you want
	(the last word is still either "ok" or not).

Who should I complain to about WvTest?

	Email me at: Avery Pennarun <[email protected]>

	I will be happy to read your complaints, because I actually really
	like it when people use my programs, especially if they hate them.
	It fills the loneliness somehow and prevents me from writing bad
	poetry like this:

		Testing makes me gouge out my eyes
		But with WvTest, it takes fewer tries.
		WvTest is great, wvtest is fun!
		Don't forget to call wvtestrun.