Cabal Test 2.0

[ttuegel]

I would like to revise the test interface in Cabal 2.0 to reflect what we have learned in the last six years.

In no particular order, here are some lessons I've picked up:

1. The standard I/O interface is good enough for most people. In no small part, this is due to the proliferation of good tools from other frameworks for dealing with this style of test.
2. The detailed interface is over-engineered. Few people need the few additional features it provides. The interface is hard to use; the cost of implementing it is far too high.
3. It is not as important as we thought for Cabal to handle logging. Many of our users need only direct-to-terminal output. (Again, this facilitates interaction with other tools, like continuous integration systems.) For the remaining users, simple log-to-file suffices.
4. It is important to provide some facility for dynamically generating tests. Integration with doctest is much more difficult than it should be. This should not require a custom Setup.hs!

I propose to design a new test interface to become the default interface in Cabal 2.0. In this document, I will refer to it as "test interface 2.0". I want to start by building off our success with the standard I/O interface; therefore, a "test suite" will consist of an executable which Cabal will compile. Test interface 2.0 will define a standard command-line interface for test suites; this allows us to incorporate desirable features (mainly introspection) with minimal overhead for users.

There is discussion in #3545 regarding pre-processors; it would be natural to treat test generators like doctest as pre-processors.

The test logging features should be removed from the Cabal library. We can provide minimal logging features in cabal-install.

[ezyang]

OK, if this solves doctest I'm all for it!

[23Skidoo]

Related: #2327

[ttuegel]

OK, if this solves doctest I'm all for it!

Well, that depends on a satisfactory resolution of the preprocessor issues stemming from #3545.