clojure.test integration

[bbatsov]

clojure-test-mode is an unsupported mess full of inlined Clojure code. It's considered officially deprecated right now and it'd be nice if we bundled clojure.test support directly into cider as it's the standard test framework. I'm seeing two options:

1. A single command cider-test-ns that simply opens the output of the test run into a new buffer
2. Maintaining a feature set and UI similar to that of clojure-test-mode.

At any rate we'll need some basic middleware for running the tests and clearing their definitions before test re-runs. @jeffvalk Would you like to look into implementing this? Should be reasonable straight-forward. I'm tied up cleaning up clojure-mode and hunting encoding/decoding issues and won't be able to attend to this any time soon, but I want us to lay clojure-test-mode to rest once and for all.

[expez]

Any hope of this making it into 0.7? I think a lot of people are eager to get on the 'stable branch' after essentially being forced to ride master for cider and cider-nrepl the last months. Staying on 0.7-STABLE until 0.8-STABLE comes out would leave those users without a reliable way to run their tests for quite some time (development time for 0.8 + delays due to summer vacations).

[jeffvalk]

@bbatsov This had been in the back of my mind as well. I will try to carve out some time for this in the next week.

Without having looked into any implementation details yet, one design issue does come to mind: how to deal with test exceptions. With clojure-test-mode, any exceptions from tests simply had their stacktraces dumped to the REPL. As a result, I always found the signal-to-noise ratio frustratingly low whenever test errors occurred. Naturally, I'd like to see test exceptions presented using cider-stacktrace, but this implies displaying a single test's error causes (or multiple errant tests in sequence, which sounds like a mess). Our test runner will need a natural/comfortable way to manage this. One possibility that stands out to me is:

• In middleware, capture each test's results, including exceptions, and store the most recent test results in the session.
• In the client, present test results in a popup buffer, and allow the user to highlight an exception-throwing test.
• On a user's command, query the middleware for the stacktrace of the highlighted test's exception and display it using cider-stacktrace.

@bbatsov Any thoughts on this? This strikes me as a design issue @hugoduncan might like to weigh in on too.

[bbatsov]

@jeffvalk Sounds like an excellent idea to me! The UI design I had in mind is pretty much what you describe - a listing of tests in a popup buffer, with the ability to act on failing tests and see some details about the failure.

[jeffvalk]

Ok. I'll roll with that.

[hugoduncan]

Having exceptions from clojure.test displayed with cider-stacktrace would be a huge improvement.

Having test fails in a separate buffer would also be useful, espescially if displayed using the test context information provided by testing forms. Being able to re-run a specific test from the list of failed tests would be interesting too.

Integrating test fail output with ediff is something I use a lot from clojure-test-mode, as is the ability to pretty-print failures.

Note that a test function can have multiple fails, but only one error (after which the rest of the test function is not run).

[hugoduncan]

Another consideration, perhaps for a later date, would be to design the ops for the test middleware such that other test systems (such as midje) could provide an implementation.

[bbatsov]

Being able to re-run a specific test from the list of failed tests would be interesting too.

Nice idea. Perhaps we can have the option to rerun the test at point in the list or all failed tests together.

Integrating test fail output with ediff is something I use a lot from clojure-test-mode, as is the ability to pretty-print failures.

We should start small. This could be added at a later point in time.

Note that a test function can have multiple fails, but only one error (after which the rest of the test function is not run).

Indeed. We should have 3 states for the tests - passed, failed and erred (or something like that).

Another consideration, perhaps for a later date, would be to design the ops for the test middleware such that other test systems (such as midje) could provide an implementation.

That'd be great indeed, but we should definitely start small and avoid over-engineering the initial solution.

[jeffvalk]

What's in PR #625 should fit the bill.