Standard for test status

[JamesMGreene]

In this issue thread, I would like to focus on standardizing the collection of test statuses (status names) between testing frameworks.

Some of the known status types from various frameworks:

• Success — A test that succeeded without any failures, errors, or timeouts
• Failure — A test that failed, typically due to an assertion failure
• Error — A test that failed because of an error other than an assertion failure
• Timeout — A test that failed because it took too long or sat idle for too long
• Skipped — A test that was not run at all
• Pending — A test that does not yet exist, i.e. is named and called out but no function/callback body is supplied
• Inconclusive — A test that has been run and is not considered a failure despite the fact that it may have had assertion failures and/or errors. Referred to as "TODO" in TAP terminology.
• Global Error — An error occurred that cannot be associated with any test

[JamesMGreene]

Here's an initial attempt at a matrix:

Status Type	Mocha	QUnit	Jasmine	Intern	Buster.JS
Success	pass	passed	passed	/test/pass	success
Failure	fail	failed	failed	/test/fail	failure
Error	fail	failed	failed	/test/fail	error
Timeout	fail	failed	failed	/test/fail	timeout
Skipped	pending	skipped	disabled	/test/skip	deferred
Pending	pending	skipped	pending	—	deferred
Inconclusive	—	—	—	—	—
Global Error	uncaughtException	globalFailure	uncaughtException	/error, and possibly /suite/error	uncaughtException

Please advise me of updates as needed (or edit this comment yourself).

[slackersoft]

Jasmine has passed for success, failed for failure, error, and timeout, and pending for pending. It also has disabled for specs where xit has been used, this is probably closest to skipped above.

[JamesMGreene]

@slackersoft: Thanks! Does Jasmine have anything for "Inconclusive" or "Global Error"?

[slackersoft]

Jasmine tries to catch all errors unless you explicitly tell it not to, in which case they bubble all the way out and stop the entire run, so they just become uncaughtExceptions.

I'm not totally sure what Inconclusive means. The only other real state jasmine has is a spec with no expectations at all, but that isn't reported as a unique state, but as passed, but the built in reporters check the expectation counts to tell the user about the lack of expectations.

[JamesMGreene]

Inconclusive is based on the TODO status from the TAP spec (do a find for "TODO tests") but it is a weird one to me as well.

Essentially, I read it as "a test is annotated such that any failures/errors will not count as failures to the test/suite/run".

[cjohansen]

Buster also uses deferred for pending tests. Currently it does not differentiate between "pending" and "skipped" tests. I'm not even sure if that is a meaningful distinction to make?

[JamesMGreene]

Buster also uses deferred for pending tests.

Updated.

Currently it does not differentiate between "pending" and "skipped" tests. I'm not even sure if that is a meaningful distinction to make?

I broke out all of the distinct types of status that can occur, mostly in order to see if any of the frameworks thought them worthy of distinction. From the data gathered so far, only Jasmine treats "pending" and "skipped" tests differently.

[JamesMGreene]

@cjohansen: Does Buster have anything resembling the "Inconclusive" status? I'm actually assuming that none of our frameworks will but I'd like to confirm that before I mark it as such on the matrix.

[cjohansen]

Buster also has unsupported. In buster, you can provide a feature test with a test case, and then those tests will only be run in environments where the feature test passes. Other environments will only get a notice about tests being skipped because they test functionality not supported in the given environment.

I think we should settle on a set of core events, possibly small, and then also specify a wider range of events such that if someone wants more functionality, there is a common vocabulary for it.

I guess if you squint, unsupported could be considered "inconclusive", but not really from the definition above.

[JamesMGreene]

I think we should settle on a set of core events, possibly small, and then also specify a wider range of events such that if someone wants more functionality, there is a common vocabulary for it.

That's the goal. My point here is just to cast the net wide to see where the overlaps are (e.g. most frameworks treat failures/errors/timeouts as just a failure), and then we can work toward agreeing upon a smaller mutual subset of standard/"core" statuses.

I guess if you squint, unsupported could be considered "inconclusive", but not really from the definition above.

I'd say unsupported is just a conditional skip. We toyed with that idea in QUnit for a while, called skipIf.

[csnover]

Intern does not use a single status field, but we do broadcast topics that are picked up by reporters, which are documented at Using and writing reporters. Tests themselves have hasPassed (boolean) to mark whether or not a test passed, error (Error) to describe the failure if a test did not pass, and skipped (in 2.1; currently string, but I may change to an Error subtype before release).

[jzaefferer]

What exactly happens in Mocha, Jasmine or Buster.JS when an exception is thrown outside a test context? They don't handle it at all and just let the framework die?

[cjohansen]

Buster catches it and reports it as a global failure.

[mantoni]

Mocha catches global errors as well. If they happen during an async test, they are even associated with the correct test case.

[JamesMGreene]

@cjohansen @mantoni: Do following tests continue to run after the global error has been handled, or does the test run halt there?

[mantoni]

Just did a quick test to confirm that Mocha continues to run the remaining test cases:

it("fails as expected", function (done) {
  setTimeout(function () { throw new Error("GLOBAL OMG!"); }, 1);
});

[slackersoft]

Jasmine should continue to run remaining test cases if an Error is thrown from within a setTimeout. The error won't be associated with the spec being run though.

[cjohansen]

Buster associates the error with the current test when it can and keeps running.

[JamesMGreene]

So we all agree on that behavior. Awesome bonus knowledge!