Core: Implement QUnit.todo #1080

trentmwillis · 2016-12-10T20:27:57Z

Supersedes #1006. Original issue #787.

This takes a simpler approach to implementing QUnit.todo. We simply add a todo flag to log and testDone details that denotes if an assertion or a test was part of a todo. The reporter can then handle that scenario appropriately.

This approach keeps QUnit's internal logic simple and encourages reporters to report success/failure for a test suite based on the outcome of tests and not on the outcome of assertions, which is inline with the thinking of js-reporters.

While this likely means most reporters will have to implement new logic to support todo, this change should not break existing reporters and provides a relatively clear path to supporting the new feature.

Note: Assuming this implementation looks okay, I will open a PR to support todo in grunt-contrib-qunit.

leobalter · 2016-12-10T21:28:12Z

this looks amazing!

It might be a breaking change for the reporters but it's worth a major release for this, which we can bundle with a few other features, including the js-reporters.

trentmwillis · 2016-12-10T22:19:44Z

I don't think we need a major version bump. As I said, this doesn't break any existing set ups; it should only cause problems if someone starts adding todo tests.

I actually think this gives a fairly good rollout strategy as reporters can be updated over time, rather than expecting them all to do it at once.

leobalter · 2016-12-10T22:29:12Z

Got it!

I'm +1 to merge it as soon we can get the updated the grunt plugin ready.

leobalter · 2017-01-24T14:46:42Z

it is breaking in the logs file, this is weird. I need to spend a bit more time on this.

leobalter · 2017-01-24T14:51:32Z

ok, got it. Had to remind what was happening here. I'll check how is the status on grunt-contrib-qunit

leobalter · 2017-01-24T14:57:41Z

I asked @jzaefferer a little help in there and also write/publish access to the grunt plugin. I'll be able to update it as soon as we solve this.

leobalter · 2017-01-24T14:58:08Z

sorry for the spam, but I finally got access to it! Thanks @tkellen!

gibson042

I really like this approach; it seems far simpler than tracking things down to the assertion level. A few comments, but overall in favor. 👏

gibson042 · 2017-02-04T16:48:04Z

reporter/html.js

 	testItem = id( "qunit-test-output-" + details.testId );

 	assertList = testItem.getElementsByTagName( "ol" )[ 0 ];

 	good = details.passed;
 	bad = details.failed;

-	if ( bad === 0 ) {
+	let failed = details.todo ? !details.failed : !!details.failed;


Although logically correct, I find this hard to read because "failed" is used as both a boolean and a boolean-coerced number. It's purely a matter of taste, but I'd go for something like:

// This test passed if it has no unexpected failed assertions let testPassed = details.failed > 0 ? details.todo : !details.todo;

Good suggestion. I'll updated this and dependent logic.

gibson042 · 2017-02-04T16:54:54Z

reporter/html.js

+			let todoLabel = document.createElement( "em" );
+			todoLabel.className = "qunit-todo-label";
+			todoLabel.innerHTML = "todo";
+			testItem.insertBefore( todoLabel, testTitle );


Could we separate this presentation logic from the data updates?

if ( details.todo ) { let todoLabel = document.createElement( "em" ); todoLabel.className = "qunit-todo-label"; todoLabel.innerHTML = "todo"; testItem.insertBefore( todoLabel, testTitle ); } … if ( failed ) { stats.failedTests++; } else if ( todo ) { stats.todoTests++; } else { stats.passedTests++; }

Yep, this helped simplify logic and keeps things clearer.

gibson042 · 2017-02-04T17:03:58Z

reporter/html.js

@@ -1,6 +1,14 @@
 import QUnit from "../src/core";
 import { window, navigator } from "../src/globals";

+const stats = {
+	totalTests: 0,


Could we document the relationships between these values? It's worth commenting that totalTests equals the sum of the other values, and maybe even keeping two variables (e.g., let totalTests = 0; const testPartitions = {…}).

I'll remove totalTests from being tracked altogether. We only use it after the run has finished to display the number of tests that ran, so I just created a local variable at that point which clearly shows that it is the sum of the other categories.

gibson042 · 2017-02-04T17:14:06Z

build/tasks/test-on-node.js

 			testActive = false;
+
+			if ( details.failed && !details.todo ) {
+				stats.failedTests++;


This will fail to increment failedTests when a TODO test has no failing assertions. I think it should instead use the same logic as the HTML reporter.

trentmwillis · 2017-02-05T05:04:46Z

Updated with the suggestions from @gibson042 and opened a PR to update grunt-contrib-qunit to support todo: gruntjs/grunt-contrib-qunit#137. Linked the two locally and verified that it works.

gibson042 · 2017-02-05T16:47:47Z

build/tasks/test-on-node.js

+
+			if ( details.skipped ) {
+				stats.skipped++;
+			} else if ( details.failed && !details.todo ) {


Todo tests can also "fail". I think this would be clearest if it mirrored the HTML reporter:

// This test passed if it has no unexpected failed assertions var testPassed = details.failed > 0 ? details.todo : !details.todo; if ( details.skipped ) { stats.skipped++; } else if ( !testPassed ) { stats.failed++; } else if ( details.todo ) { stats.todo++; } else { stats.passed++; }

trentmwillis · 2017-02-06T06:00:01Z

Updated once more. @leobalter when you have a chance, would appreciate your help reviewing and landing this (will require a new grunt-contrib-qunit release first).

leobalter · 2017-02-07T16:40:32Z

LGTM.

We can already include the expected grunt-contrib-qunit 1.3.0

trentmwillis · 2017-02-08T04:43:24Z

Updated with the dependency update; tests are now passing!

gibson042 · 2017-02-12T03:26:11Z

reporter/html.js

+		stats.todoTests++;
+	} else {
+		stats.passedTests++;
+	}


This need to be moved up inside the else block so skipped tests don't increment two stats counters.

Good catch.

trentmwillis · 2017-02-17T05:24:21Z

Updated again. Unless any other issues are found, I think this should be good to land (will open corresponding documentation PR soon)

leobalter · 2017-02-17T16:23:51Z

(with the docs)

trentmwillis · 2017-02-17T18:25:18Z

Corresponding docs PR is now open: qunitjs/api.qunitjs.com#146

@gibson042 if this seems good to you, we can merge (and finally close this feature out!)

leobalter · 2017-02-17T19:13:40Z

Waiting for @gibson042 to confirm the changes after his feedback, LGTM.

trentmwillis mentioned this pull request Dec 10, 2016

Core: Implement QUnit.todo #1006

Closed

gibson042 reviewed Feb 4, 2017

View reviewed changes

trentmwillis force-pushed the todo-2 branch from 2bc1052 to 99a995d Compare February 5, 2017 01:07

gibson042 requested changes Feb 5, 2017

View reviewed changes

gibson042 mentioned this pull request Feb 5, 2017

Implement support for todo tests and revamp reporting logic gruntjs/grunt-contrib-qunit#137

Merged

trentmwillis force-pushed the todo-2 branch from ee3c254 to 143e2f5 Compare February 6, 2017 05:51

Core: Implement QUnit.todo

c5acd4c

trentmwillis force-pushed the todo-2 branch 2 times, most recently from 26ac494 to ce025b6 Compare February 8, 2017 04:28

trentmwillis mentioned this pull request Feb 8, 2017

Core: Add remaining js-reporter events #1101

Merged

gibson042 requested changes Feb 12, 2017

View reviewed changes

Trent Willis added 2 commits February 16, 2017 14:36

HTML Reporter: Support QUnit.todo

5898f34

Build: Update test-on-node task to support QUnit.todo

162b80d

trentmwillis force-pushed the todo-2 branch from ce025b6 to 162b80d Compare February 16, 2017 22:37

trentmwillis mentioned this pull request Feb 17, 2017

Entries: Document QUnit.todo qunitjs/api.qunitjs.com#146

Merged

gibson042 merged commit 9c6efad into qunitjs:master Feb 18, 2017

trentmwillis deleted the todo-2 branch March 2, 2018 16:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core: Implement QUnit.todo #1080

Core: Implement QUnit.todo #1080

trentmwillis commented Dec 10, 2016 •

edited

Loading

leobalter commented Dec 10, 2016

trentmwillis commented Dec 10, 2016

leobalter commented Dec 10, 2016

leobalter commented Jan 24, 2017

leobalter commented Jan 24, 2017

leobalter commented Jan 24, 2017

leobalter commented Jan 24, 2017

gibson042 left a comment

gibson042 Feb 4, 2017

trentmwillis Feb 5, 2017

gibson042 Feb 4, 2017

trentmwillis Feb 5, 2017

gibson042 Feb 4, 2017

trentmwillis Feb 5, 2017

gibson042 Feb 4, 2017

trentmwillis commented Feb 5, 2017

gibson042 Feb 5, 2017

trentmwillis commented Feb 6, 2017

leobalter commented Feb 7, 2017

trentmwillis commented Feb 8, 2017

gibson042 Feb 12, 2017

trentmwillis Feb 16, 2017

trentmwillis commented Feb 17, 2017

leobalter commented Feb 17, 2017

trentmwillis commented Feb 17, 2017

leobalter commented Feb 17, 2017

Core: Implement QUnit.todo #1080

Core: Implement QUnit.todo #1080

Conversation

trentmwillis commented Dec 10, 2016 • edited Loading

leobalter commented Dec 10, 2016

trentmwillis commented Dec 10, 2016

leobalter commented Dec 10, 2016

leobalter commented Jan 24, 2017

leobalter commented Jan 24, 2017

leobalter commented Jan 24, 2017

leobalter commented Jan 24, 2017

gibson042 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trentmwillis commented Feb 5, 2017

Choose a reason for hiding this comment

trentmwillis commented Feb 6, 2017

leobalter commented Feb 7, 2017

trentmwillis commented Feb 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trentmwillis commented Feb 17, 2017

leobalter commented Feb 17, 2017

trentmwillis commented Feb 17, 2017

leobalter commented Feb 17, 2017

trentmwillis commented Dec 10, 2016 •

edited

Loading