Provide a public API for manually advancing test execution

[mdebbar]

Tell us about your runtime:

• QUnit version: 2.4.0
• What environment are you running QUnit in? (e.g., browser, Node): both
• How are you running QUnit? (e.g., script, testem, Grunt): testem

What are you trying to do?

I'm trying to run the tests and pause after each test. Then I need to manually resume the tests.

I've tried using QUnit.config.blocking = true; after a test is finished. It works as expected. The tests are paused. But there's no way to tell QUnit to resume the execution of tests. Setting QUnit.config.blocking to false doesn't help. Looking at this code it seems like QUnit stops polling for the queue when blocking is set to true.

What are you proposing?

Expose the advance method from src/core/processing-queue.js as a public API. Maybe add some checks to make sure advance is used correctly (e.g. it can't be called when a test is already running).

[platinumazure]

Have you tried using assert.async()?

You could use that in afterEach to stop the test processing and pass the returned callback function to whatever validation function or event handler needs it.

// Pause tests for 1 second in between each test
QUnit.module( "my module", function( hooks ) {
    hooks.afterEach(function( assert ) {
        const advanceTests = assert.async();
        setTimeout(advanceTests, 1000);
    });

    QUnit.test( "some test", function( assert ) {
        assert.ok(true);
    });
});

[rwjblue]

@platinumazure - Ya, it absolutely allows pausing the test, but there is no way to pause the test after full completion.

The goal that @mdebbar is shooting for is to do things use chrome's remote debugging infrastructure to be able to measure memory usage, pending async, pending promises, grab per-test code coverage, etc. This, for example, would allow us to determine if there is leakage between tests. I believe that for this to work we actually need to run after all of the afterEach's run (since those are what are doing much of the cleanup work).

[trentmwillis]

This used to be possible via QUnit.start and QUnit.stop, but those APIs have since changed/been removed. I'm wary of giving full control over processing as it has potential to be a foot-gun from my experience.

Is the code executing after the tests intended to be async? You could use the QUnit.on('testEnd') callback if it is synchronous.

If the code execution needs to be async, then perhaps we should evaluate if the reporting callbacks should support returning Promises. As I believe those callbacks cover any remaining "gaps" in the processing timeline.

[rwjblue]

Yes, I believe that it needs to be async. I believe that supporting returning promises in the reporting callbacks should probably work fine for this use-case.

@mdebbar - Can you confirm?

[mdebbar]

What @rwjblue said is correct (thanks for clarifying). We want to be able to do async work between tests.

I like the idea of returning promises from QUnit hooks! Simpler than what I originally had in mind, and less likely to be a foot-gun 🙂

Made a quick POC #1230. I appreciate review and guidance on how to write tests for it. Thanks!

[Krinkle]

From #1230:

@mdebbar wrote on 20 Oct 2017:
Test: Pause when promise is returned from testEnd callback

Note that you can return a promise and/or use async functions with testStart() and testDone(). See also docs.

These are native to QUnit and not part of the js-reporters spec (which calls these testStart and testEnd instead, and requires them to be synchronous).

If you'd want the test to be "fully" finished, I suppose you could also use testStart() and done() instead of testDone(), that way you're always after a test has fully ended, and can then do any work afterwards whilst holding back the next test.

I'm closing this for now as it's been open for a while without activity. As Trent said on the PR, happy to help think this out further and/or consider new hooks to help make things work more easily.