Referencing async values with dynamically generated test cases?

[sgarbesi]

Can someone enlighten me on how to reference async values with dynamically generated test cases?

The scenario is as follows:

• Tests are dynamically generated through a Function.
• A async Function needs to be invoked to fetch the value for which the tests will run against.
• The value from the async Function needs to be passed to the test generator Function.

Illustration that doesn't work:

        var testArray = function(input, results) {
            it('should be a Array', function() {
                expect(input).to.be.a('array');
            });

            if (Object.prototype.toString.call(results) === '[object Array]') {
                it('should contain `' + JSON.stringify(results) + '`.', function() {
                    expect(input.join()).to.equal(results.join());
                });
            }
        }

        var input;

        before(function(done) {
            require(['some/random/library/on/the/fly'], function(libraryValue) {
                input = libraryValue;
            });
        });

        testArray(input, ['test']);

Issue(s):

1. We cannot nest it statement(s). If I could in this scenario, it would solve the problem.
2. Wrapping another describe Function will be invoked immediately and not prior to the before Function completing. If this worked, I could just wrap the invocation for testArray within a describe.

So I really don't see a way around this here without re-writing code.

A hacky solution could be to use a Object reference, but I'm against it.

Example:

        var testArray = function(input, results) {
            it('should be a Array', function() {
                expect(input.results).to.be.a('array');
            });

            if (Object.prototype.toString.call(results) === '[object Array]') {
                it('should contain `' + JSON.stringify(results) + '`.', function() {
                    expect(input.results.join()).to.equal(results.join());
                });
            }
        }

        var input = {};

        before(function(done) {
            require(['some/random/library/on/the/fly'], function(libraryValue) {
                input.results = libraryValue;
            });
        });

        testArray(input, ['test']);

I'm unable to find a adequate solution. Any ideas?

[sgarbesi]

A working hack:

        var testArray = function(input, results) {
            it('should be a Array', function() {
                expect(input).to.be.a('array');
            });

            if (Object.prototype.toString.call(results) === '[object Array]') {
                it('should contain `' + JSON.stringify(results) + '`.', function() {
                    expect(input.join()).to.equal(results.join());
                });
            }
        }

        before(function(done) {
            require(['some/random/library/on/the/fly'], function(input) {
                describe('hack describe', function() {
                    testArray(input, ['test']);
                    done();
                });
            });
        });

        it('hack it');

Despite this working, I really would like to see a valid illustration/example if possible.

[ScottFreeCode]

[EDITTED TO ADD: After really overthinking this whole matter, I discovered that Mocha has a built-in solution for this sort of thing, as described in this later comment. While there's some interesting discussion here, skip to there if you just want the real answer.]

I can think of a few good solutions. (Well, the first one's arguably not very good, but I'll leave it up to you to decide whether it's better than the hack you've explored. I think it's less of a hack but only marginally easier to read and understand. I'd recommend either or both of the other two if you're willing to depend on promises and/or an AMD loader.)

• Instead of giving the test-defining function the value you asynchronously retrieved, give it a callback to asynchronously retrieve it, to which it will in turn pass the actual test code as callbacks for when the value is asynchronously retrieved.

var testArray = function(getAndUseInput, results) {
    it('should be a Array', function(done) {
        getAndUseInput(function(input) {
            expect(input).to.be.a('array');
            done();
        });
    });

    if (Object.prototype.toString.call(results) === '[object Array]') {
        it('should contain `' + JSON.stringify(results) + '`.', function(done) {
            getAndUseInput(function(input) {
                expect(input.join()).to.equal(results.join());
                done();
            });
        });
    }
}

testArray(function(testInput) {
        require(['some/random/library/on/the/fly'], testInput);
    }, ['test']);

Note that if this uses an AMD loader (as it appears to judging from the asynchronous callback require function) then the value found by require will be cached, so the second test will get the same value to test as the first did without waiting for it to be resolved again. If whatever you get the input value from does not cache it, your function for getting it may need to do the caching instead.

Also don't forget to use done so that Mocha knows the test runs will complete by an asynchronous callback.

• If you're using an AMD loader, let AMD handle it all.

// --- testArray.js
// Node.js shim, uses synchronous Node.js require to retrieve the dependencies and should therefore work with the commandline test runner.
var define = typeof define === "function" ? define : function define(deps, factory) { module.exports = factory.apply(undefined, deps.map(require)) }

define(function(){ // testArray will be used by other modules to define specific tests; it has no dependencies of its own.
    return function(input, results) {
        it('should be a Array', function() {
            expect(input).to.be.a('array');
        });

        if (Object.prototype.toString.call(results) === '[object Array]') {
            it('should contain `' + JSON.stringify(results) + '`.', function() {
                expect(input.join()).to.equal(results.join());
            });
        }
    }
}


// --- testSomeRandomLibraryOnTheFly.js
// Node.js shim, uses synchronous Node.js require to retrieve the dependencies and should therefore work with the commandline test runner.
var define = typeof define === "function" ? define : function define(deps, factory) { module.exports = factory.apply(undefined, deps.map(require)) }

define(['./testArray', 'some/random/library/on/the/fly'], function(testArray, libraryValue) {
    testArray(libraryValue, ['test']);
});


// --- use in a page
// Run Mocha only after the tests have the chance to load their libraries.
require("./testSomeRandomLibraryOnTheFly", function() {
  mocha.run()
});

If the reason you're using the testArray function to define your tests is so that the same set of tests can be defined for multiple libraries, splitting it up with modules like this may be exactly the sort of thing you're looking for.

• Use promises (requires a polyfill for Internet Explorer and/or for older browsers). This is sort of like your initial hack idea inasmuch as instead of passing the value, you synchronously pass a way to retrieve the asynchronously resolved value. It's a better abstraction for it though, doesn't depend on abuse of the before hook or convoluted suite definitions, and makes the eventual use of the value asynchronous as well as the retrieval of it. Basically, promises let you inject asynchronous value retrieval into synchronously structured code by changing only the point where the asynchronous process is kicked off and the point where its result is used. Mocha has direct support for promise-based tests by letting you return a promise -- Mocha will wait for the promise to resolve or error without you having to call done(); this is an alternative way to wrap ordinary asynchronous actions in a test rather than having to use a callback to call done() when the action completes. Handily for your use case, promises also chain -- that is, the promises's then function that takes a callback to run with the resulting value will also itself return a promise, so we can pass a promise into the test definitions and use it naturally by returning the result of the test's use of the promise. And if you think all that sounds complicated, it actually comes out to something really simple and easy to follow in practice:

var testArray = function(input, results) {
    it('should be a Array', function() {
        return input.then(function(value) {
            expect(value).to.be.a('array');
        });
    });

    if (Object.prototype.toString.call(results) === '[object Array]') {
        it('should contain `' + JSON.stringify(results) + '`.', function() {
            return input.then(function(value) {
                expect(value.join()).to.equal(results.join());
            });
        });
    }
}

testArray(new Promise(function(resolve, reject) {
        require(['some/random/library/on/the/fly'], resolve);
    }), ['test']);

I like the fact that this code is pretty much self-explanatory and no longer than my introductory explanation of promises in the first place. I probably don't even need so much introductory explanation given the clarity of the code, but I like talking about the concepts going on.

Another example of promise-based tests is in the Mocha docs themselves; that documentation also links to an interesting library for assertions in this style of test.

Tests using promises can also use done in the promise callback instead of returning the promise, although it's more boilerplate and I'm not sure you'd ever need to do it this way:

    it('should be a Array', function(done) {
        input.then(function(value) {
            expect(value).to.be.a('array');
            done();
        }).catch(function(error){
            done(error)
        });
    });

So, hopefully at least one of those ideas will work for you; let me know what you think!

[sgarbesi]

The first example wouldn't work well for my scenario since I need to share the same async value across multiple tests. I would need to re-invoke the require for each of the separate tests. I know in my initial illustration I'm only showing the testArray Function, but there would be many other tests which would run against the async value. If I basically switch to using require inside a before and passed the callback Function to the testArray Function, that may work. It just seems like an awfully lot of extra unnecessary code to pull this off. Would you not agree?

I'm writing tests for a AMD library. In my scenario, I don't want to use the define Function for these given tests. They're to be strictly isolated to only use the require Function.

As for the promises, I have the same argument/complaint as with the callback Function(s).
I also have to provide browser support going back to IE6 (fun times, I know.)
I could use a polyfill, but I still feel that it's cumbersome and counter-productive.

Thanks for all the suggestions, the callbacks may be the way to go. As of right now this is an example of what I'm using to get it to work. But in this example the test counter in the console progress output is incorrect. The counter will say 3/1 (instead of 3/3) tests ran successfully, since I'm hacking the test cases in.

describe('test/functional/require/module/anonymous/array', function() {
    'use strict';

    // Run the following operations before the tests are invoked.
    before(function(callback) {
        // Load the mocked module.
        require(['base/test/mock/module/name/array'], function(input) {
            // Define the tests for the asynchronous results.
            describe('The mock module should pass the following tests.', function() {
                // Tests the `input` argument.
                _.testArray(input, 1, ['test']);

                // Invoke the callback.
                callback();
            });
        });
    });

    // Mocha work-around to allow tests to be defined through the `before` Function.
    it('asynchronous');
});

Maybe instead I'll switch to something like below.
I would now need to have 2 testArray Functions.
One for sync tests and one for async tests.
Not the end of the world, but far from ideal.

    // The output value from the async callback.
    var output;

    // Run the following operations before the tests are invoked.
    before(function(callback) {
        // Load the mocked module.
        require(['base/test/mock/module/name/array'], function(input) {
            // Assign the `output` value.
            output = input;

            // Invoke the callback.
            callback();
        });
    });

    // Generate the Array tests.
    _.testArrayAsync(function() {
        return output;
    }, ['test']);

[sgarbesi]

@ScottFreeCode The callback does actually work better, as the test progress indicator actually reflects the correct number of tests which were run.

[ScottFreeCode]

(Editted my initial callback-based example upon realizing I had a little boilerplate in there that wasn't doing anything that couldn't be done directly):

testArray(function(testInput) {
        require(['some/random/library/on/the/fly'], function(libraryValue) {
            testInput(libraryValue);
        });
    }, ['test']);

...is equivalent to...

testArray(function(testInput) {
        require(['some/random/library/on/the/fly'], testInput);
    }, ['test']);

(That's the only change I made.)

[ScottFreeCode]

If you want to use the callback-based approach with the same library and multiple test[some thing] functions, I'd probably do something like this:

// test sets
function testArray(...) {
...same as my earlier callback-based example...
}

function testSomethingElse(...) {
...equivalent in design pattern, but checking something else about it...
}

// testing a specific library
function testSomeRandomLibraryOnTheFly(testSet) {
    testSet(function(testInput) {
        require(['some/random/library/on/the/fly'], testInput);
    }, ['test']);
}

testSomeRandomLibraryOnTheFly(testArray);
testSomeRandomLibraryOnTheFly(testSomethingElse);

Or, to run multiple different sets of tests against multiple different libraries, something to this effect:

// test sets
function testArray(...) {
...same as my earlier callback-based example...
}

function testSomethingElse(...) {
...equivalent in design pattern, but checking something else about it...
}

function testMultipleLibraries(...) {
...check something about two different libraries?...
}

// This reduces boilerplate at the call site for tying a library to a set of tests and expected result.
function testLibrary(testSet, libraries, expectedResult) {
    testSet(function(testInput) {
        require(libraries, testInput);
    }, expectedResult);
}

// libraries to test
var setsForSomeRandomLibraryOnTheFly = [testArray, testSomethingElse];
for (var index = 0; index < setsForSomeRandomLibraryOnTheFly.length; index += 1) {
    testLibrary(setsForSomeRandomLibraryOnTheFly[index], ['some/random/library/on/the/fly'], ['test']);
}

var setsForSomeOtherLibraries = [testArray, testMultipleLibraries];
for (var index = 0; index < setsForSomeOtherLibraries.length; index += 1) {
    testLibrary(setsForSomeOtherLibraries[index], ['some/other/library1', 'some/other/library2'], ['library 1 value', 'library 2 value']);
}

Since you mentioned using both synchronous and asynchronous code, here's a way to work a synchronous value into the same type of callback-based test sets:

// test sets
function testArray(...) {
...same as my earlier callback-based example...
}

function testSomethingElse(...) {
...equivalent in design pattern, but checking something else about it...
}

// This reduces boilerplate at the call site for tying a synchronously resolved value to a set of tests and expected result.
function testSynchronousValue(testSet, value, expectedResult) {
    testSet(function(testInput) {
        testInput(value);
    }, expectedResult);
}

// values to test
var synchronouslyDefinedArray = ['test'] // This is standing in for wherever the value is being synchronously retrieved from.
var setsForSynchronouslyDefinedArray = [testArray, testSomethingElse]
for (var index = 0; index < setsForSynchronouslyDefinedArray.length; index += 1) {
    testSynchronousValue(setsForSynchronouslyDefinedArray[index], synchronouslyDefinedArray, ['test']);
}

(So you'd have two functions, one for asynchronous libaries and another for synchronously resolvable values, but both would be reusable for multiple different sets of tests and multiple different libraries rather than having to write two of every test set that needs to be run asynchronously or synchronously.)

Does that work out better?

[sgarbesi]

@ScottFreeCode I wound up using a wrapper inside of my test Functions to automatically invoke in the input argument if it's flagged as callback. Thanks for the help.

Do you know what their reasoning was for not allowing nested it statements? It seems like this would be a proper use case.

[ScottFreeCode]

Glad I could help you to get it working!

I'm fairly new to Mocha, so I can't really say much as to the design decisions of the developers, but for what it's worth, I don't think nested it would solve what you think it would. (I should add at the beginning here the caveat that all this is assuming I've understood correctly; if I've got it wrong, feel free to provide a non-working example of what your preferred code would be if nested it were allowed, or any other clarification merited -- also, I'm going to be saying things like "you want" a lot not because I mean to tell you what you think but because to discuss some of this I have to address your goals, so I apologize in advance if I infer any of them incorrectly.) Practically, what you seem to want to do by calling nested it and what you did in your hacked version where describe is called inside before are the same: delay the definition of the tests by testArray and similar functions until the asynchronously retrieved value to test is resolved. Yet the reason before counts the wrong number of tests is likely also the same as the practical reason it isn't allowed to nest: Mocha seems not to have been designed to add to the set of tests to be run while in the process of running them. Even if it worked inside another it, I suspect that without redesigning Mocha to properly handle/count tests that are defined during the run you would have the same miscounting problem. It might be a little clearer to write because of being able to call testArray directly instead of inside another describe, I suppose, but for all practical purposes we could just as easily fix calling describe inside before and get the same result -- and there's a problem with calling it inside it on a higher level: namely that you'd effectively be creating a collection of tests, and a collection of tests is normally what a suite (created by describe) is, and the difference between a collection of tests defined by a suite and a collection of tests defined by a test isn't obvious on a conceptual level... Having tests sprouting out of tests makes the test program harder to reason about than it would be with a neat distinction between tests and collections of tests (aka suites).

And for all that, even granting that a test system like Mocha could be designed to allow adding to the set of tests to run in the suites and/or tests, I don't think it's a direct solution to your real goal: dealing with asynchronously retrieved values when using a separate function to define a set of tests that can be applied to more than one library or value. You want a sort of dynamic addition of tests at test runtime, whether it nested in describe nested in before or just it nested in it, because you're trying to make the test definition wait for the asynchronously retrieved value to be resolved. In turn, you want the test definition to be delayed until the variable is ready because your test-defining function will capture the variable's state at the point that definition is called. So you're tackling multiple layers of problems each because the way you tackled the layer below it created more problems; in the final analysis, the last problem created is that Mocha doesn't support dynamic addition of tests at test runtime, but that's only indirectly related to what you were originally trying to do: define tests using a separately encapsulated*1, reusable function, yet test a value that must be retrieved asynchronously.

In contrast, both promises and leveraging the loader would address the asynchronicity directly. Using the loader would explicitly declare that these tests will not be defined till the value is retrieved and the whole set of tests is not run till all the tests are defined (rather than not defining some of the tests till the value is retrieved by other tests being run). Using promises would, likewise, explicitly declare that the value being tested will in fact be resolved asynchronously; the need to delay the definition itself is then eliminated neatly. Both are obvious and conceptually simple (even if not necessarily simple to implement); more importantly, both tackle the problem of asynchronicity head-on rather than trying to work around it. I don't think I can say that about either nested it or describe called in before -- in fact, I don't even think nested it is as clear a way to address the issues as using a property in order to refer to the value updated by a blocking before (as in your first post) would be (assuming that could be made to work).

(And callbacks? They're really just a poor-man's ad-hoc substitute for promises. It's conceptually the same: instead of passing the value, pass an object that can handle a callback to use the value when the value is resolved. It's just that the object passed is itself, directly, another callback; it reads more like purely layers of callback nesting -- callbacks taking callbacks! -- and, because of lack of encapsulation*2 of that model, requires a little more boilerplate.)

So, there's the long version of my opinion on it. As with all programming, ultimately, it comes down to tradeoffs and you can choose what matters most to you. As I alluded to initially, I'm really more of a clever outsider when it comes to Mocha; so, maybe the team will decide there are good reasons to define tests during the run, whether this scenario is one of them or not. All I can do is lay out how I reason about it, which solutions I think are better and why, and hope my thoughts on the matter are at least helpful in some way. ;^)

*1, 2 I say "encapsulation" here meaning not data hiding but rather organization of code so that a set of functionality or even a design pattern is written in one place in a decoupled manner and referred to or applied multiple times rather than being written multiple times; I suppose there's probably a better (or at least less ambiguous) term for that, but I can't recall one off the top of my head.

[ScottFreeCode]

So, apparently it's been too long since I looked closely at the Mocha documentation, because I was just now playing around with some ideas, tried to figure out how to get a commandline option to work, and thanks to reading the list of commandline options I discovered I'd been missing the --delay flag, which if I understand the documentation correctly makes this much easier to do:

var testArray = function(input, results) {
    it('should be a Array', function() {
        expect(input).to.be.a('array');
    });

     if (Object.prototype.toString.call(results) === '[object Array]') {
        it('should contain `' + JSON.stringify(results) + '`.', function() {
             expect(input.join()).to.equal(results.join());
        });
    }
}

require(['some/random/library/on/the/fly'], function(libraryValue) {
    describe("whatever suite(s) this is in, or remove this if none", function() {
        testArray(libraryValue, ['test']);
    });
    run();
});

I'd have to do some digging to find out if this is supported in the in-browser HTML reporter.

[danielstjules]

#2221 (comment) should work :) In regards to using delay in the browser, just be sure to watch out for #1799 (order matters due to an unfortunate bug)

[danielstjules]

Also, here's the relevant PR that introduced the feature:
#1439

Thanks @ScottFreeCode for your help! :)