Define a policy for the packaging of tests

[AdrianVasiliu]

Would be nice to define a "standard" packaging for tests (both unit and functional tests).

Currently, the packaging is not consistent between delite and deliteful:

• delite: intern-based tests (both unit and functional) are at the root of delite/tests
• deliteful : intern-based tests are in tests/intern; some non-intern tests are at the root of deliteful/tests.

Some points:

• As long as all tests are intern-based, there doesn't seem to be a need for a subdirectory "intern" inside "tests"?
• Should functional and unit tests be intermixed in the same directory? Or distinct directories would make the picture clearer?
• Should there be a subdirectory grouping the tests of a given module? (Currently no subdirectory in both delite and deliteful). Subdirectories might be nice in case there will be a very large number of tests. On the other side, subdirectories don't fit that well for functional tests that cover several modules, not just one.

[cjolif]

deliteful : intern-based tests are in tests/intern; some non-intern tests are at the root of deliteful/tests.

My guess is that this is just temporary and that in the end this should ends up like delite.

As long as all tests are intern-based, there doesn't seem to be a need for a subdirectory "intern" inside "tests"?

I think there are still some manual (not intern not doh) tests which might explain why this is like that to differentiate those ones from the intern ones.

I would personally prefer having functional and unit tests in 2 different sub directories but I have no real issue having them all together as in delite. If nobody is pushing hard for that I guess the easiest is to keep it as it is.

[pruzand]

My preference would be to have the unit tests and functional tests split
into their own directory. Is there a technical reason to have them mixed
together today in the same directory ? (like some dependenciess between
some unit tests and functional tests or the ability to run them in a row ?)

On Thu, Jan 9, 2014 at 8:23 PM, Christophe Jolif
[email protected]:

deliteful : intern-based tests are in tests/intern; some non-intern tests
are at the root of deliteful/tests.

My guess is that this is just temporary and that in the end this should
ends up like delite.

As long as all tests are intern-based, there doesn't seem to be a need for
a subdirectory "intern" inside "tests"?

I think there are still some manual (not intern not doh) tests which might
explain why this is like that to differentiate those ones from the intern
ones.

I would personally prefer having functional and unit tests in 2 different
sub directories but I have no real issue having them all together as in
delite. If nobody is pushing hard for that I guess the easiest is to keep
it as it is.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-31966474
.

Patrick

[wkeese]

Note that there are three types of files:

• myUnitTest.js
• myFunctionalTest.js
• client.html

myFunctionalTest.js loads client.html and runs tests on it, but client.html could (and likely will) also be used as a stand-alone manual test. The accessibility folks, for example, always do that.

So when you talk about "splitting unit tests and functional tests", it's more complicated than it sounds. That's why I would just leave them all in one directory. But I really don't care. What directory structure would you prefer for the three files I listed above?

[AdrianVasiliu]

By "client.html" you mean a non-intern functional test (say, myFunctionalTest.html)? In this case it could go together with myFunctionTest.js in a "functional" directory. But maybe you mean the same html file would also be loaded by myUnitTest.js? Looks a bit strange that a unit test would rely on a functional test but might happen, I don't know.

Anyway, one practical concern would be that, the day the "tests" directory will contain tons of files (unit and functional tests plus various auxiliary files used by these tests), it will be a bit hard to determine for instance if no unit or functional test has been forgotten for inclusion in unit.js and functional.js. Even with distinct directories for unit vs functional it wouldn't be easy because of the "pollution" by auxiliary files, but a bit less I'd say.

For that, maybe the separate subdirectory for the tests of each module would help - although it's unclear where to put functional tests that cover more than one module.

Also, especially if we keep all tests in the same directory, maybe it would help to have a file naming pattern allowing to distinguish a) real tests vs auxiliary files, and b) unit vs functional tests? (for instance, file name suffixes such as -aux, -unit and -functional)

[wkeese]

By "client.html" I just meant the file that was loaded by "myFunctionalTest.js". You could call it "myFunctionalTest.html" if you wanted although that might not make sense if myFunctionalTest.js loaded multiple HTML files, or conversely if client.html was loaded by multiple functional tests. Dijit did that, for example how the Button_a11y and Button_mouse functional tests both load test_Button.html.

I agree the test directory might get crowded. I suppose Chris might lobby for putting each widget's tests inside the widget's directory, i.e. inside Button/tests.

[pruzand]

I agree the test directory might get crowded. I suppose Chris might lobby
for putting each widget's tests inside the widget's directory, i.e. inside
Button/tests.

Having the tests in the widget directory would indeed solve the "get
crowded" issue while allowing the use case you mentioned (consumin the html
from the unit or/and functional tests). Personaly I would be in favor of
such configuration, but I'm not sure how it could apply to delite where
there's no such feature-specific dir structure.

On Sat, Jan 11, 2014 at 11:48 AM, Bill Keese [email protected]:

By "client.html" I just meant the file that was loaded by
"myFunctionalTest.js". You could call it "myFunctionalTest.html" if you
wanted although that might not make sense if myFunctionalTest.js loaded
multiple HTML files, or conversely if client.html was loaded by multiple
functional tests. Dijit did that, for example how the Button_a11y and
Button_mouse functional tests both load test_Button.html.

I agree the test directory might get crowded. I suppose Chris might lobby
for putting each widget's tests inside the widget's directory, i.e. inside
Button/tests.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-32092743
.

Patrick

[cjolif]

I guess it is (a bit) harder to strip out each tests directory in each widget than a single one or at least one per major directory when it comes to distribution. Also as the main file of the widget is not in the dir of the widget itself it is not like if we had really standalone dirs... so I'm not sure we should put test in there (also causing the issue of functional tests using several widgets)

My feeling is that if we correctly unit test things and correctly automatize them a crowded directory is less of a problem than when (in dojox/mobile) we had to manually explore a crowded directory because we don't really look into it and just run automated tests.

Also I think some big deliteful components like List or Tree (basically the ones that needs more than one module to work, like the List widget, the list item widget, the editable mixin...) deserves their own sub-package:

deliteful/list/[ListClasses]
deliteful/tree/[TreeClassses]

in which case they might get their own tests directory in their as well?

deliteful/lists/tests
deliteful/tree/tests

removing some of the burden on the main test directory.

Finally as I think the unit & functional tests can finally be split (as Adrian said only functional is importing the HTML)

I would do:

deliteful/tests/unit.js
deliteful/tests/runXXXX
deliteful/tests/functional.js

deliteful/tests/unit/myUnit.js
deliteful/tests/functional/myFunctional.html (even if that is used standalone this is still some kind of functional tests not a unit test for sure)
deliteful/tests/functional/myFunctional.js

and similarly for deliteful/list/tests etc...

[pruzand]

But in this case, we don't reallly address the the idea of providing self-contained Widget packaging as ccmitchellusa proposed (ibm-js/delite#14) which was the reasons of the per-widget directory structure. It sounds weird to me to have css, nls,etc resources but not the tests (at least the unit tests, which are really widget-specific resources; I agree that functional tests dependending on other widgets could be an issue in this scenario).

[pruzand]

After discussing with Christophe, it appears a 100% self-contained widget approach is not really settled as it would be very difficult to achieve in facts. So, while unit tests are indeed in majority widget-specific with no cross-dependencies (there are some exceptions though), the functional tests + samples are not. Having different rules for funtional tests, samples, and units would eventually add extra complexity to the project.

[cjolif]

Ok so seems there are no objections to #2 (comment) based on that I suggest we adopt it.

For samples I suggest we have a samples directory just as we have a tests directory at the top level.

For the test and sample naming pattern we can just continue using what we have in delite that is

WidgetName(-possiblesubtestssampledetails).[js|html]

[sbrunot]

Rather than having a unit.js and a functional.js file in the tests directory, I think it would be better to have a tests/unit/all.js file and a tests/functional/all.js file. Note that it seems to be the solution retained for dojo:

https://github.com/bryanforbes/dojo/blob/intern-conversion/tests-intern/unit/all.js

[pruzand]

I don't have strong opinion on this, I understand the reasoning of the dojo code layout, but what I like with the current structure that has been decided is it is clean: unit/ and functional/ contains tests (and only test files), and the root tests/ dir holds the scripts/config file required to run them.

[sbrunot]

.js files under tests/unit and tests/functional are test suites. all.js is also a test suite, that contains all the tests defined in all the other test suites. Intern will accept any of the unit/.js and functional/.js files as a test suite. This is a pretty common standard with other languages and test frameworks (I'm thinking Java and JUnit there), so that might explain why it has also been choosen by the dojo team.

[cjolif]

thanks @sbrunot for the suggestion. So if there is no objection let's move to this updated decision:

delite(ful)/tests/unit/all.js
delite(ful)/tests/unit/MyClass(-options).js
delite(ful)/tests/functional/all.js
delite(ful)/tests/functional/MyClass.html (even if that is used standalone this is still some kind of functional tests not a unit test for sure)
delite(ful)/tests/functional/MyClass(-options).js

which complies to dojo in terms of layout. And this time let's stick with it ;)

@lbod are you still ok to work on fixing that structure accordingly for both delite & deliteful?

[lbod]

Yes fine with that, I spent time on it over the weekend but running into a few issues mapping dojo/dcl etc

I'll let you know how I get on but may need someone more learned than me to help

[cjolif]

thanks @lbod don't hesitate to drop me a mail if you have questions.

[cjolif]

see ibm-js/deliteful@d966cd3 for deliteful