Set up testing

[sgomes]

• Investigate alternatives
• Investigate Chrome headless
• Choose something

[Garbee]

AVAJS is an alternative to mocha. I really like how it is kept simple and allows for concurrent test files to run. Tests shouldn't take forever to run, this helps speed them up even if you have tons of files.

[traviskaufman]

I would love to use Ava but the problem with it is that browser testing is unsupported. Normally I'd be cool with using JSDom but given that we're building user interface components in a vanilla manner, I feel like we really should test in all of the browsers we support. I'm subscribed to avajs/ava#24 but it's marked as low priority.

[Garbee]

Ah yea, if we start testing across multiple browsers specifically AVA won't work right now. Which, I'm a big fan of testing across platforms. I haven't personally used any tooling for that, so no immediate recommendations come to mind for it.

[traviskaufman]

Testing Options

Karma + Mocha + Isparta

In the wild

• http://mike-ward.net/2015/09/07/tips-on-setting-up-karma-testing-with-webpack/
• https://github.com/mvader/react-es6-webpack-karma-boilerplate

Pros

• Highly configurable
• Robust, mature tools that other UI frameworks rely on (e.g. Angular)
• Works across all browsers when combined with BrowserStack / SauceLabs
• Gives us pretty much everything we'll ever need in terms of tooling between sinon, chai,
  mocha, and isparta

Cons

• Complex setup - lots of interdependent moving parts
• Karma and Mocha are a bit crufty and slower than some leaner alternatives

Karma + Tape + Isparta

In the wild

• https://rmurphey.com/blog/2015/07/20/karma-webpack-tape-code-coverage
• https://bl.ocks.org/insin/593d7d70331f95bf62ff

Pros

• People seem to like tape more
• Comes with a powerful built-in assertion library, removing the need for chai.
• Simpler to understand and therefore easier to use.
• Modular; can be easily extended to facilitate things such as Promises (see blue-tape), esnext (see babel-tape-runner), etc.
• No global pollution
• No shared state

Cons

• Modularity comes at the cost of having more dev dependencies for tape
• Newer and (arguably?) less battle-tested than mocha
• ???

Ava + JSDom + NYC

In the wild

• https://github.com/avajs/ava/blob/master/docs/recipes/browser-testing.md
• https://github.com/avajs/ava/blob/master/docs/recipes/code-coverage.md
• http://wecodetheweb.com/2016/04/19/effortless-unit-testing-with-ava/

Pros

• Simple and ergonomic
• Extremely fast - runs tests in subprocesses
• Isolates global state
• Supports ES2015 out-of-the-box - no need to run webpack
• Supports async tests out-of-the-box
• Excellent watch mode support

Cons

• 🙅 No browser support. Because of the nature of MDL, it might preclude us from using this altogether. Even with JSDom we still need to test functionality in legacy browsers e.g. IE11/etc. I agree in our case that real browser testing requires testing in real browsers

[traviskaufman]

In terms of cross-browser testing for CI, here's a solid comparison of Browserstack vs. SauceLabs vs. Rainforest QA

Browserstack has the highest google trends but SauceLabs looks like it's the favorite.

Headless chrome looks awesome but also looks like there still work to be done before it's stable

[Garbee]

Yah, AVA is out right away due to no browser support. We need something that is cross-browser.

I also think that with our new direction, we shouldn't worry as much whether the tests run super-fast or take some time. As long as they are accurate and we build wide coverage, that is the important thing.

I believe we'd be better off with option 2 since it doesn't have shared global state. It is fairly critical that one test can't cause an accidental collision in another.

[Garbee]

We also need to think about how Closure compiler to check the comments is going to run into our testing. Should we continue with Closure comments? That is another thing to consider.

[traviskaufman]

@Garbee I agree I'm partial to option 2 as well since tape is leaner and seems to require a lot less config song and dance.

re: Closure I spoke with the Angular team and apparently they're working on having Tsickle be able to create externs files from .dts files. Externs files means that we can forego comments and since we'll probably want to write .dts files for the TS community anyway, this could be a good solution that doesn't clutter our code.

[sgomes]

Option 2 SGTM, as it gives us a modern architecture that we can easily integrate with CIs, and use for both headless and cross-browser testing.

Using externs means our code wouldn't be compiled by Closure, but would at least have an interface to Closurized code. I'm OK with that, as out main goal for Closure support is to make sure that Closure code can use MDL, and it sounds like .dts files are becoming a way of having that.

[traviskaufman]

Resolved by #4567