Some thoughts on AVA

[tomazzaman]

I went through the code yesterday and have a pretty good idea of it's inner workings so I'd like to give a bit more extensive input and suggestions how it could be made a bit simpler and less opinionated (mostly about babel):

• I would drop babel support almost entirely. If babel was necessary, then a babel: true key would be added to the configuration and AVA would take my project's babel configuration (either in .babelrc or package.json) to transpile files. This would allow me to to use babel in exactly the same way I use it in development - meaning I could easily use JSX in tests. I configured babel to best fit my needs and I don't like AVA forcing me its configuration which is different.
• in AVA's configuration, I'd have a sources key (glob) which would make AVA transpile sources (and cache them) before running any tests. This would be great for the runner's performance because it wouldn't need to require babel runtime but would just load the files from cache. Requiring the runtime with every forked process adds roughly 300ms to execution time, regardless whether any files were changed from the previous run or not. It's unnecessary imho.
• in turn, this would make running test for babel users faster as well as the runner would only transpile changed files and since tests files are forked this would be done in a separate process and thus non-blocking
• make both test and source files go through a watcher which would be the only point of caching/transpilation. Depending on the project size it would take a bit to transpile for the very first time (because cache would be filled after that) and then each subsequent run would be fast. As a babel user, I'm willing to accept that, and non-babel users will get a smooth experience like they do now.

Here's how my development process would look like ideally (with babel support):

• I run AVA in watch mode
• AVA transpiles tests and sources (with my own babel config) and creates cached, transpiled versions with sourcemaps (in ./node_modules/.cache/ava)
• on every save of either sources or tests, AVA updates the cache for that file only, then runs the suite. It's fast because AVA only transpiled the changed file
• Once I'm done with development, I run AVA on single-run mode and don't really care how long it takes, I just need the final report. In most cases this step would be skipped because I know CI will run and fail to deploy if anything goes wrong.

Let me know what you think.

[vadimdemedes]

Thank you for taking the time & effort to dig into AVA and report your thoughts back to us, we really appreciate that.

Below are my comments on your suggestions/statements.

I would drop babel support almost entirely

At the moment, I don't see that happening, as ES2015 support in tests out-of-the-box is one of they key selling points of AVA. Also, rich assertions rely on babel too. So removing babel would also mean removing one of the coolest features.

If babel was necessary, then a babel: true key would be added to the configuration

I was actually thinking to do the opposite (#424). Be able to turn off AVA's babel, so that user is free to set it up however he wants. Imho it would satisfy both sides.

in AVA's configuration, I'd have a sources key (glob) which would make AVA transpile sources (and cache them) before running any tests.

This essentially is related to the previous answer. However, I'd like to note that we always transpile and cache test files before running any tests.

Requiring the runtime with every forked process adds roughly 300ms to execution time, regardless whether any files were changed from the previous run or not

It's not true anymore, since we added babel caching to AVA. If there's a cached version of a file, we don't even require babel at all.

Hope I clarified a few things, let me know if you've got any more questions or suggestions.

[tomazzaman]

About caching, I've noticed AVA caches test files, but not source files, maybe I missed that?

[vadimdemedes]

Yes, exactly, AVA transpiles & caches only test files, not source files.

[novemberborn]

@tomazzaman I've opened #577 for your source file transpilation suggestion. Watch mode is in 0.12 now (undocumented) and #448 would allow Babel to be disabled if you so desired. Thanks!