Improve elm-test.js

[lydell]

Fixes #211
Addresses #442 (comment)

This PR makes a bunch of improvements to elm-test.js:

• Properly validate CLI arguments:

  • Error on unknown flags.
  • Validate flag values early (seed and fuzz must be numbers, report must be console/json/junit).
  • Default flag values early, instead of in code generation or even in Elm.

• elm-test.js is now 100% definitions, except the last line which is main(); to kick things off. Previously, it was difficult to understand in what order things executed. There was some top-level code, a bunch of function definitions, some more top-level code, more functions, and so on. Now, you can follow the call stack from main(). Also, this means that there are way fewer global variables – functions now get passed what they depend on.

• Use async/await where it helps (in preparation for merge conflicts with Moving to async/await: lib/Runner.js #464 – it should be possible to just pick this PR’s code for every conflict in elm-test.js).

• Improved help message. Also, add some aliases, such as -h, for showing help. I find it super annoying when CLI tools make it difficult to find the help – which is the key to all other commands and flags.

• Various cleanups.

Help before:

Usage: elm-test init # Create example tests

Usage: elm-test install PACKAGE # Like `elm install PACKAGE`, except it installs to "test-dependencies" in your elm.json

Usage: elm-test TESTFILES # Run TESTFILES, for example tests/**/*.elm

Usage: elm-test [--compiler /path/to/compiler] # Run tests

Usage: elm-test [--seed integer] # Run with initial fuzzer seed

Usage: elm-test [--fuzz integer] # Run with each fuzz test performing this many iterations

Usage: elm-test [--report json, junit, or console (default)] # Print results to stdout in given format

Usage: elm-test [--version] # Print version string and exit

Usage: elm-test [--watch] # Run tests on file changes

Intermediate help (outdated):

elm-test init
    Create example tests

elm-test
    Run tests in the tests/ folder

elm-test TESTFILES
    Run TESTFILES, for example "src/**/*Tests.elm"

elm-test install PACKAGE
    Like `elm install PACKAGE`, except it installs to
    "test-dependencies" in your elm.json

Options:
    --compiler /path/to/compiler
        Use a custom path to an Elm executable. Default: elm
    --seed INT
        Run with a previous fuzzer seed. Default: A random seed
    --fuzz INT
        Run with each fuzz test performing this many iterations. Default: 100
    --report json
    --report junit
    --report console
        Print results to stdout in the given format. Default: console
    --version
        Print version and exit
    --watch
        Run tests on file changes

commander help:

Usage: elm-test [options] [globs...]

elm-test
  Run tests in the tests/ folder

elm-test "src/**/*Tests.elm"
  Run tests in files matching the glob

Options:
  --compiler <path>              Use a custom path to an Elm executable (default: elm)
  --seed <int>                   Run with a previous fuzzer seed (default: 68917302767402)
  --fuzz <int>                   Run with each fuzz test performing this many iterations (default: 100)
  --report <json|junit|console>  Print results to stdout in the given format (default: "console")
  --watch                        Run tests on file changes (default: false)
  --version                      Print version and exit
  -h, --help                     Show help

Commands:
  init                           Create example tests
  install <package>              Like `elm install package`, except it installs to "test-dependencies" in your elm.json
  make [globs...]                Check files matching the globs for compilation errors
  help [command]                 Show help

[lydell]

@harrysarson I think I’ve addressed all comments now!

[harrysarson]

I do not like that we have a for loop and then a closure that mutates the index variable. My lunch break is almost over though and I haven't come up with a more constructive suggestion yet. Let me go and think about this.

[harrysarson]

We talked about this on discord. @lydell is going to look into some alternatives to minimist

[harrysarson]

Just a couple of nits. I like the flag parsing with commander and I think you have done a nice job of hacking around/dealing with its shortcomings such that is is clear what is going on.

Have you tested these changes against any project test suites?

[lydell]

Have you tested these changes against any project test suites?

I’m not sure what to test?

[harrysarson]

I was just thinking

cd /some/ideally/complex/project
/path/to/test/runner/bin/elm-test 

as a sanity check.

[lydell]

Ok! I tried that at work and in the elm-review repo and it worked fine. --watch worked too.