New test format

[Teemperor]

As discussed with @sunfishcode a small example how the test would be more confirming to the design document.

Notes:

• The $init function is the placeholder name for a function that is called after the module is loaded (same idea as behind ES6 main or .init_array sections. See Define a start function. design#398 ). All test code (e.g. invoke ...) that was previously floating around without a related module can be moved there.
• (assert_invalid ...) is unchanged.

[jcbeyler]

I like this though for me this would not work in a long term. You'd like in theory to have a debug version that follows this and a non-debug that does not require the testing like this.

So it seems to me that we would need a $init_debug or something. Plus we still need to define correctly how the module defines this $init method, no?

[kg]

I don't think making invoke etc valid inside of function bodies is a good idea. That introduces the concept of host-specific opcodes and I don't think that's something we want.

Right now all our tests are valid module definitions with a bit of non-spec-compliant goo outside for testing, and it's trivial to strip the outside stuff. This approach destroys that.

[rossberg]

The invoke's also make no sense because the exports have been removed in the patch.

I suppose the actual idea would be to use regular calls. But either way, I don't think this is a particularly useful direction:

• For real-world modules, you strongly want to separate tests from the module implementation.
• Most interesting tests consist of assertions, which are not expressible inside a module.

[Teemperor]

Thanks for all the feedback! The initial PR really had a few errors,
so here is the (hopefully better) second version.

The idea of the PR is to remove three things that are AFAIK not
compatible with the design document:

• The global namespace for exports
• AST nodes that are outside of any module like the child nodes of assert_return, invoke and so on.
• The statements assert_return, assert_trap, assert_return_nan and invoke.
  assert_invalid is not targeted in this PR.

The reasons for the removal are:

• It's easier to compile the tests into valid binary modules later because
  features like invoke are probably not representable in the binary format.
• Implementers of wasm compilers/interpreters don't need to implement
  support for above features just for the test suite. They can verify
  their implementation with normal standard-compliant wasm programs.
• The tests are probably shorter if we don't have to export all functions.

The way this PR addresses those issues is that it

• introduces an assert module. This module and it functions replace
  all assert_* statements beside assert_invalid.
• moves all code inside a module. The PR currently proposes a main
  function for that but any other function will also do. This is
  supposed to replace the global namespace, code outside of modules and the invoke statement.
  The interpreter has to call the main function after all modules
  in a file are parsed, so the main function behaves similar to the
  one we all know from C-style languages.

It should be mentioned that the proposed changes are only intended
on making the test programs inside the test directory more like a 'real'
test suite that contains only valid programs or negative tests. See the GCC test suites (e.g.
this test)
or the Plum Hall test suites as examples.

I don't want to propose a testing framework (think xUnit) or any
other new features beside the assert module. I also don't want to
propose main as the official entry point for a non-web wasm program.

I don't have a problem with implementing the proposed changes,
so this PR will hopefully not create any additional workload for anyone.

Feedback is welcome!

NOTE: The first version proposed a _start clone as the entry point
for a test. I changed it to main as I don't think that _start is the
appropriate place to make self-checks.

[kg]

Making the test assertions a part of the compiled module is mostly a distraction. They're not a part of the spec, they're host features. Exhaustively authoring tests as part of the module means that we need to introduce a lot of code into the modules to support the tests, and now the tests are a part of the module so we need exhaustive automated testing for the test framework (since it's wasm and can break, and if it breaks we're no longer actually validating whether the wasm modules work.) We also have to add new features to enable this wholesale transition - how will you write assert_trap assertions as part of a wasm module?

I don't think generating the current test syntax from code is that bad, either. It's awkward, but feasible:
https://github.com/WebAssembly/ilwasm/blob/master/third_party/tests/Puzzle.cs#L111

[ghost]

Support for a debug trap operator seems fine to me. These can be very helpful in complex debug situations, and upper layers might insert them when debugging code, for example to watch an access to a specific memory location etc. The operator could be very simple and just include a text string with no arguments to limit it's effect on code generation, or optionally take some arguments.

[Teemperor]

Woops, totally forgot to update this.

Closing this PR. If someone else has the problem handling the .wast files, then you could look at the wasm-module testsuite-converter which transforms the wast files into a set of positive/negative checks that are compatible with the standard (seems as if other implementations are doing something similar).

@JSStats there is a unreachable now that is good enough to simulate asserts