Native ESM support

[mstoykov]

History

k6 uses a goja as JS VM. Historically goja does not have support for every JS feature, and as such babel has been used for years to supplement that. The way this works is that if goja returns an error while parsing, k6 will try run it through babel (see compatibility mode for more info)

Over the years though all the things babel has been used have been added to goja. Actually goja does now have support for features that the version of babel used by k6 does not have support for:

• Object spread #824 has been supported for years at this point
• Optional chaining #2168 has also been supported for years in goja
• Class private fields #2887 was added along all the class support

All of those are currently not really support in k6 as they are not supported by the internal babel which needs to parse them even if it won't transform them. As such using this features with ESM lead to parsing errors.

Solution:

The solution here is to have ESM support in goja and used by k6. This will:

1. drop babel from the k6 binary
2. drop babel from needing support for whatever k6 needs support. Letting us only worry about how goja supports features.
3. Due to babel being JS that needs to be run by goja
   1. speed up starting times by 200-500ms (in my small level testing).
   2. speed up starting times for big scripts by quite a lot more. Big scripts are > 10k lines and they start to take multiple seconds (sometimes minutes) to be parsed by babel, but goja parses them much faster (<1s).

Status:

There is draft PR to add the functionality to goja, with the following high level problems:

1. It currently doesn't work with eval - which is definitely blocking
2. The current way some of the evaluation happens predates async/await support and is not based on it - it is actually 100% a hack. This seems to also be a reason for somewhat more complicated scripts to not work.
3. Connected to the above the current code does predate async/await and does not have support for top-level-await. This is likely blocking due to goja usually adding support for the full set of features around a given concept when that one is introduced. My experience is that this will be super simple once 2 is done
4. It has only been lightly reviewed by the goja author and maintainer as it predating async/await was a blocking at the time.
5. The API exposed to users of goja (k6 for example) - evolved naturally while I was making progress. This again predates async/await - it even predated modules being asynchronous in the specification, which was added in ES2022. As such at that point the API suddenly bubbled up and is currently - "not great™". As such it will likely be a good idea to try to fix that once the first 3 parts are fixed and some discussion with the maintainer of goja.

There is an even more WIP/PoC PR to k6 core that is very dated at this point. This should be updated along/after the goja PR is in better shape.

Big parts of the original k6 PR have been backported to the current k6 codebase making this a lot easier, but still some work is required.

The major blocking part is the goja PR which also happens to be more complicated than expected and all the issues there are very interconnected and connected to the internals of goja runtime.

Other connected issues with some context on them

• #2674 - currently require is not relative to the file it is written in, but to the one we are evaluating. This is not how commonjs defines it and will be somewhat harder (impossible in some cases?!?) to be implemented once ESM default support is added. Especially if users are mixing ESM and commonjs.
• #2949 - my current WIP code does not require breaking the modules API k6 exposes to extensions and internally. But in order for some additional features - for example one JS module accessing another we will probably need changes.
• #2258 - is mostly closed, the current implementation support differentiating between default and named exports. But the exact API might change with Deprecate old JS Module "API" #2949 being taken into consideration as well.

[tsloughter]

I was hoping to try https://gleam.run/ for writing k6 tests but the internal babel fails to handle the javascript generated by the Gleam compiler. I know this work here and on the roadmap is to remove the need for babel, and I wanted to ask if updating babel was something that could happen sooner and easier to support more Javascript features? (I know zero about this so just assuming that "updating babel" is a thing that would help here and not nonsense :)

[mstoykov]

I would recommend using babel (or another tool) to transpile before running in k6 instead.

"updating babel" turned out to be complicated and having too many unwanted side effects years ago before dropping it was even close to be viable. So that really isn't on the table, especially given that using another tool before k6 isn't at all that complicated ... I expect gleam already requires transpilation from it's language to JS from what I understand so ... so yeah.

Sorry, but we are not trying to "update babel".

[tsloughter]

@mstoykov ok, thanks for the explanation! And the very good point that I can just use babel manually. I'll give that a try and move to k6 Slack if I have any trouble -- not sure if it'll need special configuration to output whats supported by k6.