Feature: ESM in .js files

[GeoffreyBooth]

There needs to be a way to tell Node to treat .js files as ESM JavaScript.

Besides the issue raised in #149, there’s the use case of legacy build tools and pipelines that expect only a single file extension for JavaScript. I can give a concrete example for the case of CoffeeScript. The problem is that essentially, the CoffeeScript compiler is a string converter. It takes a string as input, like console.log 'hello!', and returns another string as output: console.log('hello!'); People who use the CoffeeScript compiler as their entire build chain (as opposed to using it within Gulp or Webpack or the like) use it to transpile projects like this:

coffee --compile --output dist/ src/

This command recursively searches src/ for all *.coffee, *.litcoffee and *.litcoffee.md files, runs them through the CoffeeScript compiler, and saves them as *.js files in a mirrored folder tree under dist/.

CoffeeScript doesn’t know the parse goals of the files in src/. It doesn’t know which ones to save with a .js extension versus an .mjs extension. About the only way it could know is if we created new file extensions, like .mcoffee, .mlitcoffee and .mlitcoffee.md, that it knew to output as .mjs. But that’s not a practical solution, as there are thousands of CoffeeScript build plugins and syntax highlighters and so on out there that would need to be updated to support these new file extensions; and let’s be honest, most of those packages are abandoned. Adding a new option, like --output-file-extension mjs, also isn’t realistic for the same reason, because of the sheer number of build plugins that would need to be updated to support it. CoffeeScript traditionally just converts one string into another, without necessarily reading or passing along any other options or metadata, and there are lots of build pipelines that expect this behavior.

Rather than figuring out a way to add a piece of metadata (the parse goal) to CoffeeScript source files, and then updating all other parts of the build chain to preserve and pass along and understand this new metadata, it would be better if the parse goal could be specified outside of the file itself.

There’s already one implementation to achieve this: nodejs/node#18392, which suggests a mode flag for package.json files to tell Node to treat .js files as ES modules within a given package boundary. There’s also the suggestion here expand that idea to create a whole new section in package.json where users can assign MIME types for file extensions, essentially telling Node how to parse any type of file. Either of these solutions would work.

We have the same issue for input that doesn’t come from files, such as from STDIN or via --eval. There needs to be some way to tell Node to treat those inputs as ESM, such as a CLI flag:

node --module --eval 'import path from "path"; console.log(path.sep);'

These suggestions are complements for .mjs, not replacements. These proposals don’t provide easy ways to keep CommonJS and ESM files side-by-side in the same folder, for example. These are solutions for use cases that go unfulfilled by .mjs.

One final note: I don’t claim to speak for other compile-to-JavaScript languages, like TypeScript or JSX, but I wouldn’t be surprised if many of them had similar issues. There are also other categories of build tools and plugins, like linters and minifiers, that may be affected. For maximum compatibility, we should provide multiple ways to signal ESM mode to Node.

[bmeck]

I think this is an interesting and exciting development. It is showing a place where there is no meta-data channel for the ecosystem affected. By being a String only input/output it isn't trying to make claims that the use case needs to be solved within Coffeescript because it simply has no place to put the data.

This feature also has an interesting approach taken in this issue since it isn't trying to solve for all use cases and is simply stating a need for the option to solve the use case, perhaps even with laternative mechanisms. It also has some suggestions of how to solve the particular use case without trying to solve for all use cases and is a good example defining of how a use case is constrained in how it can be solved separate from defining a mandated solution.

I fully think we need to look into configuration of our solutions and we can apply that in a way that works in a configurable/opt-in manner that can be used to give authors/consumers choices. We should take time to think about this very clearly.

@GeoffreyBooth thank you for the comparisons to other situations like STDIN with a similar problem and showing an example of a solution without mandating a specific way to approach it. I really like the presentation of both the problem and the neutral tone to allow maximal design space in solving it.

[WebReflection]

If I can add a use case nobody apparently ever mentioned, node as executable should be able to run as ESM too, same way evaluated code could run as ESM.

There is no extension in executable files, just a she-bang on top.

This is the past:

#!/usr/bin/env node
console.log(__filename);

This should be allowed in the future:

#!/usr/bin/env node -m
console.log(import.meta.url);

[bmeck]

@WebReflection that is quite a different issue, the problem for Coffeescript is that it doesn't involve itself in execution. This makes sense since it is a String->String transformation so it doesn't have to involve intself execution. In addition, it isn't setting the input or output format. Therefore it needs an out of band mechanism that isn't done during execution and is out of band of the output. You can feel free to make a new issue to discuss running files without .js as ESM such as your example, but that is certainly different and we should try to stay on topic. I would note that with your -m solution it is problematic for various places that don't parse the full hashbang from the shell that runs your command.

[WebReflection]

I would note that with your -m solution it is problematic for various places that don't parse the full hashbang from the shell that runs your command.

which one of these various places runs NodeJS 10 or above ?

You can feel free to make a new issue to discuss running files without .js as ESM

It's the same need to enforce ESM for a case that would ignore file extensions, not sure how this can be different.

However, imagine I have an executable called esm that needs node to work as ESM for the provided file, so instead of that problem you mention, you have #!/usr/bin/env esm and the issue is identical with eval or string to string: we need a way to bypass the need of an extension because extensions don't cover all use cases.

Happy to open a new bug about this, but I don't think is really necessary, it was just an extra use case about the need for the --module flag.

[targos]

I would note that with your -m solution it is problematic for various places that don't parse the full hashbang from the shell that runs your command.

which one of these various places runs NodeJS 10 or above ?

@WebReflection Linux shells

[WebReflection]

Linux shells

haha, how embarrassing, I'm a Linux user and indeed it just failed, I guess I've never tried.

Forget about -m on the shebang, the #!/usr/bin/env esm where esm wants to run the executable as ESM is the use case I had in mind, if this makes it a more suitable use case to consider.

Call it #!/usr/bin/env mjs if you prefer, I am sure you got the point.

[demurgos]

However, imagine I have an executable called esm that needs node to work as ESM for the provided file, so instead of that problem you mention, you have #!/usr/bin/env esm and the issue is identical with eval or string to string: we need a way to bypass the need of an extension because extensions don't cover all use cases.

The shebang issue was brought up in other threads. It's inconvenient that some shells do not fully parse the command line but a workaround was proposed. Node can parse the shebang line itself to populate its CLI args. But it would need to be better defined (deal with conflicts between the actual CLI args and shebang).

I brought up the use of multiple executables in another thread. I expressed concerns about scalability, @bmeck too:

one of the troubles with executable scaling is the number of dimensions that you scale to, right now we have node and node_g to add another dimension we would have node, node_g, node_esm, and node_esm_g. In addition, there is an ordering problem with the dimensions if you notice. node_esm and node_g both occupy the suffix, while node_esm_g ~ implies an ordering of mode followed by debug, in order to prevent possible conflicts as you add things like REPL, WASM, and Multimodule goals you need to ensure that the different tokens being used (g, esm, ...) do not conflict.

[targos]

Sure, I got your point but I agree with @bmeck that how to interpret extension-less entry points probably deserves its own issue.

[WebReflection]

@targos fair enough: nodejs/node#49444

[GeoffreyBooth]

As a sort of side note, over the weekend I put together this pull request to add CoffeeScript support to the testing framework TestCafe. I’m not sure why, but TestCafe parses the source code of all the test files it runs, and uses Babel to produce an AST from the code; and TestCafe searches the AST for fixtures and tests to run. TestCafe already supports ESNext JavaScript, that it transpiles through Babel; and it supports TypeScript. Its architecture, though, is that it supports only one file extension for each (.js or .ts); when I added CoffeeScript, I added it under .coffee, leaving out .litcoffee and .litcoffee.md. I’m sure TestCafe could be updated to support multiple extensions for JavaScript, but it currently doesn’t.

There are things like this all across the ecosystem, that expect JavaScript to exist only under .js—and only function with JavaScript under .js, and even support import and export statements there (via things like the Babel transpilation that TestCafe does). Sure, eventually many of them can and will support .mjs; but that will take time, and a lot of effort on the part of a lot of maintainers, whereas adding one of the solutions in #150 takes little effort on Node’s part and gives the ecosystem plenty of time to transition.

[ljharb]

JavaScript, the Script goal, only will - the Module goal is something else, and their parser needs to add support for it. That's what takes time - the extension doesn't make it any harder (I'd guess it makes it easier for most implementations, if anything).

[GeoffreyBooth]

@mhdawson The top post on this thread is the use case we discussed in the meeting today. It looks like it is pretty much written from an implementation-agnostic perspective, though it discusses implementations a little bit in the latter half when I suggest potential solutions for the problem posed in the first half. But if you’re looking for a use case described independent of any particular implementation, the top half of the original post pretty much is that.

It’s also already been added to the README as a feature in our features list.

[MylesBorins]

Is this feature documented? can this be closed?

[GeoffreyBooth]

It’s in the README: https://github.com/nodejs/modules#commonjs-interop

And superseded by #160