Web compatibility and ESM in .js files

[GeoffreyBooth]

This issue is a summary of a discussion starting at #142 (comment). There was a lot of debate on that thread; if you’d like to comment, please do so over there. The point of this issue is only to serve as a neutral document of the issue discussed in that thread. There’s also a separate thread for discussion of potential solutions, should the group decide to change anything.

The problem: ESM JavaScript in .js files

• Node, in --experimental-modules, always treats files with .js extensions as CommonJS files. Within an ES module, an import statement for a .js file will always try to parse that file as CommonJS; if the file is actually ESM—for example, it contains import or export statements—the import will error. ESM files must be named with an .mjs extension to be importable via an import statement in Node --experimental-modules.

• The Web always treats .js files as ESM. (This is an oversimplification, but the detailed explanation can be found in the next section.) An import statement of an URL that a webserver resolves to a .js file will be parsed by browsers as ESM.

• If the .js file being imported is ambiguous, and could be parsed as either CommonJS or ESM, it will be executed differently in Node versus in browsers. Browsers will run it in strict mode, because all ESM is strict mode, while Node will run it in sloppy mode, the default for CommonJS. Since the exact same line of code to import this file can run in both environments—import './ambiguous.js'—this is a situation where identical code runs differently in Node as in browsers (as opposed to not running at all in one environment or the other). This is an incompatibility with the Web, and one of the primary goals of Node’s ES modules implementation is to achieve equivalence and compatibility with ES modules running in browsers.

• More generally, the Web allows .js files to contain import and export syntax, whereas --experimental-modules does not. (Again, this is an oversimplification which will be explained below.) There are reasonable use cases where users targeting the Web may want (or even need) to save their ESM JavaScript in files with .js extensions. If Node disallows ESM JavaScript from .js files, this introduces another incompatibility with the Web.

But browsers don’t care about file extensions!

Indeed, they don’t. But they care about MIME types, and the simple version is that MIME types are to browsers (in ESM mode) as file extensions are to Node. When a browser evaluates import './file.js' in some ESM-mode JavaScript code, it asks a webserver for ./file.js (which is a relative URL, like what you see in <script type="module" src="./file.js">). The webserver resolves file.js however it deems fit to—maybe it finds a file named file.js, maybe it’s an API listening for a path named /file.js, maybe via some other method—and the server returns a string of JavaScript code and some headers. One of those headers must be Content-Type, and must contain one of the ESM-approved MIME types such as text/javascript or application/javascript. If it does, the browser will interpret that string as ESM-mode JavaScript and run it. If the MIME type is missing, or is a type that browsers don’t recognize (such as application/node, the MIME type for CommonJS) an error is thrown.

JavaScript is a static asset, so most of the time a webserver (or CDN or similar) will be serving it from a file. And the server needs to decide what MIME type to choose when it serves that file. Most, if not all, servers make that decision based on file extensions. And most will look at the .js extension and serve it as text/javascript, which browsers in ESM mode will treat as ESM. Webservers will also serve .mjs as text/javascript, which browsers will also treat as ESM. The .js and .mjs extensions are interchangeable as far as webservers and browsers are concerned. These file extension-to-MIME type mappings are usually configurable, though obviously not every user will have sufficient access to change these settings.

Node --experimental-modules as compared to browsers and webservers

Node choosing how to parse a file, for example deciding if a .js file should be treated as CommonJS or as ESM, is equivalent to the webserver deciding what MIME type to serve for the file: application/node (CommonJS) or text/javascript (ESM). Because no browser supports CommonJS, though, there are no known webservers in the world that serve .js as application/node. So webservers generally serve all .js files as text/javascript or application/javascript, which are interchangeable.

The complication for --experimental-modules is that webservers also serve .mjs as text/javascript. So both .js and .mjs are served with the same MIME type on the Web, and therefore webservers aren’t disambiguating anything beyond that the string to be served is some form of JavaScript. You can save an ESM JavaScript file with either a .js or an .mjs extension, and either will work just fine on the Web. Node’s --experimental-modules, on the other hand, doesn’t treat these identically—it loads .js as if it were a webserver serving it as application/node, a.k.a. CommonJS, so if a .js file contains import or export statements, that file will throw an error.

Just show me some code!

Here’s a demo you can run. The JavaScript code in that repo runs in both Node --experimental-modules and in browsers (as served by a typical webserver) and evaluates differently in each environment. The README shows the expected output, if you’d rather not clone the repo and run it yourself.

What about the spec?

The relevant specifications are silent on this matter. The import specifier (the string in import './file.js') can be anything per the JavaScript spec, and the browser spec currently requires it to be an absolute or relative URL. Node --experimental-modules essentially allows most of what require allows, and Node decides this on its own. There’s work to bring package imports, like import _ from 'lodash', to the Web via the package name maps proposal. There is no spec that governs what MIME type should be chosen for a file extension, nor could there by one, as JavaScript has several that are each valid depending on context.

It is this ambiguity that allows --experimental-modules to have its current behavior without violating any specs, though its behavior is clearly different than that of browsers and webservers.

So what should we do about this, if anything?

Because this isn’t meant to be a discussion thread, please use this thread to discuss potential solutions (or arguments for keeping the --experimental-modules current behavior).

[devsnek]

Because this isn’t meant to be a discussion thread, please use this thread to discuss potential solutions.

Why are we discussing solutions to something we haven't agreed is a problem?

[GeoffreyBooth]

Why are we discussing solutions to something we haven’t agreed is a problem?

https://github.com/GeoffreyBooth/modules/blob/85f0bffde459f82e1603a149d60d2ca7bec4105d/proposals/esm-in-js-files.md#do-nothing

[devsnek]

No. That's a red herring. If we agree it's a problem, then the logical progression would be to find a solution. If we don't agree it's a problem, then we do nothing.

Until then it's a waste of group member's time to encourage finding solutions, because it's not a problem yet. You can find solutions on your own (which it looks like you're doing) but we have a lot of stuff to focus on already without more things piled on top. Couldn't this be a discussion topic for tomorrow? It's <24 hours away...

edit:
to clarify, i'm asking for this issue to be marked as agenda, and for the "discussion thread" to be removed until we decide its something where solutions need to be discussed.

[bmeck]

I think this issue is just recording a need for there to be a way to allow .js to be ESM, just like we probably need to allow .js to be a variety of other things. The way in which that is achieved it seems like is being moved to #150 as this just seems to be an issue compiling records of the use case needs. There are tons of ways to achieve this, many are not ambiguous in terms of file format and have backwards compatibility. We can discuss the validity of the use case here, but it would be hard to disprove since the issue is talking about cases where they are stuck with existing behavior, similar to how Node has backwards compat with CJS.

[MylesBorins]

Closing this as there has been no movement in a while, please feel free to re-open or ask me to do so if you are unable to.