Skip to content
This repository has been archived by the owner on Sep 2, 2023. It is now read-only.

Rewrite readme #482

Merged
merged 15 commits into from
Feb 22, 2020
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 38 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,75 +2,64 @@

## Purpose

The Node.js Modules Team maintains and actively develops the ECMAScript Modules (ESM) implementation in Node.js Core.
The Node.js Modules Team maintains and actively develops the ECMAScript Modules (ESM) implementation in Node.js core.

Work includes:

* ESM Loader implementation
* ESM Module Specifier Resolution Algorithm
* Loader Hooks
* ESM and CommonJS Interoperability
* Node.js and Browser interoperability
* VM Modules implementation
* ESM loader implementation
* ESM specifier resolution
* Loader hooks
* ESM and CommonJS interoperability
* Node.js and browser interoperability
* VM modules implementation

## Current Efforts
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if we could include a Current Status section before all the experimental / new stuff, that captures what the group has already achieved.

Copy link
Contributor

@DerekNonGeneric DerekNonGeneric Feb 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It sounds like a good idea, I'm just a bit unclear on what you have in mind. What would the different statuses be? Here's an idea I had for displaying different statuses of various features, it might be similar to what you're after.


> ### [Current Plan for ECMAScript Modules support in Node.js](./doc/plan-for-new-modules-implementation.md)
### To Do / In Progress
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be very blunt then, since you have asked me to be, I simply do not think this content is suitable for our readme. Since it is rapidly changing, likely to get out of date, and not the right information for someone "interested in the Node.js modules group".

As said before, I think we should separate this out into roadmap.md and previous-work.md.

Copy link
Contributor

@DerekNonGeneric DerekNonGeneric Feb 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest commit in my PR to this branch may be what you're after.

GeoffreyBooth@eabe9c2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As said before, I think we should separate this out into roadmap.md and previous-work.md.

Would it be better to have something like a project board for the roadmap? So it can properly link to issues/PRs, including automatic status updates as things get closed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkrems, the argument against that approach, which was brought up to me when I proposed the exact same idea to @GeoffreyBooth, is that these things need to be archivable and a more-static public record of them needs to be kept.


A new ECMAScript modules implementation shipped behind the `--experimental-modules` flag in Node.js 12.0.0. The flag was dropped in Node.js 13.2.0, although the implementation remains experimental and is still subject to change. A road map of the implementation’s development is [here](./doc/plan-for-new-modules-implementation.md).
* [Loader hooks](https://nodejs.org/api/esm.html#esm_hooks) (work in progress). The goal is to create hooks that provide enough functionality to enable users to build loaders that can achieve many of the remaining [features](./doc/features.md) we hope to support.
- Overview: https://github.com/nodejs/modules/issues/351.
- A [design](https://github.com/nodejs/node/pull/30986) that enables [transpilation](https://github.com/nodejs/modules/issues/96) and [arbitrary sources](https://github.com/nodejs/modules/issues/97) and [specifier customization](https://github.com/nodejs/modules/issues/110) shipped in Node 13.7.0.
- **Status**: Work remains to be done to support further use cases including [code coverage/instrumentation](https://github.com/nodejs/modules/issues/95) and [mocks/stubs](https://github.com/nodejs/modules/issues/98).

* [WASM modules](https://nodejs.org/api/esm.html#esm_experimental_wasm_modules) (work in progress).

## Features
### Wish List (Blocked)

Based on [these use cases](https://docs.google.com/document/d/10BBsIqdAXB9JR2KUzQGYbCiVugYBnxE4REBakX29yyo/edit) ([#55](https://github.com/nodejs/modules/issues/55)), our implementation aims to support the following features (subject to change):
* Named exports for CommonJS modules, e.g. `import { shuffle } from 'cjs-pkg'` in addition to the currently supported `import _ from 'cjs-pkg'`.
- Overview: https://github.com/nodejs/modules/issues/264.
- Discussion: https://github.com/nodejs/modules/issues/448.
- Proposal: https://github.com/nodejs/modules/issues/252.
- **Status**: As far as we can tell, currently there is no spec-compliant way to achieve this.

### Baseline Modules Implementation Features:
* Spec compliance ([#132](https://github.com/nodejs/modules/issues/132))
* Browser equivalence ([#133](https://github.com/nodejs/modules/issues/133))
* Don’t break CommonJS ([#112](https://github.com/nodejs/modules/issues/112))
* No refactoring ([#87](https://github.com/nodejs/modules/issues/87))
CommonJS is not statically analyzable (that’s one of the primary benefits of `import`/`export` syntax) and therefore the exported names cannot be known to Node.js simply by parsing JavaScript source code: the code needs to actually be evaluated in order for Node.js to discover what identifiers get attached to `module.exports` for each file. But evaluating the CommonJS code in what the ECMAScript Modules spec calls the “linking” phase (when the module graph is determined) is not permitted; this would mean that CommonJS code is evaluated early, before any ES module code, whereas the spec defines the order of modules’ evaluation based on the depth of imports. This early evaluation presents other issues in that if a dependency gets converted from CommonJS to ESM, its order of evaluation would change in the user’s application; and early evaluation might trigger side effects if reading the values from `module.exports` were getters or proxies.
Copy link
Contributor

@guybedford guybedford Feb 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and early evaluation might trigger side effects if reading the values from module.exports were getters or proxies.

This is a separate point that isn't about early evaluation. This is specifically about Object.values(module.exports) having side effects, whereas those side effects would never otherwise trigger.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please rephrase the text so that it’s correct? Maybe offer a suggestion?

Copy link
Contributor

@guybedford guybedford Feb 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
CommonJS is not statically analyzable (that’s one of the primary benefits of `import`/`export` syntax) and therefore the exported names cannot be known to Node.js simply by parsing JavaScript source code: the code needs to actually be evaluated in order for Node.js to discover what identifiers get attached to `module.exports` for each file. But evaluating the CommonJS code in what the ECMAScript Modules spec calls the “linking” phase (when the module graph is determined) is not permitted; this would mean that CommonJS code is evaluated early, before any ES module code, whereas the spec defines the order of modules’ evaluation based on the depth of imports. This early evaluation presents other issues in that if a dependency gets converted from CommonJS to ESM, its order of evaluation would change in the user’s application; and early evaluation might trigger side effects if reading the values from `module.exports` were getters or proxies.
CommonJS is not statically analyzable (that’s one of the primary benefits of `import`/`export` syntax) and therefore the exported names cannot be known to Node.js simply by parsing JavaScript source code: the code needs to actually be evaluated in order for Node.js to discover what identifiers get attached to `module.exports` for each file. But evaluating the CommonJS code in what the ECMAScript Modules spec calls the “linking” phase (when the module graph is determined) is not permitted; this would mean that CommonJS code is evaluated early, before any ES module code, whereas the spec defines the order of modules’ evaluation based on the depth of imports. This early evaluation presents other issues in that if a dependency gets converted from CommonJS to ESM, its order of evaluation would change in the user’s application.
In addition to these issues of execution ordering, there is also the problem of reading those named exports. The way ES modules define their exports still means we would be applying an `Object.values(module.exports)` for all CommonJS modules as soon as they finish executing, in order to populate the ES module exports. This can lead to a number of problems:
* The triggering of getters and proxies (eg for dynamic generation / deprecation warnings) can throw errors or cause unwanted side effects.
* New exports added to `module.exports` after this time would never be available as named exports.
* Mutations to `module.exports` cannot be kept live with the ES module exports, unless we turn all CommonJS `module.exports` into a proxy. This in turn could have backwards compatibility issues.


#### Browser Interop:
* Browser and Node compatibility without building ([#107](https://github.com/nodejs/modules/issues/107))
* Browser-compatible specifier resolution ([#109](https://github.com/nodejs/modules/issues/109))
* Browser-compatible builds for ESM and CommonJS ([#108](https://github.com/nodejs/modules/issues/108))
A [“dynamic modules” proposal](https://github.com/nodejs/modules/issues/252) was created and presented to TC39, the standards body responsible for the ECMAScript Modules spec. This was a proposal to allow new export names to be discovered during the execution phase, which happens after linking. This made named exports from CommonJS work, but had the caveat that `export *` would not be supported for CommonJS sources. Without a solution for this edge case, TC39 deemed the proposal inadequate. There are currently no other proposals in development to try to surmount these obstacles.

#### CommonJS Interop:
* Named exports when importing CJS ([#81](https://github.com/nodejs/modules/issues/81))
* Multi-mode packages ([#94](https://github.com/nodejs/modules/issues/94))
* Transparent interoperability for ESM importing CommonJS ([#100](https://github.com/nodejs/modules/issues/100))
* Consumer-agnostic imports ([#105](https://github.com/nodejs/modules/issues/105))
* Mixed module types within app/module; gradual migration from CommonJS to ESM ([#99](https://github.com/nodejs/modules/issues/99))
* ESM in .js files ([#151](https://github.com/nodejs/modules/issues/151))
The approach taken by the [`standard-things/esm`](https://github.com/standard-things/esm) package is to transpile ES module sources into CommonJS on the fly, and have Node.js evaluate the CommonJS sources. This is not considered a viable option for Node.js because it means that the actual source code being executed by the V8 JavaScript engine is not the code that the user wrote.

### Existing Node.js Utility Features:
* `require` of ES modules: Support using `require` (in addition to `import()`) to load ES modules into a CommonJS context.
- Discussion: https://github.com/nodejs/modules/issues/299.
- PR: https://github.com/nodejs/node/pull/30891.
- **Status**: Blocked due to various issues discussed in https://github.com/nodejs/modules/issues/454.

* Importing non-JavaScript files ([#115](https://github.com/nodejs/modules/issues/115))
* NodeJS contextual pathing use cases ([#121](https://github.com/nodejs/modules/issues/121))
* ESM in executable files ([#152](https://github.com/nodejs/modules/issues/152))
* Callable resolver ([#157](https://github.com/nodejs/modules/issues/157))
### Not Planned

### Loader Features:
* Limited module type detection.
- PR: https://github.com/nodejs/ecmascript-modules/pull/69.
- Upstream PR: https://github.com/nodejs/node/pull/27808.
- **Status**: The group’s current thinking is to encourage explicit source type definition via file extension or the `"type"` field. Source type detection via parsing source code is not 100% accurate and should therefore be left to userland where users can choose to accept its risks.

* Code coverage/instrumentation ([#95](https://github.com/nodejs/modules/issues/95))
* Pluggable Loaders to support multiple use cases ([#82](https://github.com/nodejs/modules/issues/82))
* Runtime loaders, transpilation at import time ([#96](https://github.com/nodejs/modules/issues/96))
* Arbitrary sources for module source text ([#97](https://github.com/nodejs/modules/issues/97))
* Mock modules (injection) ([#98](https://github.com/nodejs/modules/issues/98))
* Specifier resolution customization ([#110](https://github.com/nodejs/modules/issues/110))
* Package encapsulation ([#111](https://github.com/nodejs/modules/issues/111))
* Conditional imports ([#113](https://github.com/nodejs/modules/issues/113))
* Provide a way to make ESM the default instead of CommonJS.
- Discussion: https://github.com/nodejs/modules/issues/318.
- Proposal: https://github.com/nodejs/modules/issues/335.
- **Status**: Currently one can add `--input-type=module` to `NODE_OPTIONS` to flip the default for `--input-type`; at the moment the group is deciding not to pursue providing an ability to flip the default for the `package.json` `type` field. We instead want to encourage all packages, both ESM and CommonJS, to include an explicit `type` field; leaving it out and relying on a default is something we want to discourage.

### WASM Features:
## Archives

* WASM modules (#[106](https://github.com/nodejs/modules/issues/106))
- **[Features](./doc/archive/features.md)**: As part of creating the ECMAScript Modules implementation for Node.js 12.0.0, [use cases](./doc/use-cases.md) were brainstormed, which led to a list of features. This feature list, which previously existed as a section in this repo's root README, informed what became the [plan for the implementation](./doc/archive/plan-for-new-modules-implementation.md).

### Developer and Tooling Features:

* File / path / URL resolving (#[103](https://github.com/nodejs/modules/issues/103))
* Import CommonJS without needing asynchronous syntax (#[116](https://github.com/nodejs/modules/issues/116))
* Tree shaking (#[102](https://github.com/nodejs/modules/issues/102))
* Polyfillability (#[101](https://github.com/nodejs/modules/issues/101))
- **[Plan for New Modules Implementation](./doc/archive/plan-for-new-modules-implementation.md)**: This document summarized the work that went into the new ECMAScript Modules implementation that shipped in Node.js 12.0.0.

## Members

Expand Down
55 changes: 55 additions & 0 deletions doc/archive/features.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
> _This document has been archived._
>
> As part of creating the ECMAScript Modules implementation for Node.js 12.0.0, use cases were brainstormed, which led to a list of features. This feature list, which previously existed as a section in this repo's root [README](../../README.md), informed what became the [plan for the implementation](./plan-for-new-modules-implementation.md).

# Features

Based on [these use cases](./doc/use-cases.md) ([#55](https://github.com/nodejs/modules/issues/55)), our implementation aims to support the following features (subject to change):

### Baseline Modules Implementation Features:
* Spec compliance ([#132](https://github.com/nodejs/modules/issues/132))
* Browser equivalence ([#133](https://github.com/nodejs/modules/issues/133))
* Don’t break CommonJS ([#112](https://github.com/nodejs/modules/issues/112))
* No refactoring ([#87](https://github.com/nodejs/modules/issues/87))

#### Browser Interop:
* Browser and Node compatibility without building ([#107](https://github.com/nodejs/modules/issues/107))
* Browser-compatible specifier resolution ([#109](https://github.com/nodejs/modules/issues/109))
* Browser-compatible builds for ESM and CommonJS ([#108](https://github.com/nodejs/modules/issues/108))

#### CommonJS Interop:
* Named exports when importing CJS ([#81](https://github.com/nodejs/modules/issues/81))
* Multi-mode packages ([#94](https://github.com/nodejs/modules/issues/94))
* Transparent interoperability for ESM importing CommonJS ([#100](https://github.com/nodejs/modules/issues/100))
* Consumer-agnostic imports ([#105](https://github.com/nodejs/modules/issues/105))
* Mixed module types within app/module; gradual migration from CommonJS to ESM ([#99](https://github.com/nodejs/modules/issues/99))
* ESM in .js files ([#151](https://github.com/nodejs/modules/issues/151))

### Existing Node.js Utility Features:

* Importing non-JavaScript files ([#115](https://github.com/nodejs/modules/issues/115))
* NodeJS contextual pathing use cases ([#121](https://github.com/nodejs/modules/issues/121))
* ESM in executable files ([#152](https://github.com/nodejs/modules/issues/152))
* Callable resolver ([#157](https://github.com/nodejs/modules/issues/157))

### Loader Features:

* Code coverage/instrumentation ([#95](https://github.com/nodejs/modules/issues/95))
* Pluggable Loaders to support multiple use cases ([#82](https://github.com/nodejs/modules/issues/82))
* Runtime loaders, transpilation at import time ([#96](https://github.com/nodejs/modules/issues/96))
* Arbitrary sources for module source text ([#97](https://github.com/nodejs/modules/issues/97))
* Mock modules (injection) ([#98](https://github.com/nodejs/modules/issues/98))
* Specifier resolution customization ([#110](https://github.com/nodejs/modules/issues/110))
* Package encapsulation ([#111](https://github.com/nodejs/modules/issues/111))
* Conditional imports ([#113](https://github.com/nodejs/modules/issues/113))

### WASM Features:

* WASM modules (#[106](https://github.com/nodejs/modules/issues/106))

### Developer and Tooling Features:

* File / path / URL resolving (#[103](https://github.com/nodejs/modules/issues/103))
* Import CommonJS without needing asynchronous syntax (#[116](https://github.com/nodejs/modules/issues/116))
* Tree shaking (#[102](https://github.com/nodejs/modules/issues/102))
* Polyfillability (#[101](https://github.com/nodejs/modules/issues/101))
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
> _This document has been archived._
>
> This document summarized the work that went into the new ECMAScript Modules implementation that shipped in Node.js 12.0.0.

# Plan for New Modules Implementation

This document outlines the plan for building a new implementation to support ECMAScript modules in Node.js. The effort is split up into phases:
Expand Down Expand Up @@ -123,25 +127,8 @@ Phase 3 improves user experience and extends the MVP. Phase 3 is malleable based
- At time of unflagging: `"main"` (or `"exports": { ".": "file.js" }` overriding `"main"`) points to exactly one file, and full filenames are required (by default), so there is no possibility of an `import` specifier pointing to different files in ESM versus CommonJS; unless `--experimental-conditional-exports` is used (see next bullet). Without that flag, dual packages must provide secondary entry point via a path, e.g. `'pkg/module'` or `'pkg/commonjs'`.
- With `--experimental-conditional-exports`, paths within the `package.json` `"exports"` block can have separate entry points per environment.
- Landed in https://github.com/nodejs/node/pull/29978 and https://github.com/nodejs/node/pull/30051 behind `--experimental-conditional-exports` flag.
- Some members of the group expressed hesitation about conditional exports and want to explore other potential solutions for dual packages. If no alternative proposal reaches consensus before the end of January 2020, the `--experimental-conditional-exports` flag will be dropped and conditional exports will ship.
- Unflagged in https://github.com/nodejs/node/pull/31001 and shipped in Node 13.7.0.

## Phase 4: Further Improvements After Unflagging

* Dual CommonJS/ESM packages: See previous section.
- One alternative proposal is `require` of ESM: https://github.com/nodejs/modules/issues/299.
- **Status**: `--experimental-conditional-exports` will unflag by the end of January 2020 unless consensus is reached on an alternative approach. The flag may be dropped sooner if a consensus is reached, either for an alternative solution or for `--experimental-conditional-exports`.

* A loaders solution that supports all items in the [features list in our README](https://github.com/nodejs/modules/#features).
- Discussion: https://github.com/nodejs/modules/issues/351.
- Design: https://docs.google.com/document/d/1J0zDFkwxojLXc36t2gcv1gZ-QnoTXSzK1O6mNAMlync/edit#heading=h.xzp5p5pt8hlq.
- **Status**: In development behind `--experimental-loader` flag.

* Limited module type detection.
- PR: https://github.com/nodejs/ecmascript-modules/pull/69.
- Upstream PR: https://github.com/nodejs/node/pull/27808.
- **Tabled**. The group’s current thinking is to encourage explicit source type definition via file extension or the `"type"` field. Source type detection via parsing source code is not 100% accurate and should therefore be left to userland where users can choose to accept its risks.

* Provide a way to make ESM the default instead of CommonJS.
- Discussion: https://github.com/nodejs/modules/issues/318.
- Proposal: https://github.com/nodejs/modules/issues/335.
- **Tabled**. Currently one can add `--input-type=module` to `NODE_OPTIONS` to flip the default for `--input-type`; at the moment the group is deciding not to pursue providing an ability to flip the default for the `package.json` `type` field. We instead want to encourage all packages, both ESM and CommonJS, to include an explicit `type` field; leaving it out and relying on a default is something we want to discourage.
Now that the implementation has shipped, further efforts are listed on the [README](../README.md).
Loading