Modules in Node need to be as easy to use as current solutions

[GeoffreyBooth]

I’m encouraged by the 2018-04-11 meeting and its discussion of use cases and feature requirements, and the reevaluation of the Node modules effort’s overall goals. I hope that this leads to a final result that works better for everyone.

I wanted to add another goal, or metric of success, or something; I’m not sure where it belongs so I’ll just say it here:

Node should support ES modules without requiring extra tools for what users want to do.

Over the years, when Node has added support for new syntaxes, they’ve “just worked.” If I had been using the syntax before and transpiling via Babel, say, I would turn off that transform in Babel and my code would execute the same as before. I think average users would assume that the same should be true for import and export statements.

This article by @giltayar about Node’s --experimental-modules implementation has a great section near the end about how to create dual-mode packages, that can be imported by a project using either ES modules or CommonJS. The instructions describe using Babel via an NPM script to transpile .mjs files into .js files, to create entry points for each mode. Reading this, I feel like the development team missed the point. So in order to use Node’s “native” module support, I need . . . Babel?

I think that most users would treat interoperability with the existing Node ecosystem as a non-negotiable requirement of any project they work on. If forced to choose between using Node’s native ES module support or preserving interoperability, users are going to choose interoperability (either via CommonJS or tools like Babel or esm). If too many projects have too many “must-have” use cases that preclude them from using Node’s native ES module support, the native support will struggle to find usage. There’s not much point in building a feature that most people won’t use.

Node’s native ES module support needs to be better, both in performance and convenience, than Babel and esm. Those tools are the competition. Every implementation decision that differs from how those tools do things should have a very strong justification, especially if it makes working with modules harder or requires a sacrifice that those tools don’t ask the user to make. Eliminating barriers to adoption and ensuring seamless interoperation with the existing Node ecosystem should be overriding goals of the design of the implementation, above any other concerns.

[jdalton]

Node’s native ES module support needs to be better, both in performance and convenience, than Babel and esm. Those tools are the competition.

I would love this!

Every implementation decision [...] should have a very strong justification, especially if it makes working with modules harder

👏

Eliminating barriers to adoption and ensuring seamless interoperation with the existing Node ecosystem

More of this mindset is a good thing.

[mcollina]

Definitely this is a good mindset! However, I see several issues:

1. most modules on NPM supports multiple versions of Node.js. Those will not have esm support: for those transpilation will always be required. Considering that Node.js 8 will be in LTS/maintenance for 2 more years, transpilation is going to stay around. We will start to get real adoption when we land non-experimental ESM support in a LTS release.

2. The interoperability with common.js is a very difficult topic on which this group has not reached consensus yet. However, seamless interop between ESM and common.js is not possible because of what the ESM spec contains. I do not think this is the right venue to discuss diverging from the spec.

[jdalton]

We will start to get real adoption when we land non-experimental ESM support in a LTS release.

Not if the experience is subpar.

However, seamless interop between ESM and common.js is not possible because of what the ESM spec contains. I do not think this is the right venue to discuss diverging from the spec.

As we learned at the Node Module Summit, Node's ESM implementation is technically against spec. There are grey areas that can and should be explored esp. in interop scenarios.

[mcollina]

I completely and totally agree @jdalton.

[devsnek]

@jdalton what spec are you referring to and how does it break it

[jdalton]

what spec are you referring to and how does it break it

The ECMA-262 spec. The gist was it's an order of operations nit (non-user observable) that enables Node to do something.

[giltayar]

I believe what was said (if it was in answer to my question) was that if we wanted to have named exports when importing a CJS module, we would have to evaluate the CJS script at a time that is not allowed in the ESM spec. And that would break the spec. But, since NodeJS does *not* have CJS named exports, we are currently *not* breaking the spec.

[jdalton]

@giltayar

I believe what was said

I'm aware of the question you asked during the 2nd talk but think it was a different bit. I remember a stress on the fact that the current behavior is non-user observable and downplayed as a concern. I took a mental note of it because it was a case of Node acting in a pragmatic way (a good thing).

Update:

Referenced in the 7th talk of the Module summit by @devsnek and Bradley during a PR review for named exports, which has the Module WG go-ahead to land.

[devsnek]

FWIW i already find our esm implementation pretty easy and straight-forward to use. in my opinion the most annoying things are having to use a flag (which is not a real concern :P) and not having named exports (which we can fix by breaking spec but to "downplay" this once again, i don't find that to be any sort of issue as we know everything about how builtins execute)

[robpalme]

Please could we document this non-user-observable-spec-breakage, that we seem to agree is the right move, in a clear reference location, e.g. an FAQ. We need a URL that anyone can reference whenever claims/concerns regarding non-compliance are raised.

…

On Fri, 20 Apr 2018, 22:12 Gus Caplan, ***@***.***> wrote: FWIW i already find our esm implementation pretty easy and straight-forward to use. in my opinion the most annoying things are having to use a flag (which is not a real concern :P) and not having named exports (which we can fix by breaking spec but to "downplay" this once again, i don't find that to be any sort of issue as we know everything about how builtins execute) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#70 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGni9R7ACViemhNKpjTc8EJnJl5pkyeeks5tqk9NgaJpZM4TdzxQ> .

[devsnek]

it hasn't been merged yet but i can add a note to the pr. i'd rather implementation details as implementation details though

[jkrems]

Please could we document this non-user-observable-spec-breakage

If this is about the JSON parsing - I'm not sure how that does break the spec. It's parsing to discover exports. The fact that we reuse the parse result during evaluation doesn't seem to change that fact..?

[devsnek]

@jkrems its about how i evaluate builtin modules in my pr during instantiation

[jdalton]

Folks let me shift this back on topic. The gist was that being pragmatic as opposed to dogmatic,
the intent of my example above, is a good way to approach things (esp. in the realm of interop).

[bmeck]

If we discuss non-observable spec violations as breaking then we effectively remove optimizations. I would not see non-observable differences as a strong argument. The PR in question can be to spec but optimizes in a non-observable manner that cannot be replicated by spec mechanisms.

[giltayar]

I agree with @bmeck. Breaking the spec may have consequences down the road when TC39 enhances ESM with other features. For example, for all intents and purposes, loading ESM modules can today be a synchronous process, and can be implemented as such. Therefore, theoretically, we can "break" the spec without any harm done. But if TC39 lands @MylesBorins proposal of top-level await, that breakage will mean that we may not be able to implement the spec.

As for the specific spec "breakages|": the three disussed above (evaluating JSON, evaluating built-in modules, and evaluating CJS) all concern evaluating code in a phase that is not "allowed" by the spec.

But in the case of JSON, this is not harmful, as evaluation causes no side-effects, and in the case of builtin-modules, we can argue that all builtin-modules are "evaluated" before NodeJS runs (and most, I believe, have no side-effects).

This is not true of evaluating CJS. Evaluating CJS in a phase that is not per-spec, because this evaluation may have side-effects, is visible to the end-user. Yes, in a very slight and minor way, but these things, as I discussed above, may cause problems down the road.

So as much as I hate to say it, I believe we should not break the spec in a user-visible way (even if it is slight), even for the sake of convenience, or even for the sake of babel compatibility.

[jdalton]

Getting back to the topic. The gist of @GeoffreyBooth's post is that Node should be as easy (easier) to use as current solutions. I think keeping developer experience in mind is a good thing. We've started the process of uncovering use cases and sussing out features (with a "yes, and" mindset) in the Module WG meetings. I'm encouraged by our progress and I'll keep @GeoffreyBooth's post in mind as things progress.

[benjamingr]

On a tangent, I've come across this which made a very good point against top-level await:

You've just put your entire app – anything that depends on data.js, or that depends on a module that depends on data.js – at the mercy of that network request. Even assuming all goes well, you've prevented yourself from doing any other work, like rendering views that don't depend on that data.

Note that I'm specifically referring to top-level await with import rather than in general.

---

I totally agree with this though:

I believe we should not break the spec in a user-visible way (even if it is slight), even for the sake of convenience, or even for the sake of babel compatibility.

[guybedford]

@benjamingr the same could be said of the CommonJS require statement :)

[benjamingr]

@guybedford true, and to be clear I'm not really expressing a strong opinion about it.

I'm just noting from an "as easy as userland alternatives" perspective this is a strong implication I did not understand when commenting on #7.

Do we have any use cases that cover the case #7 need/want? If not - we should definitely add one.

[guybedford]

@benjamingr that was exactly the last use case from the previous meeting - put under "dynamic specifier resolution", perhaps we need to look at splitting some of these out into smaller features.

[GeoffreyBooth]

I’m glad to see open-minded discussion of specific cases where user experience and other priorities clash, but it seems to me like these are detailed enough that perhaps these cases deserve their own threads in which to flesh them out. Perhaps people could open new issues to continue the discussions there?

With regard to the point of this thread, of trying to hold user experience as a very high (if not highest) priority, might I suggest a specific tactic? I call it playing “users’ advocate.” Like devil’s advocate, but always arguing on behalf of the user requesting a feature or certain functionality. Even if you don’t agree with the position, or the user’s request, you can still play users’ advocate to add some rigor to a decision, the same way that playing devil’s advocate exposes the holes in an argument.

What this would look like in practice is basically challenging every excuse for why something can’t be done. To take the importing of CommonJS as an example (for illustration only, I’m not trying to argue the merits here), a claim was made that parsing CommonJS in a particular phase would have side effects and therefore be observable, which is against the ESM spec and theoretically bad in the future. To which a users’ advocate could reply that either we should find a way to parse without creating side effects or creating observable side effects; or that observable side effects are already created right now when importing CommonJS so this is no different; or that the spec shouldn’t apply when CommonJS is involved; or finally the advocate could make an argument for why this isn’t as important enough of a concern when weighed against the outcome of not allowing CommonJS imports. And so on and so on.

The point is to keep going until one side or the other is exhausted. Either the “we can’t do it” side can’t come up with any more reasons why not (or the group judges the reasons to be of lesser importance than the value of building the feature); or the users’ advocate can’t come up with any more workarounds for the points raised (or make a persuasive case that the points shouldn’t take priority over the feature). But one or more people need to take each side, or both sides, and argue each side’s case in good faith and with an open mind. Users deserve their use cases implemented if at all possible, especially if an implementation can be found that doesn’t violate other priorities such as adhering to spec. I feel like I’ve seen too many comments saying that something can’t be done, without a thorough explanation of why it can’t be done, much less a rigorous analysis of why no other solutions could work. I think our users deserve as much.

[ljharb]

I do hope - even as someone who's quite difficult to "exhaust" in an argument - that we won't be making decisions solely based on who can argue the longest and exhaust everyone else. It'd be nice if nobody had to be exhausted at all.

[GeoffreyBooth]

It’s not about exhausting the people, it’s about exhausting the options. As in, don’t just say “the spec doesn’t allow it” and give up. Keep searching to find a way to implement the feature in a way that the spec allows, or find another workaround, or find some solution that we can all agree on. It’s not so much an argument as it as a collaborative search for solutions, and if each side engages in good faith it shouldn’t feel like a fight.

Also the “sides” don’t need to be separate people. Just as someone can play their own devil’s advocate, members can argue both cases at once. Again, it’s not about being proved right, but working through all options until we find one that works (or can demonstrate that we’ve eliminated every solution that we could think of).

[bmeck]

I would like to make a counter argument in favor of iterative expansion of features is better than having to argue against every possible addition a feature. Having developer centric mindsets are good, but both JS and the Node EP carried lots of discussion on topics already on how/why of things. I think that instead of trying to force all features to exist; we investigate them and need to make cases for why it isn't problematic for future iterations to include the feature as well as the above.

To provide an example based upon the above:

What this would look like in practice is basically challenging every excuse for why something can’t be done. To take the importing of CommonJS as an example (for illustration only, I’m not trying to argue the merits here), a claim was made that parsing CommonJS in a particular phase would have side effects and therefore be observable, which is against the ESM spec and theoretically bad in the future. To which a users’ advocate could reply that either we should find a way to parse without creating side effects or creating observable side effects; or that observable side effects are already created right now when importing CommonJS so this is no different; or that the spec shouldn’t apply when CommonJS is involved; or finally the advocate could make an argument for why this isn’t as important enough of a concern when weighed against the outcome of not allowing CommonJS imports. And so on and so on.

The argument for minimal iteration would state that the addition of the out order execution would prevent allowing polyfills/loggers/APM/etc. to be written in ESM since CJS would always run first and be unable to use the side effects that libraries are intending to instrument ahead of other evaluations.

We need to not just weight developer desires at a glance but seek further understanding on why it may cause problems if we break the spec and violate how ESM was designed to work. A lot of effort was made already on these fronts and we can discuss each feature in depth again, but I am not willing to do so in a light manner.

[jdalton]

The argument for minimal iteration would state that the addition of the out order execution would prevent allowing polyfills/loggers/APM/etc. to be written in ESM since CJS would always run first and be unable to use the side effects that libraries are intending to instrument ahead of other evaluations.

FWIW there can be implementations were APM just works for both even with early evaluation. As we start digging into features more we should totally use the existing WIP implementation as a reference, but not necessarily use it to box-in other possible implementations or approaches.

A lot of effort was made already on these fronts and we can discuss each feature in depth again

As mentioned before a lot of effort can be made in a direction. That doesn't make the direction inherently great good or bad. As with anything we learn from it and can apply the lessons to other approaches and challenges.

[bmeck]

FWIW there can be implementations were APM just works for both even with early evaluation.

I would be interested in an example of a situation where A needs the side effects of B but A runs first. I'm not sure I understand your statement.

That doesn't make the direction inherently great.

No, it does not; we need to reiterate those discussions and not treat all of the prior work as invalid. There was value in the past and we can use it. We are not always discussing new things.

[jdalton]

No, it does not but we need to reiterate those discussions and not treat all of the prior work as invalid.

I don't think anyone is suggesting prior work is invalid (or without value). Implementations are built on stacks of design choices, many building on previous ones. As we begin identifying features, fundamental design choices of the current WIP implementation might be challenged/changed which can open entirely new ways to frame and approach things.

[MylesBorins]

Can this be closed