only deviate from well-established ecosystem design if necessary

[devsnek]

hopefully a more structured conversation from nodejs/node#21414

so node is adding a lot of things that mesh with other things in the ecosystem (for example perf_hooks.performance and worker_threads)

like the title says, i think it would be best if when we introduce things that the rest of the ecosystem already does, we should try to stick to how they do it.

in the case of workers, that would mean a few changes to worker threads. i don't think any of the changes inhibit the use of worker threads or their usefulness. for other things like perf_hooks.performance and util.TextEncoder it would mean putting them on global.

in general i think node should need a good reason to deviate from ecosystem expectations, not a good reason to stick to them. i am not however suggesting that we MUST to stick to ecosystem expectations.

/cc @Fishrock123 @gireeshpunathil from the other thread

[addaleax]

To be clear, you’re asking for design of existing platforms, i.e. browsers, and not the ecosystem – right?

[devsnek]

@addaleax yes, thank you for clarifying. explicitly though, i don't think we should limit the definition to browsers, just for future-proof-ness.

[targos]

IMO nodejs/node is better place for this conversation

[guybedford]

I do think this is a very important point Gus brings up, and something to consider at a high level in Node - what the guiding objectives are re implementing web APIs.

Personally I tend to agree that if implementing features in NodeJS based on browser APIs, as much care should be taken as possible to keep close to the browser implementation or specification where possible. This implementation predictability provided by convergent behaviour and specifications has immense value. Just because there are edge cases that don't apply to Node, doesn't mean we can't try our best to handle those cases as neatly as possible, making the escape hatches clear.

[mcollina]

It is good to talk about this! I would clarify that deviating from the browser expectations is something positive as well. Running JS code on the server is different from running JS code in the browser, and Node.js should acknowledge this. Moreover, we need to acknowledge the fact that there is a community of developers and companies using the current Node.js API, and we need to keep compatibility with those as well.

Therefore, I propose we put the following somewhere:

Node.js strives to follow ecosystem expectations, especially around browser compatibility. Node.js is a standalone runtime, and this difference poses different challenges than browser environments. Node.js will deviate from the ecosystem expectation to address the specific challenges that building a standalone runtime poses, as well as other conventions of the Node.js ecosystem.

[ljharb]

I like @mcollina's wording, and imo it also would apply to Modules.

[guybedford]

@mcollina great to see thoughts to wording here. Note though that I think the issue @devsnek is getting at in this thread is about features that aren't strictly necessary to provide a functional standalone runtime, but that conflict with browser compatibility.

This then brings up the questions - what is necessary, and what does it mean to conflict with browser compatibility, both of which I think could be very fruitful conversations to have.

[devsnek]

@guybedford puts it rather well. i'm mostly concerned with things like workerData, which can't be polyfilled and while it is nice to have, isn't really needed. also stuff like the mountpoints of perf_hooks.performance and util.TextEncoder which are specified to globals. it wouldn't hurt us to move them, and it would benefit people looking to write portable code.

[mcollina]

@devsnek @guybedford we are mostly in agreement. However I felt that the way the issue was phrased depicted deviating from the ecosystem as bad thing, while it's a good thing when we go through the escape hatch for good reasons.

[guybedford]

@mcollina certainly, and I know we didn't yet establish as well whether this is the right place for this discussion too, although it may well be related to #532.

Also agreed considering "deviating from the platform" being good or bad isn't really productive. Rather I'd prefer to focus on understanding what helps and what does not help compatibility with the web, and ensuring that be part of the design considerations.

[mhdawson]

How about this tweak to @mcollina suggestion:

Node.js gains significant benefits by providing APIs which are familiar to those writing code for browsers and the Node.js community works to preserve compatibility when possible. At the same time, Node.js is a standalone runtime and faces different challenges than seen in browser environments. This makes it necessary (and reasonable) to deviate in some cases when there is either no choice or the benefits outweigh the cost. Given this context, additions to Node.js should strive to be compatible with existing implementations and document the rationale for deviations when they are necessary.

[mhdawson]

I'll also note that I don't think this necessarily will drive a direction for nodejs/node#21414 as I think most people agree that compatibility where it makes sense is the goal, it is just that it is not always easy to agree on "where it makes sense".

Still good to discuss/see if there is general agreement and at what level.

[Fishrock123]

I'm not really sure what "ecosystem" is meant here.

But if we want to go the full route, a good example of this idea failing horribly in the JS ecosystem is promises. Decisions to go after something that was "ecosystem-like" in part ended us up with something that works in ways which are Not Very Good. (At least, that is my understanding.)

Examples in our own "ecosystem" would be useful. Browsers certainly aren't really part of our ecosystem, but they are adjacent to it.

[devsnek]

I'm referring to the broader javascript ecosystem. it would be rather foolish to pretend that Node.js is somehow not a part of that. I'm not suggesting that node has to copy everything that already has been conceived in the rest of the ecosystem, but rather that we take into consideration the existing projects and document when/why we deviate from them. In most cases i think it's safe to take design hints from existing projects simply because its all javascript. there will always be cases where node needs to be different but we shouldn't go out of our way to be different.

[joyeecheung]

@devsnek Is this issue somewhat similar to #532 which, in my understanding, asked that we have a policy/goal regarding Web APIs integration in core? Right now the problem is: we don't have a goal, or consensus around this type of integration, so decisions regarding implementations are made on a case-by-case basis.

[joyeecheung]

@devsnek Also do you mind changing the ecosystem to Web API in the title? When I saw the issue I thought it's about npm packages

[devsnek]

@joyeecheung I'm trying not to specifically say web as I also do mean npm packages, other runtimes that aren't browsers, whatever. JavaScript ecosystem in general.

[joyeecheung]

@devsnek IMO the argument makes less sense in terms of npm packages, because it that case we either don't include it in core because "it can be done in the userland", or put it directly under deps/, or let the users use the user land package instead. And npm packages follow semver so they have more freedom in terms of breaking changes due to design choices by bumping major, and they tend to do that with codemods so it's hard to say what's "well-established", it's possible that we may try to be consistent with things that they are moving away from, whereas the Web standards are more conservative and have well-established things - the spec.

Also it's hard to say what's the well-established design in terms of npm packages, because there are usually multiple packages in each use cases, each of them may differ significantly in design. Then we will need a criteria - npm downloads? That may change very quickly since popularity is a mysterious thing. And if we favor one particular package or a set of practices followed by existing packages, it may hurt innovation in the user land in the end.

[devsnek]

@joyeecheung in terms of npm packages this would probably just mean saying "the package A does B but we do C instead because of D."

[joyeecheung]

the package A does B but we do C instead because of D.

@devsnek If B involves actual functionality, then why can't we just put A under deps, or ask them to structure the code in a way that (part of it) can be put under deps?

If B only involves certain practices or patterns, then why should we favor A? What happens if some time later, A chooses to bump major and do E instead of B? Or a new popular package emerges that does E?

[joyeecheung]

It makes more sense in the case of Web APIs because the source of truth is not in a particular implementation, and once the spec finalizes the design rarely changes in significant ways, and there won't be that many rivals since Web APIs are unique and (at least many of them) will have unparalleled adoption.

[devsnek]

my sentence from above (in general i think node should need a good reason to deviate from ecosystem expectations, not a good reason to stick to them.) sums up what i'm trying to say pretty well.

as an example of how an npm package might get involved with this... say we wanted to make an interface like https://www.npmjs.com/package/commander in node. there isn't a great reason to remake it from scratch (that i know of, this is just an example) so like you said we would want to put it in deps and re-expose it. the same idea applies to specifications of web apis except that they can't be put in a deps folder.

this weird mess of explanation is why i didn't say "npm" or "web" or whatever else. something general lets us manage this on a case-by-case basis and not be locked into anything that will hurt us in the long run.

[joyeecheung]

@devsnek The reason why I think we should limit the scope to Web APIs is that including npm packages weaken the argument, since npm packages don't usually share the stability concerns that we and the Web have in common. It's common in the user land to bump major and do a make over of a design, but much less common on the Web.

Also regarding the Web, I think it's a good thing if we try to be consistent with the Web, but we don't really need to exclude ourselves from extensions like workerData, browsers don't always respect specs and implement their own extensions all the time, as long as the extension doesn't break the compliance (e.g. failing a particular case in WPT in a way that browsers won't be able to compromise) I think it's fine since we are not browsers after all.

[devsnek]

@joyeecheung workerData is specifically a problem for me because it can't be polyfilled without source transforms, and for some other reasons i mentioned in the issue.

i'm ok with limiting the scope here as long as we don't back ourselves into any corners.

[targos]

@devsnek workerData is not the only way to pass data to a worker. If you want your code to work in a browser, you can instead only use postMessage. This is IMO what is important: to support the API that exists in the browser.
In that case, instead of pushing for the removal of workerData I would rather tweak message passing so that it conforms to the EventTarget interface.

[devsnek]

@targos

If you want your code to work in a browser, you can instead only use postMessage

its quite easy to make things browser compatible if you own the code, but you don't always own the code. like i said there are also other reasons i think it should be removed, see my pr for more

I would rather tweak message passing so that it conforms to the EventTarget interface.

i would like this too, but i don't think it really fits into the pr removing workerData at least

[joyeecheung]

@devsnek For reference, this is how Blink evaluates APIs that involves interoperability risk (not standardized or implemented by other platforms): https://www.chromium.org/blink#new-features We are not a browser, but I think we can draw from their experience that it's better to evaluate API changes based on more measures/dimensions than a black-or-white policy.

[joyeecheung]

Also, the term "ecosystem" seems to be vague. What happens when one or more major browsers implement something that is not standardized? Should we follow them because they dominate the market share? To say that we will only deviate from the Web when necessary seems to put us in a passive position as a implementer, which isn't healthy IMO, we should be able to propose changes to the spec for our use cases if necessary.

[devsnek]

it's fine if the language is less strong but I just want to make sure we really do consider these things, and for there to be a platform for collaborators who see issues to stand on instead of just saying "this needs to change"

[mhdawson]

@devsnek its been a while since there was any discussion on this issue. Should it still be open or can it be closed?

[devsnek]

since the open standards group formed I think this is safe to close