Plans for npm 10

[lukekarrys]

The npm team is working on npm 10, and we are currently in the planning and early development phase. In opening this issue our goal is to facilitate discussion about our planned breaking changes in npm 10 and work towards our goal of landing npm 10 in Node.js 20 and 18.

Breaking Changes

During the discussion about npm 9 we landed on a list of guidelines on integrating new major versions of npm with Node. We plan on following these guidelines again, and we consider all breaking changes listed below to fall within those guidelines.

• Update engines to >=16.14 ^18.17.0 || >=20.5.0
  • Tracking issue: Engines setting for npm 10 and beyond npm/statusboard#640
• Remove implicit if-present logic from run-script in workspace mode
  • Currently when calling npm run-script on workspaces if any of the scripts are missing it is treated as a silently ignored error, making there an implicit "if-present" config in this mode. Users who want this behavior will need to include this config.
• Remove the following deprecated or unused configs: tmp, ci-name, hashAlgorithm, metrics-registry
  • All these configs are either deprecated or are entirely unused in the npm CLI. We see these changes as low impact, but we always treat the removal of a config item as a breaking change.
• Use @npmcli/agent
  • The existing http agent and proxy libraries we use do not have very good timeout configurations and also diverge from the spec in odd ways. The new agent is intended to be identical to the existing way of doing things, but because of the scope of the change we are waiting on a semver major before cutting over.
• Remove "strict mode" from npm-package-arg
  • npm-package-arg can be set to run in a RFC 8909 compliant "strict mode" by setting the environment variable NPM_PACKAGE_ARG_8909_STRICT=1. We don't have any real plan with making that mode permanent so we are removing this functionality. Too many existing packages rely on our technically spec divergent behavior and it is not worth breaking so many people to force compliance. Anyone currently using this mode will need to find a third party way to strictly enforce the package specs in their package.json.
• Don't retry 409 errors during npm publish
  • Currently if the npm CLI gets a 409 from the registry during publish it attempts a single retry by re-fetching the manifest and attempting to apply a new patch on what's being published. This is not the safest approach, especially as the code used to patch is not the same code that is used to build the original manifest. It is much safer to simply have the user re-run the publish command.

Note: the following changes were originally planned but are not a part of npm@10

• Expand the list of default ignored files obeyed during pack and publish
  • Tracking issue: [FEATURE] Expand the list of default ignored files npm/npm-packlist#48
  • RFC issue: RFC: Expand list of ignored files npm/rfcs#114
  • The exact list of files is still being discussed, but npm will start ignoring more files by default from being included in packaged tarballs. The npm CLI currently defines this list in npm-packlist and it includes files like .npmignore and .gitignore. The original RFC proposes adding files to that list such as .editorconfig, .github/, and more.
• Implement the "obey user specifier" RFC
  • RFC issue: add "obey user specifier" RFC npm/rfcs#547
  • The npm CLI will now save the same specifier used to install a package to package.json, eg npm install eslint@=8.8.0 will save "8.8.0" as the specifier for eslint instead of "^8.8.0" as it does now. This will only affect installs run with an argument, and bare npm install commands will not churn the lockfile or package.json.

Testing

citgm was very helpful during the release of npm 9 in finding unintended consequences of our breaking changes. One new thing we will be doing during this release cycle is running our own modified version of citgm in GitHub Actions during all prereleases candidates of npm 10 to ensure we find breaking change bugs as early as possible.

Testing will also happen in CI during each pull request that is opened by the npm team.

Timeline

Our goals with this timeline are to ensure that npm 10 changes can be tested within Node as early as possible. We want to open PRs to main so that CI can be run against each prerelease of npm 10. The initial PRs will be labeled to only request-ci and not ready to be merged.

Once npm 10 is stable (Task 4) we will open a new PR that will be ready for review and to land on main. Once that PR is merged we will open PRs for both Node 20 and Node 18 backports. Our goal with these PRs is again to allow testing to happen as early as possible.

For Node 20 our goal is that npm 10 could be included in the tentative 2023-09-12 v20.10.0 release. For Node 18, assuming that npm 10 has shipped to users and we've fixed any bugs that have been raised, our goal is that npm 10 could be included in the tentative 2023-10-xx v18.20.0 release.

	Status	Task	Date
1	✅	First prerelease ([email protected])	2023-07-19 2023-07-26
2	✅	Open initial PR* to Node.js main branch for CI only nodejs/node#48934	2023-07-19 2023-07-26
3	✅	Second prerelease	2023-08-02 2023-08-31
4	✅	Major release ([email protected])	2023-08-16 2023-08-31
5	✅	Open PR to Node.js main branch for review nodejs/node#49423	2023-08-16 2023-08-31
6	✅	Land npm 10 on Node.js main branch	On or before 2023-08-23 2023-08-31
7	✅	Promote [email protected] to latest	2023-08-30 2023-09-08
8		Open PR to backport to Node.js 20	After npm 10 lands on main (Task 6)
9	✅	Open PR requesting backport to Node.js 18 nodejs/node#49611	After npm 10 lands on main (Task 6) 2023-09-11
10		Release npm 10 in Node.js 20	2023-09-12** v20.7.0
11		Release npm 10 in Node.js 18	2023-10-xx** v18.20.0

* Unmerged pull requests will be closed and a new one opened for each (pre)release. Any new PR will contain all the changes from all previous unmerged PRs.

** Tentative Node.js release date

[mcollina]

The plan looks solid and I don’t think there would be issue in shipping npm v10 in Node 20. However, I feel that shipping such an update to Node v18 might be too risky for two reasons:

1. Node.js v16 is out of LTS at the end of September
2. Node.js v18 enters maintenance soon after
3. Node.js v20 goes LTS end of October

I don’t think there is much time to iron out possible bugs with thr given timeline.

Lastly, I recommend the npm team to stop treating the LTS release as their target and align their releases to Node.js. We could have avoided this discussion entirely if you released npm v10 in March or April.

[ruyadorno]

Expand the list of default ignored files obeyed during pack and publish

From the list of breaking changes I believe this is the most likely to have a potential to be disruptive to the community, if the goal is to land on LTS I would advise the team to be very defensive about expanding that list. It might be worth holding on to the next major which hopefully aligns better with our release calendar.

[wraithgar]

from the list of breaking changes I believe this is the most likely to have a potential to be disruptive to the community, if the goal is to land on LTS I would advise the team to be very defensive about expanding that list. It might be worth holding on to the next major which hopefully aligns better with our release calendar.

I agree and this is really the suggestion that moved the needle back to "let's not do this". We were already on the fence on this one and the cost/benefit is not anywhere near the level that we would want on a breaking change.

[lukekarrys]

npm@10 has been released and promoted to latest. I updated my original comment which some changes as to what breaking changes actually landed in npm@10 as well as updating the dates to when those events happened.

The PR to land npm@10 in nodejs/node#main has been opened here and is ready for review: nodejs/node#49423

[lukekarrys]

I have opened a PR requesting to backport npm@10 to Node 18 here: nodejs/node#49611. This PR is just to run CI and facilitate discussion, since I expect opening the PR to generate more 👀 from contributors. npm@10 should be landing in Node 20 soon and we can continue to get and respond to user feedback.

We've already triaged and fixed one regression (npm/cli#6760) from npm 9 to 10 and are continuing to monitor new issues coming in for npm 10. At this time we have no new npm P0 or P1 bugs in npm@10.

Based on the list of breaking changes that ended up in npm@10 (2 of the more potentially disruptive changes were dropped from the release), we think it makes sense to land npm@10 in Node 18 before LTS. The smaller set of ubreaking changes means the potential surface area for user disruption is smaller than then the process of updating npm 8 to 9.

Lastly, I recommend the npm team to stop treating the LTS release as their target and align their releases to Node.js. We could have avoided this discussion entirely if you released npm v10 in March or April.

Addressing this point directly, our future plans will take this into account. Going forward we want to align more closely with Node around major versions to make this process much smoother. But we think in the case an exception for npm 10 should be discussed.

[mcollina]

It would be best to avoid shipping npm 10 in a release, given that glob now depends on BlueOak-1.0.0 licensed software, which is against the IP policy of the foundation. I've already asked an opinion to the Foundation legal staff.

I'm acting out of caution here.

[bricss]

It looks like BlueOak-1.0.0 is very much state-of-the-art permissive license, that is shipped within npm v9 in Node already.
And here is good article about it.

N.B.: Trying a new permissive software license

[mcollina]

Ref nodejs/node#49625.

[lukekarrys]

We've opened a new PR to backport npm@10 to v18: nodejs/node#50030. The npm team considers this PR ready to land in v18 now that [email protected] has shipped with v20.8.0. I will be checking the requested CI runs once they complete to ensure the same coverage as the previous draft PR (nodejs/node#49611).

The latest version [email protected] fixes one more P1 regression we found from npm@9. We've been triaging the same list of issues I linked previously and we have no new npm P0 or P1 bugs in npm@10 currently.

Based on the list of breaking changes that ended up in npm@10 (2 of the more potentially disruptive changes were dropped from the release), we think it makes sense to land npm@10 in Node 18 before LTS maintenance. The smaller set of ubreaking changes means the potential surface area for user disruption is smaller than then the process of updating npm 8 to 9.

I am tagging @nodejs/releasers @mcollina @ruyadorno in this comment since we would like to come to a resolution quickly before the final release of v18 later this month.

[richardlau]

Based on the list of breaking changes that ended up in npm@10 (2 of the more potentially disruptive changes were dropped from the release), we think it makes sense to land npm@10 in Node 18 before LTS. The smaller set of ubreaking changes means the potential surface area for user disruption is smaller than then the process of updating npm 8 to 9.

Node.js 18 is already LTS. Do you mean before it enters maintenance?

FWIW I'm preparing a Node.js 18 release for Tuesday but I'm intending it to be a revert release to undo the libuv updates in Node.js 18.18.0 -- I will not have time to do a full Node.js 18 release. Node.js 18.17.0, 18.17.1 and 18.18.0 all contained regressions which I would like to sort out before landing anything else in Node.js 18.

[lukekarrys]

Node.js 18 is already LTS. Do you mean before it enters maintenance?

Yes, that is what I meant and corrected my previous comment.

[wraithgar]

Now that Node.js v18.18.1 has shipped, do we have a better idea of whether or not we can land this?

[mark-wiemer]

Hi, very new to this issue, and a bit confused. npm 10 is out now and able to be installed without any pre-release flags. It doesn't come bundled with Node 18 yet, is that why this issue is still open? When should we expect a more formal "announcement" or summary of finalized changes with npm 10? Ref the GitHub blog for npm 9

[Apollon77]

please consider using [email protected] at least because of the current fixed node-gyp issues with Python 3.12 ... see nodejs/node-gyp#2869 ... because this already hits some users (and GitHub has alreayd started to update their runners to Python 3.12 - actions/runner#2972). The same 8also if off topic here) would be cool to see ot too late in node.js 20.

PS: I personally would love a "fix node.js 16 version" with also npm 10, but that might not going to happen because formally EOL ;-) ... it will just be forseeable that isues arise when Python 3.12 spreads on the users systems

[lukekarrys]

[email protected] has landed in main (nodejs/node#50531) and should be included in the next Node 20 and 18 releases.

I've been keeping this issue open until Node 18 officially ships with npm 10.

[placaze]

[email protected] has landed in main (nodejs/node#50531) and should be included in the next Node 20 and 18 releases.

[email protected] doesn't seem to be backported into the v20.10.0 release proposal (nodejs/node#50682).

• https://github.com/nodejs/node/blob/v18.x-staging/deps/npm/package.json --> v18.x --> [email protected] right now
• https://github.com/nodejs/node/blob/v20.x-staging/deps/npm/package.json --> v20.x --> [email protected] right now

[lukekarrys]

Thanks for the heads up @placaze. I've added a comment to the release proposal for v20.10.0 asking [email protected] to be included.

[fudom]

Any reasons why NPM 10 is suddenly included in Node 18? Expected NPM 9. It was a surprise for the CI that installed Node wildcard major 18.x. I thought Node LTS was tied to a major release including NPM. I never faced this problem before. Is this the first case of upgrading major NPM in the same major Node? A special exception? Any notes for migration 9 to 10? I see NPM 10 was merged into Node 18.19. So the last version with NPM 9 is Node 18.18 where we have to stick to the minor version now or test NPM 10.

npm ERR! notsup Required: {"node":"^18","npm":"^9"}
npm ERR! notsup Actual:   {"node":"v18.19.0","npm":"10.2.3"}

[mariusGundersen]

Didn't you promise in #825 not to make breaking changes (major update of npm) in minor releases of node? Why is it so important to update npm in node v18?

[Apollon77]

The main reason in my eyes is that node-gyp had compatibility issues with python 3.12 which means that all node-gyp binary package builds would fail. The fix was in npm 10.2.3 or such ... It seems that the npm team decided to do not do a backporting of inclusion of new node-gyp intp an older npm major version ...

So in fact the issue was really big (and with Node.js 18 being LTS sinsce aprio 2025 thats a long timeframe to risk this. But agreed ... backporting would also have had been an dea, but I assume too late now

[theJC]

We are very interested in getting 10.3.0 of npm or higher pulled in, due to the fix/revert that restores SIGTERM/SIGINT signal forwarding to child processes - npm/run-script#188

Any chance guidance can be provided as to what Node release we can get an npm version that comes with that has this fix in place? This issue has significant ramifications on the behavior of our services runtime and allowing Node services launched by npm to honor kubernetes shutdown process and the signals it sends to provide time for draining/etc.

[richardlau]

@theJC I've volunteered to do a Node.js 20 release in February for exactly this reason (to include npm 10.3.0). Our usual policy is that changes should first go out in a current release (i.e. Node.js 21) for two weeks so the exact timing depends on when we get a Node.js 21 release out with that version of npm.

[theJC]

Sorry @richardlau , I didn't indicate, but we are specifically looking for a node 18 release to contain 10.3.0 or higher (Node18 is the only version our company supports at this time) to address the repair of the SIGNAL propagation breakage on that version line.

[theJC]

Can anyone provide details on how best to provide input/encouragement or learn about the process that would be required to consider getting npm 10.3.0+ be the version distributed in a soonish/upcoming Node 18.X release?

[bricss]

It will get into next release of v21, coz it already landed in da main 3 weeks ago, and anyone can pull it up for backport in da v18 in any moment 🤷

[richardlau]

It won't need a manual backport but will need to go out in a Node.js 21 release first.

[theJC]

How do we get the next version of Node 21 (21.6.2) to be bumped to having the 10.3.0 or higher version of npm?

[mhdawson]

@lukekarrys this may be a case of changes in npm10 versus 9 causing Node.js problems when upgrading in the 18.x line -npm/cli#7272

[lukekarrys]

Thanks for the ping @mhdawson. I've collected that issue along with a few others (that I don't have links to right now) that are all around network requests and the http agent we use.

As an aside, the original purpose of this issue was tracking inclusion of npm 10 across Node.js versions which has been completed. I'm going to close this now and will track these bugs in npm/cli as they get fixed.