build: enable yarn and pnpm Corepack binaries by default

[aduh95]

Fixes: #50963

Last version of Corepack has addressed (some of) the major concerns that was discussed in the TSC meeting last time:

• feat!: prompt user before downloading software corepack#360
• feat: bump Known Good Release when downloading new version corepack#364

IMO having Corepack enabled by default will overall be a clear win for Yarn and PNPM users (and won't affect npm users). Having it enabled by default does not stop those who don't like the "jumper binary" pattern to install Yarn and PNPM as they see fit.

[MoLow]

LGTM.

[richardlau]

If we land this then we probably also need to add the additional shims to tools/msvs/msi/nodemsi/product.wxs.

[GeoffreyBooth]

Corepack needs to support npm in the same ways that it supports Yarn and pnpm, if Corepack is to remain included with Node’s distribution. While Corepack was experimental and disabled by default this incompatibility was something that could be ignored as a TODO to be solved later, but with the current proposal to enable it by default (for all intents and purposes, making Corepack stable) the time has come to reconcile its relationship with npm.

In particular, the members of the TSC who met on 2024-01-24 asked the Corepack champions to take specific steps to align design goals: #50963 (comment). This work has not yet been done. Until it has, I don’t think there should be any movement toward making Corepack stable or enabled by default.

[jasnell]

#51888, if Corepack is to remain included with Node’s distribution

I disagree. Having npm and corepack work together is an orthogonal concern. There is no impact to npm users if corepack is enabled by default. Recognizing that there still may be work remaining to do, these things can be done incrementally I don't see them as blockers here.

[GeoffreyBooth]

There is no impact to npm users if corepack is enabled by default.

If Corepack is to ever support npm in the ways that npm’s maintainers desire, then Corepack itself will likely need to undergo significant changes. It may need to drop the version pinning or jumper binaries, for example, which would be a dramatic change that would break the majority of Corepack’s current users (pnpm users who are pinning precise versions to avoid corrupting lockfiles).

I see enabling Corepack by default as equivalent to dropping an experimental flag. It’s a sign that we’re moving an experimental feature into the “release candidate” stage, where we don’t anticipate any further major API changes. But unless we’re okay with simply never fully supporting npm, Corepack needs significant changes. And shipping major changes to a feature that doesn’t require opt-in is bad UX. We should sort out these things while Corepack is still experimental and still disabled by default.

[anonrig]

There is, because if Corepack is to ever support npm in the ways that npm’s maintainers desire, then Corepack itself will likely need to undergo significant changes.

From what I've read on Github and from private discussions, NPM does not want to be shipped with Corepack. NPM wants to be shipped with Node.js itself. It was clearly communicated in previous Github issues.

We shouldn't block the progress of adding support for other package managers just because NPM isn't interested, or it's not in their roadmap or it's not in their benefit.

[GeoffreyBooth]

NPM does not want to be shipped with Corepack

So maybe Corepack doesn’t need to ship package managers. Or maybe Corepack downloads Yarn and pnpm on demand but simply errors on the wrong version of npm. We don’t have defined goals of Corepack; what is a core feature of Corepack and what isn’t? If Corepack drops the downloading ability, would it still be achieving its goals? What if it drops the version pinning?

My point is still that we should work all these things out before making Corepack near-stable. There’s still major uncertainty around Corepack. The fact that we’re considering enabling this by default before even making the effort to define what Corepack’s goals are feels very wrong to me.

[mhdawson]

@aduh95 I know that a prompt was added in nodejs/corepack#360. What I'm not sure of is how "by default it's only shown when "not using the corepack binary" (i.e. when using the binaries created by corepack enable)" affects it's use in Node.js.

I want to be sure that if corepack is enabled by default that the user will see the prompt the first time they try to use yarn, pnpm etc.

Assuming that they will see the prompt, I think we need an addition somewhere in the Node.js documentation which explains that despite corepack being present to ease the installation of these tools, they are not part of the Node.js distribution and:

• the version installed is the latest at the time of use
• any required updates (related to security vulnerabilities or otherwise) are out of scope of the Node.js project. If necessary end users must figure out how to update on their own.

In addition we should have an addition to the https://github.com/nodejs/node/blob/main/SECURITY.md#the-nodejs-threat-model which indicates that any vulnerabilities in package managers installed through corepack are outside of the Node.js threat model and why.

[mcollina]

I’m currently -1, pending a solution to nodejs/corepack#399. I think this can be solved quickly if there is agreement.

——

@GeoffreyBooth I think we can reconcile the conundrum by saying that npm will stay as it is, and it has a special relationship with Node.js. I agree that something in the corepack design docs has to change.

[GeoffreyBooth]

I think we can reconcile the conundrum by saying that npm will stay as it is, and it has a special relationship with Node.js. I agree that something in the corepack design docs has to change.

Surely there should be consensus that we need updated design docs, both on the Node and Corepack sides, before we consider enabling Corepack by default. Even if the updated docs say that npm is special and nothing that Corepack does applies to it, that feels like something that we should spell out and land using the regular PR process. I don’t understand why we would be rushing toward stability when there’s still basic consensus-seeking work to be done.

Personally I don’t think that npm should be special. If we ship both npm and Corepack, as a user I expect them to work well together and provide the same functionality as is provided for other package managers. Inconsistency is bad UX. My preference would be that the people working on Corepack devote some effort into resolving this so that Corepack can work well for all of our users, not just Yarn and pnpm users.

[jasnell]

If Corepack is to ever support npm in the ways that npm’s maintainers desire, then Corepack itself will likely need to undergo significant changes. It may need to drop the version pinning or jumper binaries, for example, which would be a dramatic change that would break the majority of Corepack’s current users (pnpm users who are pinning precise versions to avoid corrupting lockfiles)....

Many changes may be needed by lots of things over time. We don't need to solve everything before making progress. We can make breaking changes in semver-major commits at any time. The idea that we might need to make major changes at some point is no reason not to make progress now.

[GeoffreyBooth]

Many changes may be needed by lots of things over time. We don’t need to solve everything before making progress. We can make breaking changes in semver-major commits at any time. The idea that we might need to make major changes at some point is no reason not to make progress now.

This argument makes no sense to me. Generally we don’t unflag features if we anticipate major changes. That’s what I see here: Corepack will need major changes to support npm; it should support npm; therefore it should remain disabled by default while it undergoes the significant churn required to support npm.

The lack of outreach to the npm team even after requests from the TSC leads me to suspect that if Corepack is unflagged without npm support, then npm support will never happen. Therefore I think it’s all the more vital that we hold the line here until the design doc work and npm support work is complete.

[mcollina]

Marking my -1 explicit depending on the fix in nodejs/corepack#399. The lack of reproducible builds by default is too high of a risk to mark as enabled by default. It's also a clear defense against supply chain attacks etc.

(I'm positive on lgtm'ing this PR once that is solved, I think this should happen ASAP. I'm currently flagging a major issue for the "newbie" use case).

[ruyadorno]

As I mentioned in the last TSC call in which we discussed this, I believe it's important to have an "Alternatives Considered" writeup in the Corepack docs and/or readme in which the team documents the reason why the Node.js runtime should ship Corepack by default and why it should be enabled by default vs alternatives such as using asdf, Volta, Docker, Corepack as a separate binary or any other environment wrapper/manager solution.

[GeoffreyBooth]

I reached out to the npm team. They are interested in finding a solution that can work for them as well as Yarn and pnpm: #51888 (comment)

Such a solution would probably mean changing or replacing the packageManager field, which seems to be under consideration anyway: nodejs/corepack#402 (comment)

I think we should let both efforts play out before we unflag Corepack.

[nickserv]

@GeoffreyBooth Couldn't npm just be another valid package manager in packageManager in this case?

For example:

"packageManager": "[email protected]"

[GeoffreyBooth]

Couldn’t npm just be another valid package manager in packageManager in this case?

It could if the npm team would be willing to support that syntax, but they’re not. They want something that defines version ranges, not a particular version; what should happen when validation fails; and other concerns. We’ve started sketching out an alternative configuration block in openjs-foundation/package-metadata-interoperability-collab-space#15.

[MylesBorins]

One thing that was called out that still has not been clarified is

1. How do new package managers get added to Corepack
2. How do they get turned on by default in Node.js

The current documentation simply says open a PR and it will land and it can become part of the default list after a couple of months. There is no documentation at all regarding how those package managers end up shipping in Node.js

This doesn't seem like explicit enough governance for something that will result in jumper binaries being installed on every system that installs Node.js

[arcanis]

Such a solution would probably mean changing or replacing the packageManager field, which seems to be under consideration anyway: nodejs/corepack#402 (comment)

I don't foresee the packageManager field changing. You're free to debate between yourselves of another syntax to add on top of it if you feel so strongly about it, but the current one has been proved to work well for the past years, has support from both package managers Corepack was initially intended for, and is already integrated in various places. I will strongly push back on removing or deprecating it. I trust you to know how to expand a design without breaking it.

The current documentation simply says open a PR and it will land and it can become part of the default list after a couple of months. There is no documentation at all regarding how those package managers end up shipping in Node.js

What do you suggest more than what is the current status for PRs to be merged in this repository? ie.

Two collaborators must approve a pull request before the pull request can land. (One collaborator approval is enough if the pull request has been open for more than 7 days.) Approving a pull request indicates that the collaborator accepts responsibility for the change. Approval must be from collaborators who are not authors of the change.

If a collaborator opposes a proposed change, then the change cannot land. The exception is if the TSC votes to approve the change despite the opposition. Usually, involving the TSC is unnecessary. Often, discussions or further changes result in collaborators removing their opposition.

[Ethan-Arrowood]

My apologies @arcanis that you didn't receive the notification about yesterday's meeting discussing the devEngines field and how it relates to npm/corepack. I posted about it in a couple of the ongoing threads (1, 2), as well as tweeted about it. It was also included in the meetings agenda a couple days ahead of time, and mentioned in the groups slack channel. I'll be more direct next time to ensure everyone knows they are invited.

The meeting primarily discussed the "devEngines" property. Including what steps would be needed to consider it "adopted", as well as how it could impact the ongoing corepack/npm debates. You can see the meeting notes here: openjs-foundation/package-metadata-interoperability-collab-space#17

[Ethan-Arrowood]

@aduh95 I am echoing other's request for changes to ensure my opinion is similarly heard. Unfortunately, a simple +1 on someone else's request-for-changes does not have the same effect as an individualized review.

When the group can reach consensus on the inclusion of corepack I am happy to approve necessary changes.

[ronag]

This is quite a long thread now. Can some summarise the current technical objections to enable corepack by default? @aduh95 @GeoffreyBooth? Is it mainly the security parts atm?

[GeoffreyBooth]

Can some summarise the current technical objections to enable corepack by default?

I was asked this in the 2024-03-06 TSC meeting and I gave a list then, which is in the meeting notes: https://github.com/nodejs/TSC/blob/main/meetings/2024-03-06.md#nodejsnode. There have been objections raised since then, many of which are summarized in the top post of #52107.

[BridgeAR]

I believe enabling corepack is what we all want to achieve. I am not having a full picture of what measures were taken to guarantee security concerns are fully addressed. I would love to have a short writeup about all measures including all concerns in concise language. For now, I can only abstain.

[darcyclarke]

@BridgeAR / @ronag / @GeoffreyBooth, I wrote a pretty succinct list of issues I'm aware of but I'll reiterate/link them here for posterity (others should feel free to do the same to save new readers time):

1. Adds yarn-specific codepaths which must be constantly updated/maintained/monitored
   (ex. Hash validation failed for yarn when COREPACK_NPM_REGISTRY is set on one side corepack#435)
2. Inconsistent Distributions & Versions
3. Differing Artifacts
4. Broken or Missing Versions
5. Integrity Mismatches for the Same Artifact
6. Creates a New Source-Of-Truth (config.json) for Distributions instead of Relying on a Canonical Registry (registry.npmjs.org) & Aligning with Existing Tools (ex. npm, yarn, pnpm, bun etc.)
7. Changing the Recommended Package Manager should be Discussed Separately from Enabling Corepack by Default

Edit: it looks like the anchor links above won't work if you haven't expanded the conversation & #51886 (comment) is visible - if you click on a link & it doesn't work, it's likely nested in that comment

[mcollina]

I changed my mind due to nodejs/corepack#495.

Corepack poses a significant security risk as it is.

[ovflowd]

There are way too many -1 on this PR. Has the TSC conveyed and reached a consensus here? What are the next steps here?

[mhdawson]

My understanding of the current state is that:

1. The package maintenance team was asked for recommendation, which ended up as - https://github.com/nodejs/package-maintenance/blob/main/docs/version-management/proposal-revise-downloads-page.md
2. The next steps were to propose changes to the installation instructions on Nodejs.org as a step in that direction
3. This came up in a TSC meeting a number of weeks ago and people seemed willing to wait for 2)

[trivikr]

What would be recommendation by package maintenance team for Node.js users who use the packageManager field in their package.json to manage their versions through corepack?

The SourceGraph search on Open Source code shows tens of thousands of repos which use it https://sourcegraph.com/search?q=context:global+%27%22packageManager%22:+%22%27+path:package.json%24&patternType=keyword&sm=0

[mhdawson]

@trivikr. From my understanding of the discussion and recommendations, with the target end state with Node.js not being involved in setting up the bootstrap components today, it would be a few manual steps:

1. install node.js (since corepack cannot currently do node.js install)
2. install corepack
3. use corepack to install the package manager

Agreed that having multiple steps is not the final target end state, but the hope is over time if corepack is widely used somebody would provide a simpler way to do those three in one step. corepack might even grow some scripts/install helpers to help with that.

If the user uses some other tool to bootstrap it could do the equivalent steps to get a Node.js version and pacakge manager version as documented in the package.json installed.

Ideally the user installs X, and then X will install all required bootstrap components from the package.json which today I think would include Node.js and the package manager. Then the user can just do the X install.