[Feature] nodeLinker: pnpm option ?

[crubier]

• I'd be willing to implement this feature
• This feature can already be implemented through a plugin (Maybe?)

Describe the user story

• nodeLinker: node-nodules (and the npm package manager) is astronomically long to install on some large monorepos and takes an incredible amount of disk space
• nodeLinker: pnp is quick to install, but has a large runtime impact as of today ( See Benchmark PnP runtime overhead #1817 or [Bug] Extremely slow yarn run xxxx commands on large monorepo  #973 for example), and caused a lot of packages to break in the past, and still requires a lot of maintenance from yarn maintainers. In hindsight, this solution may be a bit too radical, the ecosystem is not ready, and might never be.

Describe the solution you'd like

• We could add a nodeLinker: pnpm option to yarn, to use the same strategy as pnpm uses. (See https://pnpm.js.org/blog/2020/05/27/flat-node-modules-is-not-the-only-way )
• Maybe this could be done as a plugin

Describe the drawbacks of your solution

• It would probably be a (very) large effort to implement. But since both project are coded in typescript, some code reuse might be possible.
• I know it's probably a very unrealistic feature request.

Describe alternatives you've considered

• Solving all performance impacts caused by pnp in yarn might be doable or easier. But I am starting to feel a bit desperate about pnp. It will always be high maintenance: Anyone can create a package the breaks under pnp. And once the bugs are solved, comes the runtime overhead. Solving the bugs while keeping performance high might not even be possible.
• We are considering moving our monorepo to pnpm. It makes our scripts 2x-10x faster depending on the use case... Install is only 2x longer.

Additional context

We love yarn berry and everything it provides (plugins, constraints etc.). The only source of problems with it is pnp. On the other hand we recently tried pnpm and saw large runtime performance improvements.

[arcanis]

I don't think we have the resources to support three different linkers at the moment. It's very possible to implement a pnpm style install, and we would happily provide tips to do so, but I think it should come from the outside.

Note that, personally, I don't have the same assessment regarding PnP: applications that don't follow the semantic cause as many problems, but harder to find, and the runtime overhead doesn't look that significant - that is, in master, after fixing the regression you reported. In any case, the PnP runtime never got particularly optimized (priorities!), and I'm optimistic that there are low hanging fruits to find.

[larixer]

@crubier I would be interested to experiment with putting together proof of concept pnpm-like linker when I have time, probably after making pnp + nm linker hybrid mode working in the same monorepo. Full-blown investment into pnpm-like linker on everyday basis while supporting nm linker as well, would be probably too much for me.

[zkochan]

It might be better to call it something like nodeLinker: node-modules-symlinks because pnpm will soon also support PnP (pnpm/pnpm#2908), so it might be confusing in the future, what nodeLinker: pnpm means.

(there's a funny typo in the issue: node-nodules)

[larixer]

@zkochan I don't think we have resources to recreate pnpm linker. I might just play with symlinks at some point in the future and explore what benefits we can gain from using them with node_modules installs. For sure such a linker that make some usage of symlinks will not be named pnpm.

[borekb]

Let's say we'd be fine having to install pnpm separately; would it then be possible to create a Yarn plugin that is a wrapper around the pnpm binary? I guess the main problem is that the lockfiles are different but if pnpm could read yarn.lock, are there any other major problems with this idea?

[MarcCelani-at]

Hi folks. Very excited to try out this new linker.

This plugin is missing from https://github.com/yarnpkg/berry/blob/master/plugins.yml . I'm assuming that is intentional but reporting just in case. When can we expect to see this new plugin released?

[merceyz]

It's a builtin plugin so nothing to import, it's currently in a canary release.

yarn set version berry && yarn set version canary