RFC: Add npm workspaces #103
Conversation
ruyadorno
added
Agenda
There was a problem hiding this comment.
There's a few primary use cases i'd like to see explicitly called out as supported:
- multirepo
- The root is
private: true, and only lists deps/devDeps/peerDeps that apply to repo-level maintenance - each subpackage lists every dep/devDep/peerDep that it needs, just as if it lived in a repo by itself
- subpackages depended on by other subpackages are able to be linked for dev/tests, but also able to be unlinked, so the explicit versions referenced are tested
- dep/peerDeps are always published with version numbers, never as links/file refs
- subtle peer dep conflicts are surfaced, ideally based on the published refs and based on linked refs
- something custom (eg, eslint-plugin-import)
- the root is a package that's published
- subpackages also exist that are published (and npmignored by the root)
- the rest is the same as in a multirepo
- monorepo
- basically nothing is published, everything is private: true
- the rest is the same as in a monorepo, except that everything's always linked because there's no published versions
I'm really perplexed about a workspace root that's also a package getting published. How does that work when installing a workspace root as a dependency? Do we just strip off the |
|
presumably you’d want to do that anyways if you didn’t require private: true for workspaces to work. see https://github.com/benmosher/eslint-plugin-import for that particular example. |
accepted/0000-workspaces.md
Outdated
|
|
||
| ### 4. Publishing workspaces | ||
|
|
||
| A workspace may not be published to the registry and for all publishing purposes having a valid `"workspace"` entry in a `package.json` is going to be the equivalent of `"private": true`. |
There was a problem hiding this comment.
This seems like a needless restriction and blocks some interesting use cases. I can imagine a developer experience which involves teams publishing workspaces as a mechanism for sharing a setup. You could just specify that installing a published package with a workspaces key will do nothing, and then let userland tooling extend this if it wants.
There was a problem hiding this comment.
Do we just strip off the workspaces declaration when installing
I would think for a use case like sharing a setup you would not strip it, only ignore it.
Do the workspace packages get listed as dependencies?
No, as with other workspace implementations you only get it when developing locally with it as the top level package. So to publish a package it must define all that it requires when installing from the registry.
There was a problem hiding this comment.
So if it's not a top of tree module (root or link target, not a deep in node_modules) then the workspaces object is irrelevant? That could work I guess.
There was a problem hiding this comment.
@wesleytodd this is looking interesting 👍 assuming there's no technical limitations that might prevent it - I'm onboard with removing the restriction
You could just specify that installing a published package with a workspaces key will do nothing, and then let userland tooling extend this if it wants.
just to clarify, in this case there's no filtering out mechanism at the moment of publishing a top-level workspace 🤔 meaning all the contents of nested packages within it get published along with that top-level package?
There was a problem hiding this comment.
I think it's the right call to make a "workspaces" field in package.json equivalent to "private": true. "Sharing configuration" (for an identical workspace, what?) should be a separate concern.
There was a problem hiding this comment.
meaning all the contents of nested packages within it get published along with that top-level package?
My Yeah you would use the normal methods for specifying which files make it into the tarball if you wanted different behavior.
Sharing configuration" (for an identical workspace, what?) should be a separate concern.
Just wondering, why do you think this? To me this clause of "workspaces means private" makes it a concern here. I am fine having that as a separate conversation as long as this clause does not make it into the implementation.
There was a problem hiding this comment.
@wesleytodd The manifest that coordinates the workspaces, as I've commented elsewhere, should not be conflated with a manifest that specifies a publishable package. The fact that Lerna can do that is an accident of history, and does not represent what I believe to be a robust pattern for managing multi-package repositories.
There was a problem hiding this comment.
Publishing it is a distribution mechanism, and the registry is the distribution mechanism in our ecosystem. If folks find novel ways to use this as both a distribution mechanism and a tool to build better developer experience, GREAT! I don't see this needless restriction gaining us anything at this time.
While it might not not help your monorepo-centric tool doesn't mean that it cannot help other types of setups. Do you think that your worries hold up when you move outside of a monorepo? Do you have suggestions for supporting multi project workspaces outside of a monorepo?
My single biggest concern for this implementation: that it not be tightly coupled to the monorepo setup. If npm just follows yarn and lerna on this I will continue not to be able to use them for my main use cases. Most OSS projects are a collection of projects in their own repos. Some popular ones outgrew the tooling (a separate problem I think we can solve), but the vast majority have not. If we do not make this a top priority for workspaces in npm I think we will have missed an opportunity to appeal to a much larger audience.
I would really love for us to get past this focus on "monorepo" vs "multirepo". Can we just talk about Multi-Project support? The repo part is a vestige of our node module resolution mechanism being tied to the filesystem structure and a repo being the only reliable way of ensuring file structures across systems. In the future we will get a lot more value in decoupling this all from the filesystem layout, and building a workspace implementation by copying Yarn v1 is already behind the state of the art. There are a bunch of examples of decoupling the file system from the module resolution mechanism: Yarn PnP, tink and Import Maps. I think we should strongly think about how multi-project looks when we have a future where the directory structure does not dictate where modules live on the filesystem. |
Agreed! And they're all outside the scope of this rfc ;) Our intent is to use a fallback fs implementation (ie, tink), so we still have to figure out what the file system will look like. If we then virtualize it, we still need a way to unwind it when requested and need to know what to virtualize. So, all this gross disk stuff needs to get worked out regardless. Also, files on disk has its advantages. |
What happens if they're not npmignored? I think maybe we should treat them just like normal package contents, included by default. |
|
@isaacs your points are a bit unclear to me. I am not sure what a "fallback fs" means here, and since this whole proposal is centered around the existing disk layout based methodology then how is that not applicable? As for "files on disk has its advantages", I don't disagree, but could you elaborate on how that applies specifically to this workspace implementation? |
Fallback fs is the approach used by tink. We model out the file system, and instead of providing an alternative module loader, like pnp does, we virtualization the fs module in node, such that any ENOENT in a node_modules folder falls back to the appropriate package contents in a single global cache. On advantage of this approach is that you can always unwind to the "real" location, and no changes are required in the vast majority of use cases. My point is that, even with such a virtualized system, we still need to decide the shape of it as a file system. The biggest advantage of files on disk (or a sufficiently convincing fallback fs) is that it works trivially with the other tools, including editing to float patches, webpack and other transpilers, etc. |
Ah, I hadn't looked at that part of what it was doing. I am not opposed to this approach but I would like to know why you think that is better than a loader as the runtime portion? It seems like this reaches out of the package installer's domain and is also more heavy handed.
It seems like this is possible from any of the approaches we might take here, right?
I guess since the "shape of it as a filesystem" maps directly to the shape of the optimized dep tree? Because if you choose something like an Import Map over a virtualized fs that relationship doesn't really matter anymore.
This is a great point and is the thing I have always been worries about with tink and PnP. Are those other approaches finding a solution to this? I do think we should have an answer for this as it relates to workspaces so the design of that doesn't have to change too much in relation. |
Co-Authored-By: Wes Todd <[email protected]>
Co-Authored-By: Wes Todd <[email protected]>
|
@ljharb having the final implementation being flexible enough to support these use cases is my
by "able to be linked for dev/tests" are you suggesting anything more complex than just the symlinking generated after "also able to be unlinked, so the explicit versions referenced are tested" this sounds to me like something that might be outside of the scope of an initial workspaces implementation but to be clear, in the current proposal the way to unlink a subpackage (or have it be fetched from the registry instead) would be to define a semver range that is not satisfied by the
This item seems to be outside of the scope of this initial work 🤔 at least in its current form - how could we better handle that scenario? Can you (or anyone else really) provide some examples/ideas?
Are these manually Feel free to suggest changes to the document if you want to see any specific wording in the final ratified version. And thanks for the list! 😄 it provides a great initial checklist for all test cases we'll want to make sure to have in arborist. |
the current main flaw with linking is that every peer dep has to be separately linked too; these would need to be solved.
What I mean is, there's two modes that are important: development (where links are ideal) and CI/production (where i want to test what consumers will get when they install). In other words, if subpackage A depends on subpackage [email protected], i'd expect in dev that B would be the current working directory, but that when "unlinked", or in prod, that it'd be the published v2.0.0.
Whether it's in scope or not, I'd consider it a blocker. Issues around peer deps are wildly multiplied in a workspaces scenario - for example, enzyme has a handful of "react adapters", each of which requires a different, conflicting version of react. In airbnb's javascript style guide, there's two eslint config packages (the main one, and the base), the main requires the base, and both the main and the base peer depend on eslint as well as a number of eslint plugins, some overlapping. Basically, my concern is that a workspaces development model makes it much easier to obscure issues around dependency conflicts, and I think it's highly critical that the tooling surface these.
Yes, I think it's totally fine to leave "what's in the tarball" management to the user, but it might be nice to warn the user if one published package would end up containing another. |
FWIW, I really don't like this pattern. The root manifest of a multi-package repository should be the place you coordinate workspaces, test/dev dependencies, and other bits of "global" configuration. Making the root manifest also a published package is not gaining enough benefits to outweigh the drawbacks (confusing, arbitrary ignores) in my opinion. |
|
Just to hoist this topic to the top level thread: I strongly think we need to think both about monoreps (well trodden ground today) as well as multirepo. Most projects I work on are multirepo. We have great reasons for this and we will not be changing. Today our tooling provides almost no support for this, and so I have many DX workarounds. To be clear, I am not against monorepos. I helped move my last companies codebase into one, it is great for many types of projects. But not all. We can do better for this majority of package authors, and we should. |
ftr neither do i, but the best way for repos that use it to migrate to a better pattern is if they have reliable tooling available that supports both patterns - like npm. |
Co-Authored-By: Wes Todd <[email protected]>
|
Work is progressing on this over here: npm/arborist#50 |
|
From those three use cases that @ljharb pointed out at the beginning, the third one is the most important:
The first two are already supported by third party tools, like Yarn, Lerna and Microsoft Rush. Breaking a private big project into a bunch of small private pieces is way more common than breaking it into public packages. And yet we don't have a simple solution to such a simple problem. People don't do this in Javascript projects because there is no simple way to do that. So in any big project with a single package.json, every new installed dependency are available on the entire source code, which is a nightmare from the point of view of software architecture. During the install, if some dependency is a workspace, it should be handled differently if it is a local dev install
Yarn v2 introduced the |
|
Is someone working on this? A lot of people would migrate from yarn as soon as this feature is implemented |
|
@lmcarreiro that sounds like an interesting idea! I'll bring that up during the next OpenRFC call but as work on this RFC has been finalized it might be better to follow up with it in nodejs/modules#117 @remorses that'd be me 😬 you can follow up progress in the link @isaacs shared above or just follow up v7 progress in general at npm/arborist 😊 |
|
I noted the PR doesn't contain all the historic info on the ratification process, so just for the sake of documenting these details I'm going to leave these references here:
|
|
I'm a little late to the party here, but reading the RFC it seems that there will be a single |
|
@smably that's a very good question I would love to hear @evocateur thoughts about it, since I believe there might be tooling out there from the Lerna community around it from the time Yarn v1 first shipped workspaces. That said the idea is that once npm@7 is shipped, Lerna+npm users will now have the option to migrate to a useWorkspaces bootstrap setup similar to the way Lerna+Yarn users were already familiar with - migrating a project to that kind of setup might be non-trivial though, the mismatching |
|
@ruyadorno Makes sense. I don't think there's a way to merge multiple Yarn lockfiles because Yarn's lockfile format maps each semver range directly to a resolved version, so it's not possible to maintain different resolutions for the same semver range in different workspaces. E.g., one of my (pre-workspaces) packages could have in its and another could have and there'd be no way to represent both of those with a single Whereas in this RFC, if I'm reading the |
Right right, the npm lockfile format preserves information about nested dedupped modules which makes it theoretically possible to have them merged but you would still need some manual intervention in order to handle |
Introduces support to workspaces; adding ability to build an ideal tree that links defined workspaces, storing and reading lockfiles that contains workspaces info and also reifying installation trees properly symlinking nested workspaces into place. Handling of the config definitions is done via @npmcli/map-workspaces module added. refs: - https://github.com/npm/rfcs/blob/ea2d3024e6e149cd8c6366ed18373c9a566b1124/accepted/0026-workspaces.md - https://www.npmjs.com/package/@npmcli/map-workspaces - npm/rfcs#103
Introduces support to workspaces; adding ability to build an ideal tree that links defined workspaces, storing and reading lockfiles that contains workspaces info and also reifying installation trees properly symlinking nested workspaces into place. Handling of the config definitions is done via @npmcli/map-workspaces module added. refs: - https://github.com/npm/rfcs/blob/ea2d3024e6e149cd8c6366ed18373c9a566b1124/accepted/0026-workspaces.md - https://www.npmjs.com/package/@npmcli/map-workspaces - npm/rfcs#103 PR-URL: #50 Credit: @ruyadorno Close: #50 Reviewed-by: @isaacs
npm workspaces
Summary
Add a set of features to the npm cli that provide support to managing multiple packages from within a singular top-level, root package.
Motivation
This feature has been requested by the community for a long time. The primary motivation behind this RFC is to fully realize a set of features/functionality to support managing multiple packages that may or may not be used together.
The name “workspaces” is already established in the community with both Yarn and Pnpm implementing similar features under that same name so we chose to reuse it for the sake of simplicity to the larger community involved.
Detailed Explanation
After sourcing feedback from the community, there are 2 major implementations/changes required in the npm cli in order to provide the feature set that would enable a better management of nested packages.
The set of features identified in this document are the ones that are essential to an initial MVP of the npm workspaces support. The community should expect further development of this feature based on the feedback we collected and documented at the end of this RFC.
Rationale and Alternatives
First and foremost there’s the alternative of leaving the problem set for userland to solve, there’s already the very popular project Lerna that provides some of these features.
Also available is the alternative of supporting only the install (or bootstrap as Lerna names it) aspect of this proposal, following a less feature-rich approach but one that would still enable the basic goal of improving the user experience of managing multiple child packages but from all the feedback collected during the research phase of this RFC, this alternative is much less desirable to the community of maintainers involved.
Implementation
1. Workspaces configuration: Making the npm cli workspace-aware
We're following the lead of Yarn in supporting the
workspacespackage.jsonproperty which defines a list of paths, each of these paths may point to the location of a workspace in the file system but it also support globs.package.jsonexample:package.jsonshorthand example:The npm cli will read from the paths and globs defined in this workspaces configuration and look for valid
package.jsonfiles in order to create a list of packages that will be treated as workspaces.Note: The
packagesproperty should support familiar patterns from npm-packlistfilesdefinition such as negative globs.2. Installing dependencies across child packages
Change
npm install(arborist) behavior to make it properly install dependencies for every workspace defined in the workspaces configuration described above.Arborist should also be aware of all workspaces in order to correctly link to another internal workspace should it match the required semver version of an expected dependency anywhere in the installing tree. e.g:
For the initial workspaces implementation, we're going to stick with arborist's default algorithm that privileges hoisting packages but will place packages at nested
node_moduleswhenever necessary.Examples
Expanding on symlinking structure and
package-lockfile shape.Given a npm workspaces setup with the following contents:
Will result in the following symlinking structure:
And the following
package-lock.jsonfiles:NOTE: The following lockfile is for illustration purpose only and its final shape might differ.
Dictionary
During the discussions around this RFC it was brought up to our attention that a lot of the vocabulary surrounding what the larger JavaScript community understands as "workspaces" can be confusing, for the sake of keeping the discussion as productive as possible we're taking the extra step of documenting what each of the terms used here means:
package.jsonthat declares where to find workspaces for this Top-level workspace package.package.jsonPrior Art