From 790d816ecfdddfce3fad386fec4d28a76e30b8d7 Mon Sep 17 00:00:00 2001 From: David Herman Date: Fri, 7 Dec 2018 16:03:02 -0800 Subject: [PATCH 1/7] RFC: Toolchains --- text/0000-toolchains.md | 155 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 text/0000-toolchains.md diff --git a/text/0000-toolchains.md b/text/0000-toolchains.md new file mode 100644 index 0000000..343f82a --- /dev/null +++ b/text/0000-toolchains.md @@ -0,0 +1,155 @@ +- Feature Name: toolchains +- Start Date: 2018-12-07 +- RFC PR: +- Notion Issue: + +# Summary +[summary]: #summary + +This RFC describes a design for **toolchains**, the central unifying concept of Notion. + +# Motivation +[motivation]: #motivation + +Notion toolchains attempt to achieve several goals: + +- **Reproducible project tooling.** With this design, Notion should make it convenient and reliable for all contributors on a project to get the same exact version of the Node runtime, the package manager, and any package binaries configured for use by `package.json`. +- **Reconciling the use of npm as a software distribution platform with best practices.** It's considered a best practice for project development to avoid ever installing global packages. But npm is a popular and convenient platform for distributing command-line tools. This design reconciles the tension by isolating _user tools_, which are installed for personal use, making them invisible to JS project scripts, and distinguishing them from project tools. +- **Install and forget.** This design solves the problem of _tool bitrot_, where a tool stops working because of upgrades to the system Node version. It also avoids the need to reinstall global tools every time a new Node version is provisioned. With Notion, once you install a user tool and get it working, it keeps working unless you deliberately decide to change or uninstall it. +- **Lightweight cognitive model via statelessness.** This design allows Notion users to generally avoid thinking at all about the version of Node that's currently installed, instead relying on saved configuration to ensure that tools and projects have already declaratively specified their Node platform version. + +## User stories + +To demonstrate this, here are three representative user stories. + +### Project tools: `tsc` + +A project maintainer selects a version of TypeScript in `package.json`: + +```js + ... + "dependencies": { + "typescript": "^3.2" + }, + "platform": { + "node": "10.10.0" + } + ... +``` + +and the lockfile pins TypeScript to version 3.2.2. + +Any Notion user who checks out the project gets an identical development experience when working within the project directory tree. Namely, typing +``` +tsc +``` +runs version 3.2.2 of the TypeScript compiler, using Node 10.10.0 as the runtime. + +### User tools: `surge` + +An end user of the [surge.sh](https://surge.sh) service installs their CLI tool that is deployed as an npm package: +``` +notion install surge +``` +Assuming the `surge` tool selects Node 11.4.0 in its `"platform"` manifest (it's OK if not; more details below), the CLI tool is installed in the user toolchain and pinned to Node 11.4.0. + +From this point on, unless the user changes their toolchain, running +``` +surge +``` +from the command-line always runs this tool using Node 11.4.0. + +### User tools in projects: `pexx` + +The [project-explorer](https://sdras.github.io/project-explorer-site/) tool is a useful CLI tool that lets you inspect JS projects, but is not itself a tool that a project would use in its scripts. + +A user can install this tool to their toolchain: +``` +notion install project-explorer +``` +Assuming the `project-explorer` manifest selects Node 8.12.0 in its manifest, the tool is installed pinned to that version of the Node runtime. + +When the user runs +``` +pexx myproject +``` +the `pexx` binary runs with Node 8.12.0---even if `myproject` specifies a different version of Node in its `"platform"` spec. + +# Pedagogy +[pedagogy]: #pedagogy + +This section lists the set of concepts that users may encounter using Notion. The first two, **tools** and **toolchains**, are central to using Notion. The latter two, **shims** and **platforms**, are lower-level primitives that may be helpful for more implementation-oriented users. + +## Tools + +There are three types of tools: + +- **Node runtime:** The version of Node itself, which in particular dictates which `node` binary gets invoked. +- **Package manager:** A version of npm or a version of Yarn. (In the future we may want to add support for other package managers such as pnpm or tink.) +- **Package binary:** An executable published and distributed as part of an npm package. + +## Toolchains + +There are two distinct toolchains managed by Notion: + +- **User toolchain:** This is the set of tools installed for general-purpose, personal CLI use at the console. They are not associated with any JS project. They are available for personal use when running commands manually or in shell scripts, but not when running package scripts in a JavaScript project. +- **Project toolchain:** This is the set of tools a project has pinned via its manifest and lockfile. These always take precedence over user tools, and are the only tools available in the `PATH` when running package scripts in the project. + +## Primitive: Shims + +Notion shims intercept calls to tools and redirect execution to the right executable based on the environment and current directory. + +Users may already be familiar with shims, since these are commonly used by other version managers. + +## Primitive: Platforms + +A **platform spec** is a complete description of a version of the Node platform: + +- An exact version of the Node runtime. +- An optional exact version of npm. +- An optional exact version of Yarn. + +A **platform image** is an immutable instantiation of a platform spec on disk. It can be thought of analogously to a container image, but for the Node platform as opposed to an operating system. + +# Details +[details]: #details + +## Toolchains + +A toolchain is a pair of a platform image and a set of pinned package binaries. + +A pinned package binary is a pair of an exact version of a package binary and a platform image. + +## Platform specs + +In the manifest, an omitted npm version defaults to the version bundled with the specified Node runtime. An omitted Yarn version defaults to Yarn being unavailable, meaning that invoking `yarn` will produce an error. Both of these can be explicitly set to `null`, which means the specified tool is unavailable. + +## Platform images + +The definition of a platform image is a pair consisting of a Node version and a Yarn version. + +A Node version is a pair consisting of an exact Node runtime version, and either an exact npm version or `null`. + +A Yarn version is either an exact Yarn version or `null`. + +## Pinning + +The `"platform"` section of `package.json` selects the platform image associated with a package. This is a change from earlier versions of Notion, which used the key `"toolchain"`, since the toolchain consists of _both_ the platform image _and_ the pinned package binaries (see above). + +Users can pin the platform by manually editing the `package.json` `"platform"` section or via [`notion pin`](https://github.com/notion-cli/rfcs/pull/24). + +# Critique +[critique]: #critique + +It's natural to question whether pinning for reproducibility is worth the cost of extra fetching. An alternative approach would be to allow projects to specify less precise version requirements (such as the ranges typically expressed in the `"engines"` field of `package.json`) and assume most differences will be benign. However, behavioral divergences between versions of Node do happen and are tricky bugs to nail down. Putting in extra work up front to ensure that these divergences cannot happen, by construction should pay dividends when scaled across the Node ecosystem. And over time, we can investigate optimization techniques to save time and disk space for fetching multiple similar versions. + +Theoretically, it might make more sense to put the `"platform"` section in a lockfile. But since there isn't a standardized single lockfile format for JS, and those formats aren't extensible, and we don't want to impose a whole new file to add to JS projects, using the package manifest seemed like the least imposition on users. + +Another reasonable criticism is that pinning the Node version for tools in the user toolchain means that users will not automatically benefit from performance and security improvements in Node. There are a couple of reasons this is outweighed by the benefits of pinning. First, as users upgrade the tool version itself, they will automatically get re-pinned to the newer version of Node specified by newer versions of the tool's `package.json`. Second, we should at least allow users the option to override the tool's specified Node version. But by letting the tool choose its platform version by default, users have a stronger guarantee that the tool will work and continue to work consistently based on how it was tested. + +# Unresolved questions +[unresolved]: #unresolved-questions + +- What about the (rarer) cases of user tools that want to work with the current project’s toolchain choice instead of a statically-bound choice? For example, tools that wrap the Node REPL. +- Should we consider some kind of update notification mechanism to inform you when your user tools are out of date? +- What kinds of performance optimizations can we offer to make stateless platform images super fast? From 7d79b7cc74aba4cd635d4a55d197616fed4acacb Mon Sep 17 00:00:00 2001 From: David Herman Date: Sat, 8 Dec 2018 10:15:27 -0800 Subject: [PATCH 2/7] Added details section for installation. --- text/0000-toolchains.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/text/0000-toolchains.md b/text/0000-toolchains.md index 343f82a..79835cb 100644 --- a/text/0000-toolchains.md +++ b/text/0000-toolchains.md @@ -138,6 +138,14 @@ The `"platform"` section of `package.json` selects the platform image associated Users can pin the platform by manually editing the `package.json` `"platform"` section or via [`notion pin`](https://github.com/notion-cli/rfcs/pull/24). +## Installation + +The `notion install` command installs a tool to the user toolchain. + +When installing a version of the Node runtime, Notion also installs the default version of npm bundled with that verison of Node. This can be overridden with a subsequent `notion install npm` command. + +When installing a package binary to the user toolchain, Notion checks the package for a `"platform"` key to pin the user tool to a specific platform image. If there is no `"platform"` key in the package manifest, it defaults to the current platform image configured in the user toolchain. If there is no current platform image, `notion install` fails with an error message suggesting the user choose at least a Node runtime version. + # Critique [critique]: #critique From b8649aefe463ebfa37db48139cfaa7e1761c8d96 Mon Sep 17 00:00:00 2001 From: David Herman Date: Fri, 18 Jan 2019 09:42:28 -0800 Subject: [PATCH 3/7] Updated to reflect the idea of just one toolchain, managed entirely via `install/uninstall/update`. --- .../{0000-toolchains.md => 0000-toolchain.md} | 63 ++++++++++++------- 1 file changed, 41 insertions(+), 22 deletions(-) rename text/{0000-toolchains.md => 0000-toolchain.md} (64%) diff --git a/text/0000-toolchains.md b/text/0000-toolchain.md similarity index 64% rename from text/0000-toolchains.md rename to text/0000-toolchain.md index 79835cb..9e33949 100644 --- a/text/0000-toolchains.md +++ b/text/0000-toolchain.md @@ -1,4 +1,4 @@ -- Feature Name: toolchains +- Feature Name: toolchain - Start Date: 2018-12-07 - RFC PR: - Notion Issue: @@ -6,17 +6,17 @@ # Summary [summary]: #summary -This RFC describes a design for **toolchains**, the central unifying concept of Notion. +This RFC describes a design for the Notion's central unifying design concept: the **toolchain**. # Motivation [motivation]: #motivation -Notion toolchains attempt to achieve several goals: +The Notion toolchain is designed with several goals: - **Reproducible project tooling.** With this design, Notion should make it convenient and reliable for all contributors on a project to get the same exact version of the Node runtime, the package manager, and any package binaries configured for use by `package.json`. -- **Reconciling the use of npm as a software distribution platform with best practices.** It's considered a best practice for project development to avoid ever installing global packages. But npm is a popular and convenient platform for distributing command-line tools. This design reconciles the tension by isolating _user tools_, which are installed for personal use, making them invisible to JS project scripts, and distinguishing them from project tools. -- **Install and forget.** This design solves the problem of _tool bitrot_, where a tool stops working because of upgrades to the system Node version. It also avoids the need to reinstall global tools every time a new Node version is provisioned. With Notion, once you install a user tool and get it working, it keeps working unless you deliberately decide to change or uninstall it. -- **Lightweight cognitive model via statelessness.** This design allows Notion users to generally avoid thinking at all about the version of Node that's currently installed, instead relying on saved configuration to ensure that tools and projects have already declaratively specified their Node platform version. +- **Reconciling the use of npm as a software distribution platform with best practices.** It's considered a best practice for project development to avoid ever installing global packages. But npm is a popular and convenient platform for distributing command-line tools. The toolchain design reconciles this tension by isolating _user tools_, which are installed for personal use, making them invisible to JS project scripts, and distinguishing them from _project tools_. +- **Install and forget.** This design solves the problem of _tool bitrot_, where a tool stops working because of Node upgrades. It also avoids the need to reinstall global tools every time a new Node version is provisioned. With Notion, once you install a user tool and get it working, it keeps working unless you deliberately decide to change or uninstall it. +- **Low cognitive overhead via statelessness.** This design allows Notion users to generally avoid thinking at all about the version of Node that's currently installed, instead relying on saved configuration to ensure that tools and projects have already declaratively specified their Node platform version. ## User stories @@ -39,11 +39,11 @@ A project maintainer selects a version of TypeScript in `package.json`: and the lockfile pins TypeScript to version 3.2.2. -Any Notion user who checks out the project gets an identical development experience when working within the project directory tree. Namely, typing +Users can call the TypeScript compiler directly as long as they've installed it: ``` -tsc +notion install typescript ``` -runs version 3.2.2 of the TypeScript compiler, using Node 10.10.0 as the runtime. +Afterwards, running `tsc` from within the project directory runs version 3.2.2 of the TypeScript compiler, using Node 10.10.0 as the runtime. ### User tools: `surge` @@ -78,7 +78,7 @@ the `pexx` binary runs with Node 8.12.0---even if `myproject` specifies a differ # Pedagogy [pedagogy]: #pedagogy -This section lists the set of concepts that users may encounter using Notion. The first two, **tools** and **toolchains**, are central to using Notion. The latter two, **shims** and **platforms**, are lower-level primitives that may be helpful for more implementation-oriented users. +This section lists the set of concepts that users may encounter using Notion. The first two, **tools** and the **toolchain**, are central to using Notion. The latter two, **shims** and **platforms**, are lower-level primitives that may be helpful for more implementation-oriented users. ## Tools @@ -88,12 +88,11 @@ There are three types of tools: - **Package manager:** A version of npm or a version of Yarn. (In the future we may want to add support for other package managers such as pnpm or tink.) - **Package binary:** An executable published and distributed as part of an npm package. -## Toolchains +## Toolchain -There are two distinct toolchains managed by Notion: +The toolchain is a set of tools. The user adds to their toolchain with `notion install` and removes with `notion uninstall`. -- **User toolchain:** This is the set of tools installed for general-purpose, personal CLI use at the console. They are not associated with any JS project. They are available for personal use when running commands manually or in shell scripts, but not when running package scripts in a JavaScript project. -- **Project toolchain:** This is the set of tools a project has pinned via its manifest and lockfile. These always take precedence over user tools, and are the only tools available in the `PATH` when running package scripts in the project. +Each tool installed in the toolchain has a default version and platform, which can be overridden when the tool is invoked in a project that has a dependency on that tool. ## Primitive: Shims @@ -114,12 +113,10 @@ A **platform image** is an immutable instantiation of a platform spec on disk. I # Details [details]: #details -## Toolchains +## Toolchain A toolchain is a pair of a platform image and a set of pinned package binaries. -A pinned package binary is a pair of an exact version of a package binary and a platform image. - ## Platform specs In the manifest, an omitted npm version defaults to the version bundled with the specified Node runtime. An omitted Yarn version defaults to Yarn being unavailable, meaning that invoking `yarn` will produce an error. Both of these can be explicitly set to `null`, which means the specified tool is unavailable. @@ -134,18 +131,39 @@ A Yarn version is either an exact Yarn version or `null`. ## Pinning -The `"platform"` section of `package.json` selects the platform image associated with a package. This is a change from earlier versions of Notion, which used the key `"toolchain"`, since the toolchain consists of _both_ the platform image _and_ the pinned package binaries (see above). +The `"platform"` section of `package.json` selects the platform image associated with a package. (**Note:** This is a change from earlier versions of Notion, which used the key `"toolchain"`, since the toolchain consists of _both_ the platform image _and_ the pinned package binaries -- see above.) Users can pin the platform by manually editing the `package.json` `"platform"` section or via [`notion pin`](https://github.com/notion-cli/rfcs/pull/24). ## Installation -The `notion install` command installs a tool to the user toolchain. +The `notion install` command installs a tool to the user's toolchain. When installing a version of the Node runtime, Notion also installs the default version of npm bundled with that verison of Node. This can be overridden with a subsequent `notion install npm` command. When installing a package binary to the user toolchain, Notion checks the package for a `"platform"` key to pin the user tool to a specific platform image. If there is no `"platform"` key in the package manifest, it defaults to the current platform image configured in the user toolchain. If there is no current platform image, `notion install` fails with an error message suggesting the user choose at least a Node runtime version. +### Overriding the associated platform + +When used for a package tool, `notion install` accepts an optional `--node` parameter for overriding the package's specified platform version: + +``` +notion install surge --node=latest +``` + +## Uninstallation + +The `notion uninstall` command uninstalls a tool from the user's toolchain. + +## Updating + +Taken together, `notion uninstall` and `notion install` can be used to update a tool. Note that as the tool evolves over time, its authors may update its platform to newer Node versions. So as users upgrade their tools they will automatically get platform upgrades for the tool. But they also continue to be assured they are getting a version of the platform that the tool was tested with. + +The `notion update` command is a shorthand command for doing the same thing: +``` +notion update surge +``` + # Critique [critique]: #critique @@ -153,11 +171,12 @@ It's natural to question whether pinning for reproducibility is worth the cost o Theoretically, it might make more sense to put the `"platform"` section in a lockfile. But since there isn't a standardized single lockfile format for JS, and those formats aren't extensible, and we don't want to impose a whole new file to add to JS projects, using the package manifest seemed like the least imposition on users. -Another reasonable criticism is that pinning the Node version for tools in the user toolchain means that users will not automatically benefit from performance and security improvements in Node. There are a couple of reasons this is outweighed by the benefits of pinning. First, as users upgrade the tool version itself, they will automatically get re-pinned to the newer version of Node specified by newer versions of the tool's `package.json`. Second, we should at least allow users the option to override the tool's specified Node version. But by letting the tool choose its platform version by default, users have a stronger guarantee that the tool will work and continue to work consistently based on how it was tested. +Another reasonable criticism is that pinning the Node version for tools in the user toolchain means that users will not automatically benefit from performance and security improvements in Node. There are a couple of reasons this is outweighed by the benefits of pinning. First, as described above, updating tools will typically get platform updates. Second, users can still override the default with the `--node` parameter. # Unresolved questions [unresolved]: #unresolved-questions -- What about the (rarer) cases of user tools that want to work with the current project’s toolchain choice instead of a statically-bound choice? For example, tools that wrap the Node REPL. -- Should we consider some kind of update notification mechanism to inform you when your user tools are out of date? +- What about the (rarer) cases of user tools that want to work with the current project’s toolchain choice instead of a statically-bound choice? For example, tools that wrap the Node REPL. Maybe a special keyword for the platform like `"node": "user"`? We'd need to think through the semantics. +- Should we consider some kind of update notification mechanism to inform you when your user tools are out of date? Probably at least something similar to `brew outdated` so users can explicitly ask for the information. - What kinds of performance optimizations can we offer to make stateless platform images super fast? +- What about scenarios like CI and testing matrices, where users want to be able to specify different platform configurations without having to change the configuration file? Perhaps workflows similar to `ember try`? From 5a89c4246010b2206a44f16e381a6f536184e600 Mon Sep 17 00:00:00 2001 From: David Herman Date: Thu, 31 Jan 2019 14:49:37 -0800 Subject: [PATCH 4/7] Changes based on RFC discussion thread: - Rename `"platform"` key back to `"toolchain"` - Use standard Node "engine" terminology instead of "platform" - Use `"engines"` for choosing the default engine of a package binary - Fix the definition of "toolchain" --- text/0000-toolchain.md | 63 +++++++++++++++++++++++++----------------- 1 file changed, 37 insertions(+), 26 deletions(-) diff --git a/text/0000-toolchain.md b/text/0000-toolchain.md index 9e33949..f2f71f8 100644 --- a/text/0000-toolchain.md +++ b/text/0000-toolchain.md @@ -31,7 +31,7 @@ A project maintainer selects a version of TypeScript in `package.json`: "dependencies": { "typescript": "^3.2" }, - "platform": { + "toolchain": { "node": "10.10.0" } ... @@ -51,7 +51,7 @@ An end user of the [surge.sh](https://surge.sh) service installs their CLI tool ``` notion install surge ``` -Assuming the `surge` tool selects Node 11.4.0 in its `"platform"` manifest (it's OK if not; more details below), the CLI tool is installed in the user toolchain and pinned to Node 11.4.0. +In this user story, the `surge` tool is published with `"engines": "11"` in its manifest, and at the time the command is run, the latest 11.x version of Node is 11.4.0. The CLI tool is installed in the user toolchain with Node 11.4.0 set as its default engine. From this point on, unless the user changes their toolchain, running ``` @@ -67,18 +67,18 @@ A user can install this tool to their toolchain: ``` notion install project-explorer ``` -Assuming the `project-explorer` manifest selects Node 8.12.0 in its manifest, the tool is installed pinned to that version of the Node runtime. +In this user story, the `project-explorer` manifest is published with `"engines": "8"` in its manifest. At the time the tool is installed, the latest 8.x version of Node is 8.15.0. So the `pexx` tool is installed with Node 8.15.0 as its default engine. When the user runs ``` pexx myproject ``` -the `pexx` binary runs with Node 8.12.0---even if `myproject` specifies a different version of Node in its `"platform"` spec. +the `pexx` binary runs with Node 8.15.0—even if `myproject` specifies a different version of Node in its `"toolchain"` spec. # Pedagogy [pedagogy]: #pedagogy -This section lists the set of concepts that users may encounter using Notion. The first two, **tools** and the **toolchain**, are central to using Notion. The latter two, **shims** and **platforms**, are lower-level primitives that may be helpful for more implementation-oriented users. +This section lists the set of concepts that users may encounter using Notion. The first two, **tools** and the **toolchain**, are central to using Notion. The latter two, **shims** and **engines**, are lower-level primitives that may be helpful for more implementation-oriented users. ## Tools @@ -90,9 +90,9 @@ There are three types of tools: ## Toolchain -The toolchain is a set of tools. The user adds to their toolchain with `notion install` and removes with `notion uninstall`. +The toolchain is a set of tools the user has installed for use at the command-line console. The user adds to their toolchain with `notion install` and removes with `notion uninstall`. -Each tool installed in the toolchain has a default version and platform, which can be overridden when the tool is invoked in a project that has a dependency on that tool. +Each tool installed in the toolchain has a default version and engine, which can be overridden when the tool is invoked in a project that has a dependency on that tool. ## Primitive: Shims @@ -100,50 +100,60 @@ Notion shims intercept calls to tools and redirect execution to the right execut Users may already be familiar with shims, since these are commonly used by other version managers. -## Primitive: Platforms +## Primitive: Engines -A **platform spec** is a complete description of a version of the Node platform: +An **engine spec** is a complete description of a version of the Node platform: - An exact version of the Node runtime. - An optional exact version of npm. - An optional exact version of Yarn. -A **platform image** is an immutable instantiation of a platform spec on disk. It can be thought of analogously to a container image, but for the Node platform as opposed to an operating system. +An **engine image**, or **engine**, is an immutable instantiation of an engine spec on disk. (The name "image" is meant as an analogy to a container image, but for a snapshot of the Node platform as opposed to a snapshot of an operating system.) + +Users may already be familiar with the "engine" concept because of the `"engines"` key of `package.json`. # Details [details]: #details -## Toolchain +## Engine specs + +In the manifest, an omitted npm version defaults to the version bundled with the specified Node runtime. An omitted Yarn version defaults to Yarn being unavailable, meaning that invoking `yarn` will produce an error. Both of these can be explicitly set to `null`, which means the specified tool is unavailable, i.e., the shim fails with an error when executed. -A toolchain is a pair of a platform image and a set of pinned package binaries. +## Engine images -## Platform specs +An engine image is an installation on disk of a specific version of the Node runtime, a specific version of npm (or none), and a specific version of Yarn (or none). -In the manifest, an omitted npm version defaults to the version bundled with the specified Node runtime. An omitted Yarn version defaults to Yarn being unavailable, meaning that invoking `yarn` will produce an error. Both of these can be explicitly set to `null`, which means the specified tool is unavailable. +The _intention_ of Notion is for an image never to be modified. In particular, all commands that modify the state of an engine—e.g. `npm install --global` or `yarn global add`—should fail with an error when invoked through Notion shims. This behavior can be disabled with the environment variable `NOTION_UNSAFE_GLOBAL`. The name is meant to indicate to the user that they are "voiding their warranty" and responsible for any violations of the expectation of immutability. -## Platform images +## Pinning a project -The definition of a platform image is a pair consisting of a Node version and a Yarn version. +The `"toolchain"` section of `package.json` selects the engine image associated with a package. -A Node version is a pair consisting of an exact Node runtime version, and either an exact npm version or `null`. +Users can pin the engine by manually editing the `package.json` `"toolchain"` section or via [`notion pin`](https://github.com/notion-cli/rfcs/pull/24). -A Yarn version is either an exact Yarn version or `null`. +When the `node` shim or a package manager shim is executed from within a pinned project, the shim delegates to the version of that tool from the project's pinned engine. -## Pinning +## Default engine -The `"platform"` section of `package.json` selects the platform image associated with a package. (**Note:** This is a change from earlier versions of Notion, which used the key `"toolchain"`, since the toolchain consists of _both_ the platform image _and_ the pinned package binaries -- see above.) +Every package binary in the toolchain has a **default engine** associated with it. -Users can pin the platform by manually editing the `package.json` `"platform"` section or via [`notion pin`](https://github.com/notion-cli/rfcs/pull/24). +When a package binary is executed outside of a Node project, or from a Node project that _does not_ have that package as a direct dependency, the package binary is run using its default engine. + +When a package binary is executed from a Node project that _does_ have the package as a direct dependency but _does not_ have a pinned engine, the user's installed engine is used. If the user does not have an installed engine, the shim fails with an error indicating that no engine was selected. ## Installation The `notion install` command installs a tool to the user's toolchain. -When installing a version of the Node runtime, Notion also installs the default version of npm bundled with that verison of Node. This can be overridden with a subsequent `notion install npm` command. +### Installing an engine + +The `notion install node` subcommand installs the user's engine. When installing a version of the Node runtime, Notion also installs the default version of npm bundled with that verison of Node. This can be overridden with a subsequent `notion install npm` command. + +### Installing a package binary -When installing a package binary to the user toolchain, Notion checks the package for a `"platform"` key to pin the user tool to a specific platform image. If there is no `"platform"` key in the package manifest, it defaults to the current platform image configured in the user toolchain. If there is no current platform image, `notion install` fails with an error message suggesting the user choose at least a Node runtime version. +When installing a package binary, Notion checks the package for the standard [`"engines"`](https://docs.npmjs.com/files/package.json#engines) key to select the latest engine version compatible with the package and pins the tool's default engine to that engine version. If there is no `"engines"` key in the package manifest, it defaults to the user's current engine. If the user has no current engine, `notion install` fails with an error message suggesting the user choose at least a Node runtime version. -### Overriding the associated platform +### Overriding the associated engine When used for a package tool, `notion install` accepts an optional `--node` parameter for overriding the package's specified platform version: @@ -167,9 +177,9 @@ notion update surge # Critique [critique]: #critique -It's natural to question whether pinning for reproducibility is worth the cost of extra fetching. An alternative approach would be to allow projects to specify less precise version requirements (such as the ranges typically expressed in the `"engines"` field of `package.json`) and assume most differences will be benign. However, behavioral divergences between versions of Node do happen and are tricky bugs to nail down. Putting in extra work up front to ensure that these divergences cannot happen, by construction should pay dividends when scaled across the Node ecosystem. And over time, we can investigate optimization techniques to save time and disk space for fetching multiple similar versions. +It's natural to question whether pinning for reproducibility is worth the cost of extra fetching. An alternative approach would be to allow projects to specify less precise version requirements (such as the ranges expressed under the `"engines"` field of `package.json`) and assume most differences will be benign. However, behavioral divergences between versions of Node do happen and are tricky bugs to nail down. Putting in extra work up front to ensure that these divergences cannot happen, by construction should pay dividends when scaled across the Node ecosystem. And over time, we can investigate optimization techniques to save time and disk space for fetching multiple similar versions. -Theoretically, it might make more sense to put the `"platform"` section in a lockfile. But since there isn't a standardized single lockfile format for JS, and those formats aren't extensible, and we don't want to impose a whole new file to add to JS projects, using the package manifest seemed like the least imposition on users. +Theoretically, it might make more sense to put the `"toolchain"` section in a lockfile. But since there isn't a standardized single lockfile format for JS, and those formats aren't extensible, and we don't want to impose a whole new file to add to JS projects, using the package manifest seemed like the least imposition on users. Another reasonable criticism is that pinning the Node version for tools in the user toolchain means that users will not automatically benefit from performance and security improvements in Node. There are a couple of reasons this is outweighed by the benefits of pinning. First, as described above, updating tools will typically get platform updates. Second, users can still override the default with the `--node` parameter. @@ -180,3 +190,4 @@ Another reasonable criticism is that pinning the Node version for tools in the u - Should we consider some kind of update notification mechanism to inform you when your user tools are out of date? Probably at least something similar to `brew outdated` so users can explicitly ask for the information. - What kinds of performance optimizations can we offer to make stateless platform images super fast? - What about scenarios like CI and testing matrices, where users want to be able to specify different platform configurations without having to change the configuration file? Perhaps workflows similar to `ember try`? +- `update` vs `upgrade` syntax 😬 From 839eeac859de828374678c29fe747e369a8b1d01 Mon Sep 17 00:00:00 2001 From: David Herman Date: Wed, 29 May 2019 11:18:52 -0700 Subject: [PATCH 5/7] Notion -> Volta --- text/0000-toolchain.md | 46 +++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/text/0000-toolchain.md b/text/0000-toolchain.md index f2f71f8..5a9892c 100644 --- a/text/0000-toolchain.md +++ b/text/0000-toolchain.md @@ -1,22 +1,22 @@ - Feature Name: toolchain - Start Date: 2018-12-07 - RFC PR: -- Notion Issue: +- Volta Issue: # Summary [summary]: #summary -This RFC describes a design for the Notion's central unifying design concept: the **toolchain**. +This RFC describes a design for the Volta's central unifying design concept: the **toolchain**. # Motivation [motivation]: #motivation -The Notion toolchain is designed with several goals: +The Volta toolchain is designed with several goals: -- **Reproducible project tooling.** With this design, Notion should make it convenient and reliable for all contributors on a project to get the same exact version of the Node runtime, the package manager, and any package binaries configured for use by `package.json`. +- **Reproducible project tooling.** With this design, Volta should make it convenient and reliable for all contributors on a project to get the same exact version of the Node runtime, the package manager, and any package binaries configured for use by `package.json`. - **Reconciling the use of npm as a software distribution platform with best practices.** It's considered a best practice for project development to avoid ever installing global packages. But npm is a popular and convenient platform for distributing command-line tools. The toolchain design reconciles this tension by isolating _user tools_, which are installed for personal use, making them invisible to JS project scripts, and distinguishing them from _project tools_. -- **Install and forget.** This design solves the problem of _tool bitrot_, where a tool stops working because of Node upgrades. It also avoids the need to reinstall global tools every time a new Node version is provisioned. With Notion, once you install a user tool and get it working, it keeps working unless you deliberately decide to change or uninstall it. -- **Low cognitive overhead via statelessness.** This design allows Notion users to generally avoid thinking at all about the version of Node that's currently installed, instead relying on saved configuration to ensure that tools and projects have already declaratively specified their Node platform version. +- **Install and forget.** This design solves the problem of _tool bitrot_, where a tool stops working because of Node upgrades. It also avoids the need to reinstall global tools every time a new Node version is provisioned. With Volta, once you install a user tool and get it working, it keeps working unless you deliberately decide to change or uninstall it. +- **Low cognitive overhead via statelessness.** This design allows Volta users to generally avoid thinking at all about the version of Node that's currently installed, instead relying on saved configuration to ensure that tools and projects have already declaratively specified their Node platform version. ## User stories @@ -41,7 +41,7 @@ and the lockfile pins TypeScript to version 3.2.2. Users can call the TypeScript compiler directly as long as they've installed it: ``` -notion install typescript +volta install typescript ``` Afterwards, running `tsc` from within the project directory runs version 3.2.2 of the TypeScript compiler, using Node 10.10.0 as the runtime. @@ -49,7 +49,7 @@ Afterwards, running `tsc` from within the project directory runs version 3.2.2 o An end user of the [surge.sh](https://surge.sh) service installs their CLI tool that is deployed as an npm package: ``` -notion install surge +volta install surge ``` In this user story, the `surge` tool is published with `"engines": "11"` in its manifest, and at the time the command is run, the latest 11.x version of Node is 11.4.0. The CLI tool is installed in the user toolchain with Node 11.4.0 set as its default engine. @@ -65,7 +65,7 @@ The [project-explorer](https://sdras.github.io/project-explorer-site/) tool is a A user can install this tool to their toolchain: ``` -notion install project-explorer +volta install project-explorer ``` In this user story, the `project-explorer` manifest is published with `"engines": "8"` in its manifest. At the time the tool is installed, the latest 8.x version of Node is 8.15.0. So the `pexx` tool is installed with Node 8.15.0 as its default engine. @@ -78,7 +78,7 @@ the `pexx` binary runs with Node 8.15.0—even if `myproject` specifies a differ # Pedagogy [pedagogy]: #pedagogy -This section lists the set of concepts that users may encounter using Notion. The first two, **tools** and the **toolchain**, are central to using Notion. The latter two, **shims** and **engines**, are lower-level primitives that may be helpful for more implementation-oriented users. +This section lists the set of concepts that users may encounter using Volta. The first two, **tools** and the **toolchain**, are central to using Volta. The latter two, **shims** and **engines**, are lower-level primitives that may be helpful for more implementation-oriented users. ## Tools @@ -90,13 +90,13 @@ There are three types of tools: ## Toolchain -The toolchain is a set of tools the user has installed for use at the command-line console. The user adds to their toolchain with `notion install` and removes with `notion uninstall`. +The toolchain is a set of tools the user has installed for use at the command-line console. The user adds to their toolchain with `volta install` and removes with `volta uninstall`. Each tool installed in the toolchain has a default version and engine, which can be overridden when the tool is invoked in a project that has a dependency on that tool. ## Primitive: Shims -Notion shims intercept calls to tools and redirect execution to the right executable based on the environment and current directory. +Volta shims intercept calls to tools and redirect execution to the right executable based on the environment and current directory. Users may already be familiar with shims, since these are commonly used by other version managers. @@ -123,13 +123,13 @@ In the manifest, an omitted npm version defaults to the version bundled with the An engine image is an installation on disk of a specific version of the Node runtime, a specific version of npm (or none), and a specific version of Yarn (or none). -The _intention_ of Notion is for an image never to be modified. In particular, all commands that modify the state of an engine—e.g. `npm install --global` or `yarn global add`—should fail with an error when invoked through Notion shims. This behavior can be disabled with the environment variable `NOTION_UNSAFE_GLOBAL`. The name is meant to indicate to the user that they are "voiding their warranty" and responsible for any violations of the expectation of immutability. +The _intention_ of Volta is for an image never to be modified. In particular, all commands that modify the state of an engine—e.g. `npm install --global` or `yarn global add`—should fail with an error when invoked through Volta shims. This behavior can be disabled with the environment variable `VOLTA_UNSAFE_GLOBAL`. The name is meant to indicate to the user that they are "voiding their warranty" and responsible for any violations of the expectation of immutability. ## Pinning a project The `"toolchain"` section of `package.json` selects the engine image associated with a package. -Users can pin the engine by manually editing the `package.json` `"toolchain"` section or via [`notion pin`](https://github.com/notion-cli/rfcs/pull/24). +Users can pin the engine by manually editing the `package.json` `"toolchain"` section or via [`volta pin`](https://github.com/volta-cli/rfcs/pull/24). When the `node` shim or a package manager shim is executed from within a pinned project, the shim delegates to the version of that tool from the project's pinned engine. @@ -143,35 +143,35 @@ When a package binary is executed from a Node project that _does_ have the packa ## Installation -The `notion install` command installs a tool to the user's toolchain. +The `volta install` command installs a tool to the user's toolchain. ### Installing an engine -The `notion install node` subcommand installs the user's engine. When installing a version of the Node runtime, Notion also installs the default version of npm bundled with that verison of Node. This can be overridden with a subsequent `notion install npm` command. +The `volta install node` subcommand installs the user's engine. When installing a version of the Node runtime, Volta also installs the default version of npm bundled with that verison of Node. This can be overridden with a subsequent `volta install npm` command. ### Installing a package binary -When installing a package binary, Notion checks the package for the standard [`"engines"`](https://docs.npmjs.com/files/package.json#engines) key to select the latest engine version compatible with the package and pins the tool's default engine to that engine version. If there is no `"engines"` key in the package manifest, it defaults to the user's current engine. If the user has no current engine, `notion install` fails with an error message suggesting the user choose at least a Node runtime version. +When installing a package binary, Volta checks the package for the standard [`"engines"`](https://docs.npmjs.com/files/package.json#engines) key to select the latest engine version compatible with the package and pins the tool's default engine to that engine version. If there is no `"engines"` key in the package manifest, it defaults to the user's current engine. If the user has no current engine, `volta install` fails with an error message suggesting the user choose at least a Node runtime version. ### Overriding the associated engine -When used for a package tool, `notion install` accepts an optional `--node` parameter for overriding the package's specified platform version: +When used for a package tool, `volta install` accepts an optional `--node` parameter for overriding the package's specified platform version: ``` -notion install surge --node=latest +volta install surge --node=latest ``` ## Uninstallation -The `notion uninstall` command uninstalls a tool from the user's toolchain. +The `volta uninstall` command uninstalls a tool from the user's toolchain. ## Updating -Taken together, `notion uninstall` and `notion install` can be used to update a tool. Note that as the tool evolves over time, its authors may update its platform to newer Node versions. So as users upgrade their tools they will automatically get platform upgrades for the tool. But they also continue to be assured they are getting a version of the platform that the tool was tested with. +Taken together, `volta uninstall` and `volta install` can be used to update a tool. Note that as the tool evolves over time, its authors may update its platform to newer Node versions. So as users upgrade their tools they will automatically get platform upgrades for the tool. But they also continue to be assured they are getting a version of the platform that the tool was tested with. -The `notion update` command is a shorthand command for doing the same thing: +The `volta update` command is a shorthand command for doing the same thing: ``` -notion update surge +volta update surge ``` # Critique From 60275a293491a3140985856f263a203f64dcb76e Mon Sep 17 00:00:00 2001 From: David Herman Date: Wed, 29 May 2019 11:19:37 -0700 Subject: [PATCH 6/7] `"toolchain"` -> `"volta"` --- text/0000-toolchain.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0000-toolchain.md b/text/0000-toolchain.md index 5a9892c..934c74c 100644 --- a/text/0000-toolchain.md +++ b/text/0000-toolchain.md @@ -31,7 +31,7 @@ A project maintainer selects a version of TypeScript in `package.json`: "dependencies": { "typescript": "^3.2" }, - "toolchain": { + "volta": { "node": "10.10.0" } ... @@ -73,7 +73,7 @@ When the user runs ``` pexx myproject ``` -the `pexx` binary runs with Node 8.15.0—even if `myproject` specifies a different version of Node in its `"toolchain"` spec. +the `pexx` binary runs with Node 8.15.0—even if `myproject` specifies a different version of Node in its `"volta"` spec. # Pedagogy [pedagogy]: #pedagogy @@ -127,9 +127,9 @@ The _intention_ of Volta is for an image never to be modified. In particular, al ## Pinning a project -The `"toolchain"` section of `package.json` selects the engine image associated with a package. +The `"volta"` section of `package.json` selects the engine image associated with a package. -Users can pin the engine by manually editing the `package.json` `"toolchain"` section or via [`volta pin`](https://github.com/volta-cli/rfcs/pull/24). +Users can pin the engine by manually editing the `package.json` `"volta"` section or via [`volta pin`](https://github.com/volta-cli/rfcs/pull/24). When the `node` shim or a package manager shim is executed from within a pinned project, the shim delegates to the version of that tool from the project's pinned engine. @@ -179,7 +179,7 @@ volta update surge It's natural to question whether pinning for reproducibility is worth the cost of extra fetching. An alternative approach would be to allow projects to specify less precise version requirements (such as the ranges expressed under the `"engines"` field of `package.json`) and assume most differences will be benign. However, behavioral divergences between versions of Node do happen and are tricky bugs to nail down. Putting in extra work up front to ensure that these divergences cannot happen, by construction should pay dividends when scaled across the Node ecosystem. And over time, we can investigate optimization techniques to save time and disk space for fetching multiple similar versions. -Theoretically, it might make more sense to put the `"toolchain"` section in a lockfile. But since there isn't a standardized single lockfile format for JS, and those formats aren't extensible, and we don't want to impose a whole new file to add to JS projects, using the package manifest seemed like the least imposition on users. +Theoretically, it might make more sense to put the `"volta"` section in a lockfile. But since there isn't a standardized single lockfile format for JS, and those formats aren't extensible, and we don't want to impose a whole new file to add to JS projects, using the package manifest seemed like the least imposition on users. Another reasonable criticism is that pinning the Node version for tools in the user toolchain means that users will not automatically benefit from performance and security improvements in Node. There are a couple of reasons this is outweighed by the benefits of pinning. First, as described above, updating tools will typically get platform updates. Second, users can still override the default with the `--node` parameter. From fdcda7379ff6cb9677674d9cc20dd6fe98022229 Mon Sep 17 00:00:00 2001 From: David Herman Date: Wed, 29 May 2019 11:23:54 -0700 Subject: [PATCH 7/7] Add open question about Docker layer caching --- text/0000-toolchain.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-toolchain.md b/text/0000-toolchain.md index 934c74c..4c77df4 100644 --- a/text/0000-toolchain.md +++ b/text/0000-toolchain.md @@ -191,3 +191,4 @@ Another reasonable criticism is that pinning the Node version for tools in the u - What kinds of performance optimizations can we offer to make stateless platform images super fast? - What about scenarios like CI and testing matrices, where users want to be able to specify different platform configurations without having to change the configuration file? Perhaps workflows similar to `ember try`? - `update` vs `upgrade` syntax 😬 +- Optional lockfile-like mechanism for [integration into other tooling such as Docker](https://github.com/volta-cli/volta/issues/282)?