WIP: Yarn v4 ADR

Signed-off-by: Bruno Pimentel <[email protected]>

docs/adr/0002-yarn-v4.md

@@ -0,0 +1,168 @@
+# Extend Yarn Berry support to v4
+
+## Context
+
+Yarn v4 was released in October 23, 2023, which coincided with the time that the support for Yarn v3 was being introduced in Cachi2. To limit the scope of implementation, we decided to leave support for v4 to a later time. Now, as more projects migrate to v4, introducing proper support has become a priority.
+
+This document analyzes the the most impactful changes introduced with v4, and how they relate to the current v3 implementation.
+
+Main references:
+- [v4 blog post](https://yarnpkg.com/blog/release/4.0)
+- [Breaking changes for Yarn 4](https://github.com/yarnpkg/berry/issues/3591)
+
+### Global cache is enabled by default
+
+The [enableGlobalCache](https://yarnpkg.com/configuration/yarnrc#enableGlobalCache) option allows the user to set a shared location for the cache folder. Cachi2 already treats this option the following way:
+
+- Set to `true` during the prefetch, and point it to a specific location in order to keep the prefetched dependencies.
+- Set to `false` during the build, since the build needs to read from the cache, but the dependencies need to be installed to a local folder since they'll be used during the runtime.
+
+The only action that might be necessary is to document this behavior, since now during build-time, Cachi2 is setting the opposite option that would be expected as default.
+
+### All official plugins are enabled by default
+
+And can no longer be disabled by `.yarnrc.yml` configuration. The current official plugins don't seem to introduce any behavior that would taint the accuracy of the prefetched dependencies, though. Most of them only add support to protocols (which will still be filtered using the same rules of Yarn v4) or CLI commands.
+
+The list of all official plugins can be found under the "Default Plugins" section in the [API](https://yarnpkg.com/api) page on the official Yarn documentation. Here's a short summary of every official plugin (as of 4.5.3):
+
+<details>
+    <summary>Plugins that add support for a protocol</summary>
+
+- plugin-exec
+- plugin-file 
+- plugin-git
+- plugin-http
+- plugin-link
+- plugin-npm
+- plugin-patch
+</details>
+
+<details>
+    <summary>Plugins that enable a CLI command</summary>
+
+- plugin-essentials
+- plugin-init
+- plugin-interactive-tools
+- plugin-npm-cli
+- plugin-pack
+- plugin-stage
+- plugin-workspace-tools
+- plugin-version
+</details>
+
+<details>
+    <summary>Other plugins</summary>
+
+- plugin-compat: patches packages that aren't compatible with Plug'n'Play
+- plugin-constraints: support for [constraints](https://yarnpkg.com/features/constraints)
+- plugin-dlx: install a package in temporary environment
+- plugin-github: improves the performance when cloning from Github
+- plugin-nm: support for installing packages in `node_modules`
+- plugin-pnp: support for [Plug'n'Play](https://yarnpkg.com/features/pnp)
+- plugin-pnpm: support for installing packages using symlinks
+- plugin-typescript: Automatically adds `@types/` packages into your dependencies
+</details>
+<br>
+
+We should still be able to proceed only by disabling non-official plugins without introducing any arbitrary code execution, and keep the current behavior for v3 projects unchanged.
+
+#### A note about plugin-typescript
+
+The [typescript](https://yarnpkg.com/api/plugin-typescript) plugin is used to automatically include types when adding a dependency that does not package them by default. Since these changes are reflected in the `package.json` file, Cachi2 will handle them normally.
+
+<details>
+    <summary>Example of how the Typescript plugin works</summary>
+
+```
+$ yarn add lodash 
+➤ YN0000: · Yarn 4.5.3
+➤ YN0000: ┌ Resolution step
+➤ YN0085: │ + @types/lodash@npm:4.17.13, lodash@npm:4.17.21
+➤ YN0000: └ Completed
+➤ YN0000: ┌ Fetch step
+➤ YN0013: │ A package was added to the project (+ 957.26 KiB).
+➤ YN0000: └ Completed in 0s 252ms
+➤ YN0000: ┌ Link step
+➤ YN0000: └ Completed
+➤ YN0000: · Done in 0s 313ms
+
+$ cat package.json 
+{
+  "name": "yarn-types",
+  "packageManager": "[email protected]",
+  "dependencies": {
+    "lodash": "^4.17.21"
+  },
+  "devDependencies": {
+    "@types/lodash": "^4"
+  }
+}
+```
+</details>
+
+### pnpDataPath is no longer configurable
+
+This used to be a configurable key in the `.yarnrc.yml` file. The previous default path, `./.pnp.data.json`, is now hard-coded and can't be changed.
+
+The only mention to `pnpDataPath` in Cachi2 is the [check](https://github.com/containerbuildsystem/cachi2/blob/a5f19c6f9be90be4289beee35ecccd2827bbb328/cachi2/core/package_managers/yarn/main.py#L43) for paths pointing outside of the repo. The check can be kept the same way so Yarn v3 can still be covered, and it will simply be skipped in v4.
+
+### Yarn now caches npm version metadata
+
+The default path for this newly downloaded metadata is `{globalFolder}/metadata/npm`. From the early tests, it doesn't seem that Cachi2 requests are downloading this additional metadata. Further investigation is needed. [TODO]
+
+### Changes in yarn.lock
+
+When updating v3 projects to v4, some differences in the lockfile appear. These are some picks from updating the [disallowed-protocols]() branch on the cachi2-yarn-berry test project:
+
+**`npm:` locator added to npm dependencies:**
+```
+   dependencies:
+-    chownr: ^2.0.0
+-    fs-minipass: ^2.0.0
+-    minipass: ^5.0.0
+-    minizlib: ^2.1.1
+-    mkdirp: ^1.0.3
+-    yallist: ^4.0.0
+-  checksum: ...
++    chownr: "npm:^2.0.0"
++    fs-minipass: "npm:^2.0.0"
++    minipass: "npm:^5.0.0"
++    minizlib: "npm:^2.1.1"
++    mkdirp: "npm:^1.0.3"
++    yallist: "npm:^4.0.0"
++  checksum: ...
+```
+
+**Changes to some instances of the file locator:**
+```
+-  resolution: "strip-ansi-tarball@file:external-packages/strip-ansi-4.0.0.tgz::locator=berryscary%40workspace%3A."
++  resolution: "strip-ansi-tarball@file:external-packages/strip-ansi-4.0.0.tgz#external-packages/strip-ansi-4.0.0.tgz::hash=e17689&locator=berryscary%40workspace%3A."
+```
+
+Yarn v4 introduced the subdirectory (`#external-packages/strip-ansi-4.0.0.tgz`) and the hash (`::hash=e17689`), which are already [handled](https://github.com/containerbuildsystem/cachi2/blob/a5f19c6f9be90be4289beee35ecccd2827bbb328/cachi2/core/package_managers/yarn/locators.py#L77-L79) by the v3 implementation in Cachi2.
+
+**Changes to some instances of the patch locator:**
+```
+-  resolution: "typescript@patch:typescript@npm%3A5.1.6#~builtin<compat/typescript>::version=5.1.6&hash=5da071"
++  resolution: "typescript@patch:typescript@npm%3A5.1.6#optional!builtin<compat/typescript>::version=5.1.6&hash=5da071"
+```
+
+Instances of the patch locator are currently [being ignored](https://github.com/containerbuildsystem/cachi2/blob/a5f19c6f9be90be4289beee35ecccd2827bbb328/cachi2/core/package_managers/yarn/resolver.py#L264-L268) in the v3 implementation in Cachi2.
+
+### Changes to .yarnrc.yml options
+
+TODO
+
+### Hardened mode
+
+TODO (enabled by default on clones from Github?)
+
+## Decision
+
+The changes needed to support Yarn v4 boil down to raising the maximum supported version, adding a few integration tests to cover v4 scenarios and documentation updates. The Corepack shim is compatible with v4, so the project will still be processed with the exact Yarn version that is defined in its configuration files.
+
+Enabling the newly introduced hardened mode during the prefetch might prove useful to further increase the security of the process, but is by no means necessary to introduce basic support for v4. The impact on the prefetching speed is something that needs to be investigated in depth before we decide to enable it, but the decision can be deferred to a later moment.
+
+## Consequences
+
+Support for Yarn v4 can be made available as soon as this ADR is merged, and no impacts for v3 users are expected.