Oclif can't find plugins when using yarn pnp/yarn 4

[danieltroger]

Describe the bug
I couldn't migrate my shopify app to yarn 3/4 due to shopify using oclif which can't resolve packages correctly when using yarn pnp.

The issue is that @shopify/app doesn't have . in it's package.json#exports field and therefore require.resolve can't be used to find the package path (oclif tries to find @shopify/app's package.json). And oclif's alternative resolution methods fail due to yarn pnp's way of loading packages (I presume).

A way of fixing it is to use the pnpapi when available to resolve the package location like my patch provided below.

To Reproduce

I already spent too much time on this so haven't tested the reproduction steps, LMK if you need more info here.

Steps to reproduce the behavior:

1. Create shopify app npm init @shopify/app@latest
2. Migrate everything to yarn 3 with workspaces
   • Delete package-lock.json
   • Delete node_modules
   • Run touch yarn.lock
   • Run yarn set version canary
   • Create .yarnrc.yml with nodeLinker: "pnp" inside to force yarn to use pnp (I need pnp because yarn glitches with my link setup without it)
   • Set up yarn workspaces according to https://classic.yarnpkg.com/lang/en/docs/workspaces/#toc-how-to-use-it (add workspaces into root package.json where you reference the web and web/frontend folders)
   • Run yarn install
3. Run shopify app dev
4. You should get command app:dev not found, this is due to @shopify/cli not being able to load @shopify/app as oclif plugin due to the resolving issue detailed above

Expected behavior
oclif correctly resolves the plugins -> Shopify devs my app as usual

Screenshots
If applicable, add screenshots to help explain your problem.

Plugins get requested:

The plugins failed to set up:

require.resolve failing

My patch that works (probably very hacky)

Something like this is what you'd need to change. I'm applying this with yarn patch and now things work for me :) The biggest issue with the patch in its current state, if you were to merge it:

1. it probably crashes oclif for everyone not using yarn pnp
2. If there are multiple versions of the plugin that's trying to be found, the wrong one might be selected (root is ignored)

diff --git a/lib/config/plugin.js b/lib/config/plugin.js
index f510dcaa0926ac0ae0d408a9ec12a599baaf52f0..aa1ef95f2f3369de47512d78c7f3b9a789b2c4eb 100644
--- a/lib/config/plugin.js
+++ b/lib/config/plugin.js
@@ -1,6 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.Plugin = void 0;
+const pnp = require(`pnpapi`);
 const errors_1 = require("../errors");
 const globby = require("globby");
 const path = require("path");
@@ -78,6 +79,14 @@ async function findRootLegacy(name, root) {
 }
 async function findRoot(name, root) {
     if (name) {
+        try {
+            const found = findPackageWithYarn(name);
+            if(found) {
+                return findSourcesRoot(found);
+            }
+        }catch(e){
+            console.log("patch to find with yarn failed", {name, root}, e);
+        }
         let pkgPath;
         try {
             pkgPath = (0, util_3.resolvePackage)(name, { paths: [root] });
@@ -87,6 +96,58 @@ async function findRoot(name, root) {
     }
     return findSourcesRoot(root);
 }
+function findPackageWithYarn(packageNameToFind){ // see https://yarnpkg.com/advanced/pnpapi#traversing-the-dependency-tree
+    const seen = new Set();
+
+    const getKey = locator =>
+        JSON.stringify(locator);
+
+    const isPeerDependency = (pkg, parentPkg, name) =>
+        getKey(pkg.packageDependencies.get(name)) === getKey(parentPkg.packageDependencies.get(name));
+
+    const traverseDependencyTree = (locator, parentPkg = null) => {
+        // Prevent infinite recursion when A depends on B which depends on A
+        const key = getKey(locator);
+        if (seen.has(key))
+            return;
+
+        const pkg = pnp.getPackageInformation(locator);
+
+        if(locator.name === packageNameToFind){
+            return pkg.packageLocation
+        }
+
+
+        seen.add(key);
+
+
+        for (const [name, referencish] of pkg.packageDependencies) {
+            // Unmet peer dependencies
+            if (referencish === null)
+                continue;
+
+            // Avoid iterating on peer dependencies - very expensive
+            if (parentPkg !== null && isPeerDependency(pkg, parentPkg, name))
+                continue;
+
+            const childLocator = pnp.getLocator(name, referencish);
+            const foundSomething = traverseDependencyTree(childLocator, pkg);
+            if(foundSomething) return foundSomething;
+        }
+
+
+        // Important: This `delete` here causes the traversal to go over nodes even
+        // if they have already been traversed in another branch. If you don't need
+        // that, remove this line for a hefty speed increase.
+        seen.delete(key);
+    };
+
+// Iterate on each workspace
+    for (const locator of pnp.getDependencyTreeRoots()) {
+        const foundSomething = traverseDependencyTree(locator);
+        if(foundSomething) return foundSomething;
+    }
+}
 class Plugin {
     constructor(options) {
         this.options = options;

Environment (please complete the following information):

• OS & version: macOS 14.0 (23A344)
• Shell/terminal & version zsh 5.9 (x86_64-apple-darwin23.0)

Additional context
cc some random shopify cli maintainer that I saw responding to issues while solving this: @alvaro-shopify

related: #309 and oclif/config#289 (comment)

[mdonnalley]

@danieltroger Thanks for creating the issue and doing so much investigative work on it! Would it be possible to create a repository that has all the items you included in the Migrate everything to yarn 3 with workspaces step? That would save me a ton of time. Thanks!

[danieltroger]

@mdonnalley sorry for the late response. I tried following the reproduction steps that I wrote and ended up with a quite different setup than what we're running in production, but the oclif bug is still reproducible.

I set it up for yarn 4 but if you go down to yarn 3 the same issue happens.

To reproduce:

1. git clone [email protected]:danieltroger/oclif-repro.git
2. cd oclif-repro
3. yarn
4. yarn dev

You should see this output

Cloning into 'oclif-repro'...
remote: Enumerating objects: 58, done.
remote: Counting objects: 100% (58/58), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 58 (delta 5), reused 58 (delta 5), pack-reused 0
Receiving objects: 100% (58/58), 1.09 MiB | 7.93 MiB/s, done.
Resolving deltas: 100% (5/5), done.
daniel@mmmmmmmmmm test % cd oclif-repro 
daniel@mmmmmmmmmm oclif-repro % yarn
➤ YN0000: · Yarn 4.0.0-rc.53
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Post-resolution validation
➤ YN0002: │ oclif-bug-repro@workspace:. doesn't provide @shopify/shopify-api (p3f681), requested by @shopify/shopify-app-session-storage-prisma.
➤ YN0002: │ oclif-bug-repro@workspace:. doesn't provide @shopify/shopify-app-session-storage (pb0f91), requested by @shopify/shopify-app-session-storage-prisma.
➤ YN0086: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 0s 240ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ yarn@npm:1.22.19 must be built because it never has been before or the last one failed
➤ YN0007: │ @prisma/engines@npm:4.16.2 must be built because it never has been before or the last one failed
➤ YN0007: │ esbuild@npm:0.17.6 must be built because it never has been before or the last one failed
➤ YN0007: │ esbuild@npm:0.18.20 must be built because it never has been before or the last one failed
➤ YN0007: │ @shopify/plugin-cloudflare@npm:3.49.7 must be built because it never has been before or the last one failed
➤ YN0007: │ esbuild@npm:0.18.17 must be built because it never has been before or the last one failed
➤ YN0007: │ prisma@npm:4.16.2 must be built because it never has been before or the last one failed
➤ YN0007: │ @prisma/client@npm:4.16.2 [32e86] must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 4s 860ms
➤ YN0000: · Done with warnings in 5s 280ms
daniel@mmmmmmmmmm oclif-repro % yarn dev
╭─ error ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                              │
│  command app:dev not found                                                                                                                                                   │
│                                                                                                                                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

daniel@mmmmmmmmmm oclif-repro % 

[git2gus]

This issue has been linked to a new work item: W-14331357