Running some tasks consecutively

[ErikLysne]

Goals

1. Allow some tasks to be run consecutively

Non-goals

No response

Background

I'm using Turborepo and pnpm in a monorepo with about 50 packages and a few microservices and frontend applications. I have generally been very happy with Turborepo and pnpm, but there is one kind of problem that I have not yet found a satisfactory solution to.

Let's say we have a monorepo with two packages: cli-tool and utils. The cli-tool package contains TypeScript code for a CLI tool located at ./src/bin/cli.ts relative to the package. It's package.json might look like this:

{
  "name": "cli-tool",
  "bin": {
    "cli-tool": "./dist/bin/cli.js"
  },
  "scripts": {
    "build": "tsup ./src/bin/cli.ts --format esm --outDir ./dist/bin"
  }
}

In other words, the package contains a bin, but importantly, this bin only becomes available AFTER the package has been built, as it is written in TypeScript and must be transpiled first.

The second package, utils, needs to run cli-tool as part of its build script. Its package.json might look like this:

{
  "name": "utils",
  "scripts": {
    "build": "pnpm cli-tool && tsc"
  },
  "devDependencies": {
    "cli-tool": "workspace:*"
  }
}

If we start from a fresh machine (no node_modules and nothing cached) and run:

pnpm install
pnpm turbo run build

the second build task will fail. This is because pnpm install will install all dependencies for cli-tool and utils, but as the bin referenced by cli-tool has not yet been built, it will not be put into node_modules/.bin for the utilspackage and hence not be available when utils#build runs.

Seemingly, the way to solve this is to re-run pnpm install after the first build script has run. For instance, we could modify the first package.json to:

"scripts": {
  "build:ts": "tsup ./src/bin/cli.ts --format esm --outDir ./dist/bin",
  "build:install": "pnpm install --filter=...cli-tool",
  "build": "pnpm run build:ts && pnpm run build:install"
}

Now, after building cli-tool, the script would rerun the package installation for all dependants on cli-tool. While this would work in the simple scenario here, it does not work when scaled up. If we have many packages and many tasks being run concurrently, running pnpm install randomly in the middle of a task tends to produce many errors of the kind no such file or directory, unlink ... or similar. I believe this is because pnpm install makes various modifications to the file system outside the scope of the package being built.

Please also note that it is not only by using the bin property you end up with this problem. Another example is using the dependenciesMeta property of pnpm with "injected": true.

Proposal

A workaround to the problem outlined above is to run everything consecutively, i.e., pnpm turbo run build --concurrency 1, but that takes a while in a large monorepo. A better solution would be to allow tasks to be marked as consecutive or isolated, meaning when turborepo encounters a consecutive task in the task graph, it waits for all running concurrent tasks to finish, runs the consecutive task in isolation, then resumes running concurrently.

In the turbo.json, this could be expressed as:

{
  "tasks":  {
    "build": {
      "dependsOn": ["^build", "^build:install"]
    },
    "build:install": {
      "dependsOn": ["build"],
      "consecutive": true // 👈🏻
    }
  }
}