[RRFC] npm run-series && npm run-parallel

[MylesBorins]

Motivation ("The Why")

Often an npm script is part of a more complicated pipeline, tools such as grunt + gulp have been created to help manage some of this pipeline. A repo may have multiple steps to a compilation such as clean, build various types of assets (html, css, js), etc. Currently npm itself does not have a mechanism to orchestrate more complex workflow.

Example

Most recently I utilized xargs to accomplish a similar task.

"build": "echo build:clean build:css build:js build:html build:img | xargs -n1 npm run"

if run-series were available this could be accomplished in a platform agnostic way via

"build": "npm run-series build:clean build:css build:js build:html build:img"

It could also be optimized to run parallel operations

"build": "npm run build:clean && npm run-parallel build:css build:js build:html build:img"

or potentially

"build": "npm run build:clean --then npm run-parallel build:css build:js build:html build:img"

How

Current Behaviour

Currently there is no way to accomplish this beyond using && or & which is can be platform specific

Desired Behaviour

First class support for running multiple scripts in parallel or series

References

• n/a

[rwjblue]

Currently there is no way to accomplish this beyond using && or & which is can be platform specific

@MylesBorins - Just to confirm, you are specifically talking about a built-in mechanism for this right? Is the proposal roughly that we should add a run-series command that works similarly to how npm-run-all (a somewhat popular package that provides some functionality in this space) works?

[ljharb]

for "run parallel", i haven't yet found a tool that properly handles nonzero exit codes, and also handles mixing (and prefixing) streaming stdout/stderr output. If npm can solve this that would be awesome.

[wesleytodd]

I think this is a real issue, and our current constraints have led to some great and some really awful things. If we make it more powerful we run the risk of encouraging some even more wild usage.

At work, we have a system for composing "commands" which is not make. Learning from this, it is hard to get right but is ultimately very useful. If we want to really solve this well, which I am not sure we need to do, then I think we should consider preexisting systems with features like work avoidance and DSLs for declaring the commands. Also, to share, we have a tool which uses the stmux package and is a pretty good DX for the parallel commands to solve the log-interleaving problem.

[MylesBorins]

FWIW I think that we could likely scope this solution to only run in series that is more or less sugar for npm run one && npm run two && npm run three && npm run four and significantly reduce duplication and noise in the package.json

[MylesBorins]

Removing Agenda label. A next step would be a making a more complete proposal specifically for series.

[Christian24]

As I said in the meeting I think there is also a use case I can see for parallel, although probably in a second RFC: In web development there are often watch tasks that run Typescript, Sass, webpack or whatever. It would be useful to have a way to run them in parallel through npm and if one dies, just kill the entire thing.

[ColtHands]

I can definitely find this useful, not even parallel, but running multiple scripts in series as an array:

"scripts": {
  "build": [ "webpack --verbose", "npm run test", "git add .", "echo 'all done'" ]
}

npm run build

I think run-parallel should be an addition to yet not implemented run-series

Will run-series be an easier implementation than run-parallel as npm can run one script and in theory it could run multiple?

[ljharb]

The array approach is a pretty straightforward way to implement "series", fwiw - I like it.

[wesleytodd]

Not that I like this line of thought, but to toss out another complicated option:

• Array: run in series
• Object: run in parallel where the names are headings for the grouped logs

You could (although I strongly believe we need to move this all outside of npm) even allow for nesting:

"scriptfoo": [{
  "build1": "thing",
  "build2": "otherthing"
}, "test"]

Again, I really dont like the complexity this would add to npm, but if someone were to work on a declarative and simple command runner, one might do it this way.

[muuvmuuv]

Uhh, I like this solution a lot @wesleytodd because it does not introduce commands/flags and is straight forward! Through a flag like --no-fail we could say that parallel or series tasks wouldn't exit on error.

Our scripts are looking like that with npm-run-all:

  "scripts": {
    "start": "ng serve",
    "prod": "ng serve --prod",
    "clean": "del dist tmp out-tsc coverage results abstracts .lighthouseci",
    "proxy": "make restart",
    "build": "run-s build:*",
    "build:ng": "ng build --prod",
    "build:min": "node scripts/minify-files.js",
    "docs": "typedoc --options .typedoc.js src",
    "docs-open": "open-cli docs/index.html",
    "all": "run-s clean lint build docs test-ci e2e-ci perf",
    "−−−−−−−−−−−−−−−---−−−---- TESTS −---−−−−−−---−−−−−−−−−−−": "",
    "test": "ng test",
    "test-ci": "ng test -c ci",
    "e2e": "ng e2e",
    "e2e-ci": "ng e2e -c ci",
    "−−−−−−−−−−−−−−−---−−−- PERFORMANCE −−−−−−−---−−−−−−−−−−−": "",
    "perf": "run-s build proxy perf-lhci perf-open",
    "perf-lhci": "lhci collect",
    "perf-open": "run-p perf-open:*",
    "perf-open:lhci": "lhci open",
    "−−−−−−−−−−−−−−−−−−------ LINTING −−−−−−−------−−−−−−−−−−": "",
    "lint": "run-p lint:*",
    "lint:tsc": "tsc --build tsconfig.app.json",
    "lint:tsc-worker": "tsc --build tsconfig.worker.json",
    "lint:source": "ng lint",
    "lint:cypress": "eslint --ext .ts,.js cypress",
    "lint:engine": "eslint --cache *.js scripts/**/*.js",
    "lint:scss": "stylelint --cache src/**/*.scss",
    "−-−−−−−−−−−−−−−−−-----  FORMATING  −−−−−−--−−−−−−−−−−-−−": "",
    "format": "run-p format:*",
    "format:typings": "prettier --write 'src/**/*.d.ts'",
    "−-−−−−−−−−−−−−−−−-−----−  TOOLS  −−−−−−----−−−−−−−−−−-−−": "",
    "preinstall": "node scripts/welcome.js && sh scripts/nvm-use.sh",
    "postinstall": "node scripts/setup.js",
    "generate-graphql": "graphql-codegen && npm run format:typings",
    "changelog": "conventional-changelog -p angular -i CHANGELOG.md -s -r 0",
    "release": "node scripts/release.js"
  },

This is how it could look like then (sub-tasks would be indented with there title):

{
  "scripts": {
    // "start": "ng serve",
    // "prod": "ng serve --prod",
    // "clean": "del dist tmp out-tsc coverage results abstracts .lighthouseci",
    // "proxy": "make restart",
    // "docs": "typedoc --options .typedoc.js src",
    // "docs-open": "open-cli docs/index.html",
    "all": [
      "clean",
      "lint",
      "build-docs",
      "test-ci",
      "e2e-ci",
      "perf"
    ],
    "−−−−−−−−−−−−−−−---−−−---- BUILD −---−−−−−−---−−−−−−−−−−−": "",
    "build": {
      "Build application": "build-ng",
      "Minify non-minified": "build-min"
    },
    "build-ng": "ng build --prod",
    "build-min": "node scripts/minify-files.js",
    // "−−−−−−−−−−−−−−−---−−−---- TESTS −---−−−−−−---−−−−−−−−−−−": "",
    // "test": "ng test",
    // "test-ci": "ng test -c ci",
    // "e2e": "ng e2e",
    // "e2e-ci": "ng e2e -c ci",
    "−−−−−−−−−−−−−−−---−−−- PERFORMANCE −−−−−−−---−−−−−−−−−−−": "",
    "perf": {
      "Building": "build",
      "Starting proxy": "proxy",
      "Running performance tests": "perf:*",
      "Opening reports": "perf-open"
    },
    "perf:lhci": "lhci collect",
    "perf-open": "perf-open:*",
    "perf-open:lhci": "lhci open",
    "−−−−−−−−−−−−−−−−−−------ LINTING −−−−−−−------−−−−−−−−−−": "",
    "lint": "lint:*",
    "lint:tsc": "tsc --build tsconfig.app.json",
    "lint:tsc-worker": "tsc --build tsconfig.worker.json",
    "lint:source": "ng lint",
    "lint:cypress": "eslint --ext .ts,.js cypress",
    "lint:engine": "eslint --cache *.js scripts/**/*.js",
    "lint:scss": "stylelint --cache src/**/*.scss",
    "−-−−−−−−−−−−−−−−−-----  FORMATING  −−−−−−--−−−−−−−−−−-−−": "",
    "format": "format:*",
    "format:typings": "prettier --write 'src/**/*.d.ts'",
    // "−-−−−−−−−−−−−−−−−-−----−  TOOLS  −−−−−−----−−−−−−−−−−-−−": "",
    // "preinstall": "node scripts/welcome.js && sh scripts/nvm-use.sh",
    // "postinstall": "node scripts/setup.js",
    // "generate-graphql": "graphql-codegen && npm run format:typings",
    // "changelog": "conventional-changelog -p angular -i CHANGELOG.md -s -r 0",
    // "release": "node scripts/release.js"
  }
}

* would just serve as a glob like npm-run-all handles it :)

[guybedford]

Perhaps the array should always be the names of other scripts in the script map? This way the individual tasks remain named scripts that can be run separately. In addition it removes the need to inline npm run:

{
  "scripts": {
    "build-all": ["npm run build1", "npm run build2"],
    "build1": "...",
    "build2": "..."
  }
}

and instead just write:

{
  "scripts": {
    "build-all": ["build1", "build2"],
    "build1": "...",
    "build2": "..."
  }
}

[guybedford]

It might be nice to reserve the sweeter array syntax for parallel runs, or even make the series v parallel thing a flag of npm run itself - npm run build-all -j8 :)

[Eomm]

This feature could be adapted to be used on workspaces too.

npm run-parallel lint --workspaces

executes in parallel the lint script for each project in the monorepo.

[dannyvv]

I would like to make an argument to allow the dependency graph to be 'statically' extracted.
Either by ensure the graph has a static declaration, or if that is not desirable, have an 'export' mode i.e.

• npm run build --export-dependency-graph
• npm run-export-graph build
  or something similar where the graph can be emitted cheaply without actually executing the tasks.
  My main motivation for this request is so that an external build scheduler can take this information and optimize the javascript build even futher.

At Microsoft we have very large JavaScript repositories as well as mixed repositories where nmake (C++), gradle (java), msbuild (c++ and C#) and javascript projects all live together in a single build graph.
Today it is already a struggle to use a single verb per project as the critical build path is too long.
We leverage tools like Lage to create finer grained jobs at the 'verb' level. Think scripts called build, bundle, lint, test, test:e2e where lage declares finer dependencies i.e. 'build' depends on the build verb of dependent projects. test depends on the 'build' verb, test:e2e depends on the bundle verb etc etc.
We already leverage a library to perform what this RFC promotes just-scripts that performs parallel or sequential subtasks in a single build verb as well as having support for flags to control the subgraph.
i.e. wether it should create bundles for all platforms (win32, ios, android, web) or just a single etc.

Lage and Just mentioned in the above paragraph support running locally on a single devbox.
We also have robust build engines like BuildXL that can take the graph emitted by Lage (as well as the graphs from other projects in hybrid case like nmake, msbuild, etc) and merge them into a giant graph of work to be executed.
The BuildXL build engine supports reliable caching, distributing the build accross multiple machines, handle safety policies (i.e. prevent one JS project from using ../.. to read files from other js projects). It has lots of smarts to make sure to use historical execution data to make sure the build machines are loaded as effeciently as possible. It has support for features like a reliable shared developer cache etc etc.

The finer grained the nodes that the build engine can operate on the better it can optimize itself.. Lage gives a graph with multiple verbs on a given project... But today it cannot peek into the commands of each verb. i.e. of 'build' is serial(parralel(codegen1, codegen2), typescriptcompiler) that is executed as a unit by the executed npm run build process in the build graph...
Having a way to cheaply statically get the information of what would happen inside the build verb would allow a finer grained build graph for the build engine and better machine utilization.

It would be amazing if this functionality could be built-in into npm! I really appreciate that this is being considered!

[justinfagnani]

I think it would be great to get scripts out of the business of directly running sub-steps in their command invocation and instead have them explicitly declare their dependencies, then have the script runner run sub-steps in parallel or serial as required.

So instead of:

"scripts": {
  "build": "npm run build:css && npm run build:js && npm run build:html && build:img"
  "build:css": ...

we might have scripts declared as objects with explicit dependencies:

"scripts": {
  "build": {
    "dependencies": [
      "build:css",
      "build:js",
      "build:html",
      "build:img",
    ]
  }
  "build:css": "...",
  "build:html": {
    "command": "some-html-build-tool",
    // tasks can depend on each other. The runner should figure out the correct order & parallelism
    "dependencies": [
      "build:css",
    ]
  }

[aomarks]

We built and published (today!) https://github.com/google/wireit, which is an attempt at bringing a declarative script-level dependency graph to npm, including parallel execution.

It uses a syntax just like what @justinfagnani showed in the comment above — except in a separate wireit section of the package.json. For example:

{
  "scripts": {
    "build": "wireit",
    "bundle": "wireit"
  },
  "wireit": {
    "build": {
      "command": "tsc",
      "files": ["src/**/*.ts", "tsconfig.json"],
      "output": ["lib"]
    },
    "bundle": {
      "command": "rollup -c",
      "dependencies": ["build"],
      "files": ["rollup.config.json"],
      "output": ["dist/bundle.js"]
    }
  }
}

(So this works by having the actual npm script delegate to the wireit binary, and the wireit binary figures out which script was invoked by reading the npm_lifecycle_event and npm_package_json environment variables.)

I concur that it would be awesome to have functionality like this built into npm. I'd be interested in starting/contributing to an RFC along these lines.

[JaneJeon]

This would be amazing. npm-run-all has not been maintained for several years and yet it is the only way to run a series of npm scripts in a controlled, cross-platform way

[Raffaello]

it would be a good start if myTask defined in all workspaces, could be run in parallel npm run myTask -ws with simply adding a --parallel option.

otherwise an hack and bash/PS7 hack must be done:

WORKSPACES=`jq  -r '.workspaces | @sh' package.json | tr -d \\'` && for w in ${WORKSPACES[@]};  do echo \"$w\" && pushd . && cd $w && npm run myTask; popd; done;

not really convinient...

[bhouston]

Is this going to happen? It seems like no one is implementing this.

This is massively useful for a number of incredibly common scenarios:

• A React frontend combined with a custom server.
• A few microservices that are all running locally on different ports. For example, I often have a webserver and then a backend worker handling long running tasks.

[billyzkid]

My wish list for npm scripts:

• Cross-platform operators for combining, grouping, piping, and redirecting both serial and parallel commands/scripts without having to configure a specific shell or add any dependencies
• Syntactic sugar for running npm scripts similar to what npm-run-all and concurrently have
• New options for npm run to support both serial and parallel execution of scripts and workspaces

For example:

{
  "scripts": {
    // cross-platform operators for commands
    "script1": "cmd -a & cmd -b",
    "script2": "cmd -c && cmd -d",
    "script3": "cmd -e || cmd -f",
    "script4": "{ cmd -a & cmd -b } || { cmd -c && cmd -d }",
    "script5": "cmd -a | cmd -b",
    "script6": "cmd -c 2>&1 outputfile",
    "script7": "cmd -d < inputfile",

    // cross-platform operators for scripts
    "script8": "npm run script1 & npm run script2",
    "script9": "{ npm run script1 & npm run script2 } || { npm run script3 && npm run script4 } | format -abc > logfile",

    // syntactic sugar and new options
    "script10": "npm run script:*",
    "script11": "npm run script1 script2 script3 --parallel",
    "script12": "npm run lint --workspaces --parallel",
    "script13": "npm run build --workspace=a --workspace=b --serial"
  }
}

Bonus points for being able to set environment variables in scripts cross-platform and support for json comments as shown above (if that's even possible).

[justinfagnani]

I think as soon as you add running scripts sequentially and concurrently, you quickly end up wanting DAG of scripts where you run common script dependencies only once. You relatively quickly then end up wanting to cache common script dependency results so that subsequent runs are faster.

This is what Wireit does, while maintaining npm as the user interface and package.json as the config file. I think it's a great approach, but short of building those features into npm itself, I think npm could also enable pluggable script runners that can add those features. I propose that in #691

[domharrington]

This feature could be adapted to be used on workspaces too.

npm run-parallel lint --workspaces

executes in parallel the lint script for each project in the monorepo.

@llimllib and I found a fairly clean way to do this using npm query, JSON parsing the result and xargs:

npm query .workspace | \
  node -p 'JSON.parse(fs.readFileSync(0)).map(ws => ws.name).join("\n")' | \
  xargs -I {} -P 0 npm run lint --if-present --workspace "{}"

Got the JSON parsing approach from this gist: https://gist.github.com/kristopherjohnson/5065599?permalink_comment_id=4775925#gistcomment-4775925

[bhouston]

BTW I just adopted nx / lerna and it works great and it also caches. Example project here: https://github.com/bhouston/template-typescript-monorepo