[rush] add rush run to run script in all packages

[wbern]

Shortly put, Lerna has lerna run and lerna exec which have been convenient to run things across all packages.

I know Rush doesn't go the way via npm to execute npm scripts in each package, and I think that's great. I found myself wanting to run a simple script across all packages which I think would be nice if Rush could help me with.

[iclanton]

This functionality actually already exists in Rush via "bulk commands" in common/config/command-line.json. Take a look at the docs here, or at an example (but commented out) config here.

Bulk commands do have the overhead of having to be defined in command-line.json and in each package's package.json's scripts object, but we could explore adding the ability to run an ad-hoc command in each package (perhaps resolving to the global common/temp/node_modules/.bin directory in addition to whatever is on the PATH). I'm not sure if that should be something like rush run XYZ, as a rushx command, or as a different verb on the rush command.

Can you give an example or two of the kind of thing you'd want to run in all projects? That can help inform the design discussion.

I'm sure @octogonz will have input on this as well.

[wbern]

@iclanton I read the docs in the place you specified and it seems like bulk only ever runs packages according to the dependency graph, meaning it will wait for some packages to finish executing before running the next packages.

In many cases this is the perfect scenario, but in my case (watch instances everywhere), we have a naive approach right now to just start watch processes for every project in the monorepo. This is how we do it for now, until we work out a better solution (either coming directly from rush as per my other feature requests), or something we incorporate ourselves.

Looking at lerna run -parallel for example, but not really running scripts via that local npm scope. At least that's not part of the scope of my request.

--parallel
Similar to --stream, but completely disregards concurrency and topological sorting, running a given command or script immediately in all matching packages with prefixed streaming output. This is the preferred flag for long-running processes such as npm run watch run over many packages.

$ lerna run watch --parallel
Note: It is advised to constrain the scope of this command when using the --parallel flag, as spawning dozens of subprocesses may be harmful to your shell's equanimity (or maximum file descriptor limit, for example). YMMV

https://github.com/lerna/lerna/blob/master/commands/run/README.md

[octogonz]

We would accept an enhancement for Rush's custom commands that would enable processes to be spawned for many projects ignoring the topology, if you have a realistic use case that needs it. (But I'm curious how this relates to your solution in PR #1152 - does that eliminate this need? Or is this a prerequisite for that PR?)

I recognize that at some point Rush will eventually need to provide the generic feature for "execute an abitrary shell expression for all projects". It would be handy in some cases. But I have been procrastinating it because it can easily be abused to implement everyday operations for the monorepo, which could be a bad user experience: Rush's philosophy is that everyday monorepo commands should have command-line help, parameter validation, and proper error checking. Ideally custom commands should be designed and implemented thoughtfully. "Run this shell expression in 100 folders" usually doesn't meet any of that criteria.

[wbern]

We would accept an enhancement for Rush's custom commands that would enable processes to be spawned for many projects ignoring the topology

This is really all that's necessary in my case. The rest is loose discussion and, judging by what you mentioned of potential abuse, I don't want to suggest any of that functionality because I don't believe it is needed for the scope of this issue (IE. When you run everything and let some other process maintain order of compilation in one way or another).

It doesn't really relate to my other issue at all actually! 🙂 But we still need this feature in order to run watch instances before that enhancement is added, and I dare say this is an easier enhancement to add and, for now, enough to satisfy the switch to Rush for us (I know I've said this before but we realized this need later on).

[wbern]

So I managed to run watch commands externally, but for small monorepos where the intention is to never go above 5-10 packages, I think this could be a useful feature.

[slavafomin]

We really miss this feature after migrating from lerna 😢
It would be very useful to be able to run an ad-hoc command over many repositories.

Things to consider:

• be able to run command topologically or just in parallel as suggested in the discussion
• be able to scope the packages using some pattern, e.g. run the specified command only for a subset of packages in the monorepo

[slavafomin]

Also, the approach where you need to specify the actual command in each and every package is terrible when it's the same for all packages. This breaks DRY principle and creates a lot of overhead where you need to first add the script to each package and then to update it when the need arise.

I would suggest to:

1. Allow to specify the default implementation of the command directly in command-line.json

2. Allow to override this command in packages when you need to.

The logic is simple: if the script is defined in the package in question — run it, otherwise run the default command.

[octogonz]

I would suggest to:

1. Allow to specify the default implementation of the command directly in `command-line.json`

2. Allow to override this command in packages when you need to.

@slavafomin This is a good idea. I would vote for this feature. It is not difficult to implement. Could you create a separate GitHub issue for it?

Let's continue to use #1168 for the request to specify arbitrary commands on the command line like lerna exec. (We'll probably get this feature for free with #1553)

[slavafomin]

@octogonz sure, I've forked the issue here: #1797

[slavafomin]

WORKAROUND

While this issue is still pending, I've created a small CLI tool which allows to run arbitrary shell commands in multiple packages, it's called for-each-package. It works in any directory by doing a recursive search, ignoring common directories like /node_modules and it respects the paths in .gitignore. Filtering packages by name is also possible.

Here's how it could be used:

• npm i -g @sfomin/for-each-package

• for-each-package -n "@acme/*" "pwd && ls -l"

[octogonz]

Nice!

[matthiasg]

We would accept an enhancement for Rush's custom commands that would enable processes to be spawned for many projects ignoring the topology, if you have a realistic use case that needs it. (But I'm curious how this relates to your solution in PR #1152 - does that eliminate this need? Or is this a prerequisite for that PR?)

I recognize that at some point Rush will eventually need to provide the generic feature for "execute an abitrary shell expression for all projects". It would be handy in some cases. But I have been procrastinating it because it can easily be abused to implement everyday operations for the monorepo, which could be a bad user experience: Rush's philosophy is that everyday monorepo commands should have command-line help, parameter validation, and proper error checking. Ideally custom commands should be designed and implemented thoughtfully. "Run this shell expression in 100 folders" usually doesn't meet any of that criteria.

Right now I would have loved to run a script in parallel on all packages to ensure sorting of all dependencies is in alphabetical order. Sadly only a package.json script can be called per package. So I have to fall back to writing a custom script (redoing all the work of parameter validation if required).

BTW this is needed as rush add does not always seem to add in alphabetical order (pnpm) ? Which blows up when trying to publish.

[bacebu4]

Is there any new ideas how to implement it directly in Rush in order that is not going to be abused?

I got the following idea: add a new flag to rushx that will take projects where the command should be executed

It seems like the easiest solution for me right now since we probably don't need the way to run script in all packages at the same time
Generally only couple packages needs to be run that way and the described solution might come really handy

[eighty4]

Does rushx only work in a single package and on that single package and can only be used to run a script configured in that package's package.json?

Today is the first time I've discovered rushx. I never saw it in the docs.

I just used rush build for the first time and thought it was great. Is creating a custom Rush command still the only solution for running any other package.json script in each package other than build?