[rush] Custom "watch" command -- only running on subset even with high -p, no log passthrough

[natew]

Hey, just playing with Rush and it seems pretty snazzy.

We use a big monorepo and have something in Lerna now that basically does npm run build-watch parallel in all packages. This is to run babel and watch for changes as you develop.

I wasn't sure the best way to do this in Rush. I added a custom command that seems to work, and then run it like: rush watch -p 100 -v.

Two things:

1. While it says "Registered" for all our packages (25) and says Executing a maximum of 100 simultaneous processes..., it only says started for 14 of them and doesn't seem to run beyond those 14.
2. It doesn't output any logs from the corresponding processes, so if there's a syntax error in one it's hidden.

Perhaps I'm not doing things the Rush way here. Is there support for long-running development processes in some better way?

[natew]

One note, the watch command doesn't rebuild anything it uses the babel-cli flag --skip-initial-build, which is helpful so you don't have to re-run builds all the time.

[nickpape]

Hey @natew , sorry for the delay in response.

You are correct that it registers all the packages, however, it still runs them in dependency order, which in your case means that it can only run 14 at a time.

We just added a rush build --changed-projects-only flag which should only rebuild the projects that change. You could probably run gulp serve (or equivalent) in your app project, and then run rush build -cpo whenever you make changes. This will just rebuild the project that changed (useful when you aren't making sweeping API changes, rather just trying to fix a bug in the middle of your dependency graph).

However, we don't currently have a watch command.

As for not outputting logs, that is by design. Rush only captures output to stderr and highlights it in the summary at the end of the build. [We chose this design since we have over 100 projects and >10,000 lines of logs every build]. However, you could use the --verbose/-v flag which will print all of your build logs.

[HipsterZipster]

@nickpape-msft Thanks again for creating such an amazing project that's designed for enterprise. We are major users of Rush and have been promoting it across my org.

Can you please provide a link to the pr or details on this "changed-projects-only" functionality? I searched in the repo and couldn't find it. More specifically, I'd really like to understand how this fits into the SDLC? Does this assume that devs check in the build artifacts for each project? If not, how else would a build server be able to proceed with building the downstream packages?

Thanks

[halfnibble]

Rush looks into each project in your mono repo and creates a dependency graph. It determines which projects have changed, and then builds everything that has changed and/or is affected by another package's change in the order necessary to propagate said changes. rush build does this. If you have 25 projects in your mono repo, then you would need a CPU with about 16 cores and at least 32 GB of RAM to run a rush command on every project at the same time because rush runs commands in separate child processes.

Basically, rush is not designed for this use case (according to my understanding). However, I want to look at some alternative ideas (maybe webpack's devServer can be configured to load 25 different projects?). If I can think of something, then I'll try to integrate it with rush.

[halfnibble]

I'm going to close this for now. If you have more ideas or questions, then send me an email.

[octogonz]

Agreed. This issue is the same as #1202, which includes a somewhat complete current design proposal for this feature.

@halfnibble