[rush] make incremental build happen on file changes (watch mode)

[wbern]

I'd love a feature where you would type something like rush watch, and then it would invoke rush build whenever it sees that a file has been touched. This would be amazingly performant for a large repo.

Behavior would be is somewhat similar to how nodemon works. Some suggestions by current lerna maintainer here lerna/lerna#1722 (comment)

The main motivation behind this is that we'd be able to skip having watch instances running for each package, and instead have one watch at the top.

I think this would be a big step forward for scaled micro frontend monorepos. We have 30 packages now and the RAM usage of each is killing those poor developer laptops (for each package, npm is 20 mb, the process itself a minimum of 60mb).

Since rush already features incremental builds, and building upstream and downstream, I think this would be the perfect thing to build onto what is in Rush.

Full disclaimer, I'm not using rush yet but Lerna. This feature would tip the scale for me use rush, no doubt.

Another minor thing, I'd love to be able to have this watcher actually start watch instances of webpack. I know this will block execution, so some sort of daemonization would be necessary, and likely not something that rush could offer without some complementary implementation in each micro app. I thought perhaps it would be neat if rush could help the daemonization process by either documenting it, or letting the micro apps do process.send() with some daemon activation message. This area isn't my strong suit but this is a need that we also have, and that I plan to document here later on when it's done in my project (currently using lerna but that doesn't matter in this context).

[octogonz]

@iclanton FYI. We should totally do this. But the design needs to be compatible with the plan for sharded builds.

[wbern]

@octogonz, out of curiosity, what has been your goto approach when you need to do incremental builds locally during your day-to-day development?

[wbern]

I've been doing a lot of thinking of how I would solve the issue of rush building using webpack watch instances and avoid blocking execution.

My idea consists of these 4 components

• Rush watch
• JobRequest
• IPC Server
• one or more packages offering build+watch instances

1. rush watch triggers rush build, wants to build Package X
2. nodejs process "JobRequest" starts and connects to IPC server
   2.a if there is no compile status for package X, let JobRequest spawn a detached process of Package X's build script instance, then let JobRequest exit when it's in compile status done. If it fails, return non-zero.
   2.b if there is a compile status already for X, assume that the watch instance already knows it needs to rebuild, and just wait until it reaches status "done". When it's done, let JobRequest exit.
3. Repeat first two steps for remaining packages, if any, which is rooted in the orchestration of the rush build command.
4. When all JobRequests/packages have been built, rush watch is completed
5. When exiting rush watch, shut down ipc server which in turn should shut down all detached processes.

Possible concerns:

• We don't know for sure if the watch instance picked up that it needs to recompile. This may increase solution complexity.
• When JobRequest queries the status of a package, the build may have completed already and we see a "done" status from the get go. This could possibly be circumvented by requiring sign-off inside the webpack watch instance from the IPC server, which in turn is approved by the JobRequest, but this is problematic because we want the watch instances to rebuild without rush, otherwise it would be pointless to have them in the first place (if rush implements the watch logic it may be slightly more relevant).
• webpack 5 features a lot more caching which may make the webpack watch solution less relevant, but that's at least 6-12 months away.

I think the easiest solution, in 1 sentence, is to create JobRequests that in turn request detached instances of the build, reused or spawned anew based on existing processes.

Update: if this solution is very complex or overwhelming, a lock file based ipc solution might work better. I'm going to think about it.

Update 2: it would be relatively easy to incorporate something like PM2 into this, making the observation of watch instances more maintainable and perhaps also easier to shutdown.

[wbern]

Final update to all the above: we just execute build watch scripts on select packages now in my org. A script lists all projects and we choose among them. No upstream or downstream support, so normally you'd start off with a "rush build" before that.