roachprod: refactor Monitor

Previously, Monitor was only able to monitor processes already started. It could
not detect a new processes if it was started after monitor started monitoring.

This change uses the new monitor scripts `monitor_local.sh` &
`monitor_remote.sh` which produces a frame of processes when any cockroach
process changes occur including when a new cockroach process is started.

The logic for detecting changes in processes has been moved to the "client"
side. The scripts are now less complicated and only have the responsibility of
sending through a frame of processes. A frame consists of the cluster label,
process id, and processes status of all cockroach processes.

In addition, Monitor has been moved into its own source file for better
separation of logic. The `ignore empty nodes` functionality has also been removed,
as it is an unreliable way of detecting "empty nodes", and does not serve any
real purpose anymore.

Informs: cockroachdb#118214
Epic: None

pkg/cmd/roachprod/flags.go

@@ -311,11 +311,6 @@ func initFlags() {
 	logsCmd.Flags().StringVar(&logsProgramFilter,
 		"logs-program", "^cockroach$", "regular expression of the name of program in log files to search")
 
-	monitorCmd.Flags().BoolVar(&monitorOpts.IgnoreEmptyNodes,
-		"ignore-empty-nodes", false,
-		"Automatically detect the (subset of the given) nodes which to monitor "+
-			"based on the presence of a nontrivial data directory.")
-
 	monitorCmd.Flags().BoolVar(&monitorOpts.OneShot,
 		"oneshot", false,
 		"Report the status of all targeted nodes once, then exit. The exit "+

pkg/roachprod/install/BUILD.bazel

@@ -10,6 +10,7 @@ go_library(
         "expander.go",
         "install.go",
         "iterm2.go",
+        "monitor.go",
         "nodes.go",
         "run_options.go",
         "services.go",
@@ -21,6 +22,8 @@ go_library(
         "scripts/download.sh",
         "scripts/start.sh",
         "scripts/open_ports.sh",
+        "scripts/monitor_remote.sh",
+        "scripts/monitor_local.sh",
     ],
     importpath = "github.com/cockroachdb/cockroach/pkg/roachprod/install",
     visibility = ["//visibility:public"],