Reduce CPU usage of gradle run

[jaymode]

The RunTask is responsible for logging output from nodes to the console
and also stays active since we want the cluster to keep running.
However, the implementation of the logging and waiting resulted in a
spin loop that continually polls for data to have been written to one
of the nodes' output files. On my laptop, this causes an idle
invocation of gradle run to consume an entire core.

The JDK provides a method to be notified of changes to files through
the use of a WatchService. While a WatchService based implementation
for logging and waiting works, a delay of up to ten seconds is
encountered when running on macOS. This is due to the lack of a native
WatchService implementation that uses kqueue or FSEvents; the current
WatchService implementation in the JDK uses polling with a default
interval of ten seconds. While the interval can be changed
programmatically it is not an acceptable solution due to the need to
access the com.sun.nio.file.SensitivityWatchEventModifier enum, which
is in an internal package.

The change in this commit instead introduces a check to see if any data
was available to read and log. If no data is available in any of the
node output files, the thread sleeps for 100ms. This is enough time to
prevent consuming large amounts of cpu while still providing output to
the console in a timely fashion.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[alpar-t]

LGTM, thanks for catching it !

[pugnascotia]

LGTM.

In case we ever want to revisit this, there seems to be a couple of MacOS implementation that use JNA and the Carbon APIs to provide a native watch implementation. For example:

https://github.com/gmethvin/directory-watcher

[jasontedor]

The Carbon APIs were never made 64-bit and have been removed since macOS Catalina when 32-bit support was dropped, we definitely wouldn't want to build a solution on them.

[mark-vieira]

I remember we encountered these same limitations when implementing continuous builds in Gradle. In the end we just lived with the fact that the MacOS experience has potential for high latency.

Thanks for catching this Jay. The only other option I'd see is to refactor this to use JavaExec and just directly stream back stdout. That'd have potential to be significant though, as we'd effectively have to reimplement any desired logic in our launch script in the build.