LSFManager: optionally stream worker stdout to master #168
Conversation
This updates the LSFManager to allow stdout and stderr streams of workers to be streamed (optionally) to master. The `bpeek` invocation when passed the `-f` flag behaves similar to `tail -f`. This is the default set for `bpeek_flags`. The resultant stream is then passed on to the Distributed module after parsing the worker connection string and some initial error handling for the case where job startup is delayed or failed. The Distributed module then takes over the subsequent display of worker's stdout messages on master. If streaming of worker's stdout messages is not desired, then the `-f` flag can be omitted from `bpeek_flags`. Then `bpeek` behaves similar to `cat` and is used only to read the initial connection string. As a positive side effect of this change, it now also avoids the earlier `redirect_stderr()` call which could have caused race conditions with multiple async invocations of the `bpeek` method. Few methods are also renamed to have a `lsf_` prefix to avoid unnecessary clashes with other parts of the package.
|
looks good. but what is wrong with |
|
Thanks for revewing! Yes dispatch would take care, but would likely not serve any purpose? Because internal methods are unlikely to have any common purpose across managers. Also prefixing would avoid having another poorly qualified method of a different manager with same name trigger method ambiguities I felt. Does this seem reasonable? |
|
Unless there are any further major concerns, can this be merged? |
|
i don't want to bikeshed, so i'll merge. will be great to have all i/o displayed by master. thanks! but i will say that adding thanks again! |
|
Thanks! I was not so hung up on the naming too... agree with your points too. Maybe we need #58 to avoid such worries. |
This updates the LSFManager to allow stdout and stderr streams of workers to be streamed (optionally) to master. The
bpeekinvocation when passed the-fflag behaves similar totail -f. This is the default set forbpeek_flags. The resultant stream is then passed on to the Distributed module after parsing the worker connection string and some initial error handling for the case where job startup is delayed or failed. The Distributed module then takes over the subsequent display of worker's stdout messages on master.If streaming of worker's stdout messages is not desired, then the
-fflag can be omitted frombpeek_flags. Thenbpeekbehaves similar tocatand is used only to read the initial connection string.As a positive side effect of this change, it now also avoids the earlier
redirect_stderr()call which could have caused race conditions with multiple async invocations of thebpeekmethod. Few methods are also renamed to have alsf_prefix to avoid unnecessary clashes with other parts of the package.