-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Need better control over worker logging #51
Comments
Yeah, this would be very convenient! |
Also the workaround does not seem to work for me initially. I found you need to call the using Distributed
function Distributed.redirect_worker_output(ident, stream)
@schedule while !eof(stream)
println(readline(stream))
end
end before calling Also, if julia is started with |
I think the ideal fix here would need both
For this to be reasonably scalable some element of log filtering will be required on the worker nodes and it might also be necessary to designate a worker for log aggregation and sinking rather than using the master. |
Note that you don't see this unless you manually set the logger due to JuliaLang/julia#26798 (that doesn't fix the issue though). |
One thing I have done recently for distributed logging:
but I agree that @c42f ideas are probably worth exploring. |
Yeah I still think it would be nice to have logging "just work" with However it's also clear that redirecting logging to the master node will fall over if there's many nodes or high log volume. So for serious HPC work some distributed solution also seems necessary... such as dumping to a distributed filesystem, if you have one. @kpamnany wrote an interesting comment on that at CliMA/ClimateMachine.jl#134 (comment) |
Currently, even if you define your own
AbstractLogger
and setBase.global_logger(mylogger)
, worker loggers still prepend" From worker X:
to all logs.My current work-around is doing
So hurray for JuliaLang/julia#265 and all, but we really need better controls here.
The text was updated successfully, but these errors were encountered: