-
-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Watchtower + Multiprocessing #31
Comments
This now works when I set
and
|
Solved this issue by using this repo: https://github.com/jruere/multiprocessing-logging, which was spun out of this post: http://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python. All it resulted in was importing multiprocessing_logging and then adding |
Thanks, I probably need to add docs on how to deal with this, so reopening this issue to keep track of that. |
Just experienced this issue, thanks for the fix @Audace! @kislyuk it may be worth updating the docs to include a reference to that library. I'm working with django rq and any logging within a worker process was not making it into the watchtower batch. I'd assume other worker libraries like Celery would also experience this issue. Similar to @Audace 's answer, I updated my django app's ready() function (guaranteeing the install_mp_handler() is called after logging is setup and before any logs are sent)
|
Just wanted to add that although this seemed solved, I still encounter situations where all logging stops reaching CloudWatch suddenly, but continues to successfully log to local log files. I do not see any obvious causes, and after restarting my workers everything is back to normal as if nothing was wrong. If anyone has any thoughts, please feel free to share, and I will update if I come across a solution. @Audace not sure if you saw anything similar after implementing your solution? |
@kislyuk Is this the suggested way to deal with multi-process logging for this library? |
The suggested way to use logging in multiprocessing pools is to share nothing. Use one logger per worker process (or thread) and initialize the logger after forking. A shared logger will not work correctly with multiprocessing due to the stateful nature of the logger and race conditions that will arise between different copies of the logger in the different processes. |
@kislyuk So, I'm not using a multiprocessing pool, just manually creating all the child processes I want. If I pass a separate logger instance to each child process, does each logger instance log to a separate file / stream? Is there a way to collate everything based on based on some metric(like timestamp)? |
@redixhumayun that is for you to configure. Check the project documentation for how to configure the log stream names. Check the cloudwatch documentation for how to collate logs. |
I wanted to post my solution here, just in case it is useful to anyone else. The Python Logging Cookbook has some great examples of how to log to a single file form multiple processes, a sample of that can be found here. The sample linked above makes use of the Integrating with WatchTower is as easy as adding an additional handler at the output end of the queue. You can use the sample code from the link above and just modify the
|
The logger is not successfully writing to CloudWatch when using multiprocessing. I tested to see whether this was my configuration by dropping a watchtower handler and using a file handler. This logged perfectly, however, when switching back to the watchtower handler only messages before and after
outputs = pool.map(worker, inputs)
worked.Any idea how to fix this? Setting
use_queues
toTrue
didn't help.Sample code:
The text was updated successfully, but these errors were encountered: