Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement max_tasks_per_child #373

Open
ddelange opened this issue Nov 2, 2022 · 2 comments
Open

Implement max_tasks_per_child #373

ddelange opened this issue Nov 2, 2022 · 2 comments

Comments

@ddelange
Copy link

ddelange commented Nov 2, 2022

Hi 👋

Analogous to concurrent.futures.ProcessPoolExecutor's max_tasks_per_child (added in cp3.11) and multiprocessing.pool.Pool's maxtasksperchild (added in cp3.2) keyword arguments, it would be great to be able to control after how many completed tasks a loky subprocess is flushed and replaced with a new subprocess.

Our dask workers are currently consistently facing loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.:

Most likely caused by upstream memory leaks in lxml, hitting our 60GiB mem limit over time due to running the same loky pool subprocesses over 5+ hours. Periodically flushing the workers (spawn start method) will most likely fix these errors.

Many thanks!

@ogrisel
Copy link
Collaborator

ogrisel commented Feb 21, 2023

That sounds like a good idea, feel free to submit a PR and link it back to this issue.

@ddelange
Copy link
Author

I think this is a pretty low-level change in need of someone who's deep into the source code, especially with regard to start methods:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants