Dask worker having memory leaks

[dickreuter]

I have a process that runs around 500 client.submit() with dask distributed.

I have noticed that memory usage is gradually increasing on each worker. This only happens when I'm calling the tasks over dask distributed, but not when I call them directly in sequence.

What's the best way to detect / trace down memory leaks that coud be caused the a dask worker?

[mrocklin]

What's the best way to detect / trace down memory leaks that coud be caused the a dask worker?

Just however you would do this normally in Python. You might find things easier if you put them all in the same process/thread with the async mechanisms.

[mrocklin]

(my apologies for the delayed response)

[TomAugspurger]

Just FYI, there are many issues reporting seeming memory leaks. #2757 went into some depth and things maybe point to Python's object allocater, but it's hard (for me) to say for certain.

[mrocklin]

Thanks for the summary @TomAugspurger . Closing this as it's handled elsewhere.