WIP Gc scheduler taskstate #6246
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fjetter
commented
Apr 29, 2022
Comment on lines
+4400
to
+4411
| # _workers = self.workers | ||
| # _events = self.events | ||
| # def remove_worker_from_events(): | ||
| # # If the worker isn't registered anymore after the delay, remove from events | ||
| # if address not in _workers and address in _events: | ||
| # del _events[address] | ||
|
|
||
| # cleanup_delay = parse_timedelta( | ||
| # dask.config.get("distributed.scheduler.events-cleanup-delay") | ||
| # ) | ||
| # self.loop.call_later(cleanup_delay, remove_worker_from_events) | ||
| # logger.debug("Removed worker %s", ws) |
There was a problem hiding this comment.
tornado call_later, add_callback, etc. things keep a reference around. With the above construction, we can break references to the scheduler but the sets/dicts still include references to WorkerState instances which themselves hold references to TaskState objects.
| @@ -49,7 +49,6 @@ class SemaphoreExtension: | |||
| """ | |||
|
|
|||
| def __init__(self, scheduler): | |||
| self.scheduler = scheduler | |||
There was a problem hiding this comment.
this is obvious but the GC should be able to break this cycle
| @@ -3078,7 +3091,7 @@ def __init__( | |||
| resources = {} | |||
| aliases = {} | |||
|
|
|||
| self._task_state_collections = [unrunnable] | |||
There was a problem hiding this comment.
This construction feels a bit off and is a relict of Cython. I will come back to this again once we removed the SchedulerState
This was referenced Apr 29, 2022
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I tried introducing a weakref instance check for the TaskState objects as proposed in #6226 (comment)
but I failed because they are actually never GCed.
It is not surprising that our classes are highly interconnected things and I am not at all surprised that we explicitly need to collect but the cycle detection doesn't suffice to release these objects. Turns out, we're leaking a bunch of explicit references all over the place.
Some of these are not obvious and I figured this bit of effort I invested here is worth preserving. I'll comment in-code for a few (to me) surprising things