Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queued tasks could deadlock in combination with secede #7223

Closed
Tracked by #7213
gjoseph92 opened this issue Oct 28, 2022 · 0 comments · Fixed by #7224
Closed
Tracked by #7213

Queued tasks could deadlock in combination with secede #7223

gjoseph92 opened this issue Oct 28, 2022 · 0 comments · Fixed by #7224
Assignees
Labels
bug Something is broken deadlock The cluster appears to not make any progress scheduling

Comments

@gjoseph92
Copy link
Collaborator

When a task secedes, we aren't scheduling any queued tasks right now. So if a root-ish task calls secede, and all workers are full, the next task on the queue won't be scheduled, even though a slot has opened up on the worker. This could cause a deadlock if the long-running task has created another task and is waiting for it to run, but it's stuck in the queue.

@gjoseph92 gjoseph92 added bug Something is broken scheduling labels Oct 28, 2022
@gjoseph92 gjoseph92 self-assigned this Oct 28, 2022
@crusaderky crusaderky added the deadlock The cluster appears to not make any progress label Oct 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is broken deadlock The cluster appears to not make any progress scheduling
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants