[DNM] queued<->no-worker transitions
#7267
Conversation
|
AFAICT, We'd need the TaskGroup or cluster size to change so a task no longer looks root-ish (easy). Then, we'd need an event to occur which tries to schedule a task off (If there are running workers, then the task will just scheduled immediately thanks to I can't think of an event which would try to schedule tasks off the queue when there were no running workers. All selection of queued tasks happens through distributed/distributed/scheduler.py Lines 7854 to 7859 in 00bf8ed The events that cause a task to be scheduled off the queue are:
But again, in all these cases, we only pick from the queue if the worker is running. And if at least one worker is running (and the task has no restrictions), it means |
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 15 files ± 0 15 suites ±0 6h 29m 46s ⏱️ + 21m 38s For more details on these failures, see this check. Results for commit 04201ff. ± Comparison against base commit 69a5709. |
|
This, plus Here, when root-ish-ness of a task changes, we're not eagerly moving tasks between the Specifically, if there are tasks in That means the tasks will still mostly be scheduled as though they're queued root-ish tasks. They won't all be submitted to workers up front, like non-rootish tasks should. The only difference will be that when another task completes and exactly one gets picked from the queue, it will always be scheduled, whereas a "proper" queued task might stay on the queue if there was a downstream task to run. This effectively adds two more implicit scheduling cases: "queued but not root-ish", and "in I'm currently leaning away from this approach and think caching root-ish-ness #7262 is still the most consistent and maintainable option. Plus, it's the closest to what we want the end state to be (where root-ish-ness is a static property). |
There was a problem hiding this comment.
I'm currently leaning away from this approach and think caching root-ish-ness
We discussed that caching is out of the question. It would create conditions that are incredibly difficult to debug.
Only one of the asserts that were added here fail on master. If this is the only failure case we're concerned about, I suggest to drop the issue entirely. Queuing a little bit too much is a conservative error we can afford in a rare situation.
What I understood from the initial problem report is that there are potential deadlocks. If that's true, I'd like them to be reproduced first before we discuss solutions. All of these tests run fine for me on main
| fs = [c.submit(inc, root, key=f"inc-{i}") for i in range(s.total_nthreads * 2 + 1)] | ||
| # ^ `c.submit` in a for-loop so the first tasks don't look root-ish (`TaskGroup` too | ||
| # small), then the last one does. So N-1 tasks will go to `no-worker`, and the last | ||
| # to `queued`. `is_rootish` is just messed up like that. |
There was a problem hiding this comment.
This should verify the assumption of when the tasks are classified as root-ish. This is otherwise incredibly opaque and brittle
| @gen_cluster( | ||
| client=True, | ||
| nthreads=[("", 1)], | ||
| config={"distributed.scheduler.work-stealing": False}, | ||
| ) | ||
| async def test_queued_rootish_changes_scale_up(c, s, a): | ||
| "Tasks are initially root-ish. After cluster scales, they aren't." | ||
|
|
||
| root = c.submit(inc, 1, key="root") | ||
|
|
||
| event = Event() | ||
| clog = c.submit(event.wait, key="clog") | ||
| await wait_for_state(clog.key, "processing", s) | ||
|
|
||
| fs = c.map(inc, [root] * 5, key=[f"inc-{i}" for i in range(5)]) | ||
|
|
||
| await async_wait_for(lambda: len(s.tasks) > len(fs), 5) | ||
|
|
||
| if not s.is_rootish(s.tasks[fs[0].key]): | ||
| pytest.fail( | ||
| "Test assumptions have changed; task is not root-ish. Test may no longer be relevant." | ||
| ) | ||
| if math.isfinite(s.WORKER_SATURATION): | ||
| assert s.queued | ||
|
|
||
| async with AsyncExitStack() as stack: | ||
| for _ in range(3): | ||
| await stack.enter_async_context(Worker(s.address, nthreads=2)) | ||
|
|
||
| if s.is_rootish(s.tasks[fs[0].key]): | ||
| pytest.fail( | ||
| "Test assumptions have changed; task is still root-ish. Test may no longer be relevant." | ||
| ) | ||
|
|
||
| await event.set() | ||
| await clog | ||
|
|
||
| # Just verify it doesn't deadlock | ||
| await c.gather(fs) |
| # KEY ASSERTION: | ||
| # the next task on the queue got scheduled, exceeding worker-saturation, because | ||
| # even though it was in `queued`, it was no longer root-ish. | ||
| assert len(s.workers[a.address].processing) == a.state.nthreads + 1 |
There was a problem hiding this comment.
Of all the new tests, this assert is the only one that fails on main
There was a problem hiding this comment.
This assertion states that a task that is no longer classified as root-ish is still queued even though.
Is this the only error case we are concerned about? If so, I suggest to just drop the issue. If there is something else, the tests should reproduce the issue first.
Yet another approach to #7259. See #7259 (comment).
decide_worker_or_next_statefunction, which picks the appropriatedecide_workerfunction for the task (based on root-ish-ness and worker-saturation setting) and either returns a worker, or the state the task should transition to if no worker is availablequeuedandno-workerstates. These can only occur when we try to schedule a task in one of those states, no workers are available, and the root-ish-ness status has flipped.pre-commit run --all-files