Error transitioning worker task from waiting to fetch

[jrbourbeau]

Similar to #4721, I had some folks report offline that they were encountering an error due to an unsupported task state transition on their workers (traceback shown below)

Traceback (most recent call last):
  File "/usr/local/python/python-3.7/std/lib64/python3.7/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/usr/local/python/python-3.7/std/lib64/python3.7/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/usr/local/python/python-3.7/std/lib64/python3.7/site-packages/distributed/worker.py", line 2261, in gather_dep
    self.transition(ts, "fetch", worker=worker)
  File "/usr/local/python/python-3.7/std/lib64/python3.7/site-packages/distributed/worker.py", line 1584, in transition
    func = self._transitions[start, finish]
KeyError: ('waiting', 'fetch')

which was observed using dask and distributed 2021.04.0

cc @fjetter @gforsyth

[fjetter]

FYI That's a work stealing transition

We could patch this but I'm not very eager of introducing more and more transitions right now without a solid concept to avoid this kinds of problems.

[gforsyth]

I think while patching #4721 was more of a "well, that shouldn't happen but we can work around it for now" this seems like more of an oversight (by me). waiting -> fetch is definitely a valid transition (and we have handling for the inverse)

[jrbourbeau]

Yeah, I with both of you. We should continue to develop higher-level solutions for improving worker state transitions, which I believe Florian is working on now, and also add a waiting -> fetch transition in the meantime.