LIFO task slot optimization in worker is potential footgun when a task doesn't yield.

[tobz]

Version
tokio 1.14.0

Platform
Linux derp 5.11.0-31-generic #33-Ubuntu SMP Wed Aug 11 13:19:04 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description
In a small application used to benchmark the performance of some code that dealt with writing to and reading from an external data source, I encountered a particular mysterious issue related to how Tokio handles the worker scheduler optimization of storing the "next task to poll" in a slot that is not considered by the normal work-stealing algorithm.

Essentially[1], the code has a #[tokio::main] annotated async fn main(), which then spawns two tasks -- one for the reader, and one for the writer -- and runs them until both complete. The complication comes in where the writer, as written, has no need to yield: its work happens off-thread and so it is wrapped in an asynchronous interface (Sink) but never does anything to trigger a yield, or manually yields.

Where this caused an issue is that this writer task was on the same worker as the reader task, and additionally, it actively notifies the reader of progress (via AtomicWaker), which lead to a situation where the worker was holding on the reader task in its "next task to poll" slot, which I've been lead to understand is not considered when the normal work-stealing algorithm runs.

While the documentation, in many places, talks about tasks not yielding as being detrimental, and having the ability to cause other tasks to not be scheduled/polled, it was very unintuitive that having only two tasks spawned onto a multithreaded runtime with 16 worker threads still had no way to push the second task to another worker.

I'm not sure if there's even a reasonable way to avoid this, and maybe the answer is simply having something like tokio-console be able to better surface this issue, but it definitely felt like a quirk, and ultimately required an answer from a core Tokio dev: there was no existing blog post, Github issue, or other piece of information that explained this particular quirk.

[carllerche]

Closing in favor of #4941