How does Tokio decide how many threads to spawn/use and when?

[ilmoi]

I've been going through Tokio's mini-redis tutorial where it says:

Tasks are the unit of execution managed by the scheduler. Spawning the task submits it to the Tokio scheduler, which then ensures that the task executes when it has work to do. The spawned task may be executed on the same thread as where it was spawned, or it may execute on a different runtime thread. The task can also be moved between threads after being spawned.

So it sounds like Tokio itself decides when to spawn new threads / move tasks to those threads. If that's the case - how is that determined?

Context: for one of the apps I'm building I'm thinking whether to keep a task on the main async runtime or whether to spin up a new thread and start another async runtime there. The latter feels structurally cleaner, but if under the hood Tokio starts threads itself then I'm doing double work.

How should I think about it? Any links I can read to educate myself super welcome.

[Darksonn]

Tokio has two kinds of threads:

1. Worker threads. These run the tasks you spawn with tokio::spawn
2. Blocking threads. These run the tasks you spawn with tokio::task::spawn_blocking.

To answer your question, we need to consider these separately.

Worker threads. The number of worker threads never changes. Each worker thread can execute an unbounded number of asynchronous tasks concurrently. When you start the runtime, it looks up how many CPU cores you have and spawns that many worker threads. You can configure the amount of worker threads on the runtime builder, or as an option to #[tokio::main]. Asynchronous tasks may be moved from one worker thread to another at any .await to more evenly distribute the workload.

Blocking threads. Tokio will not spawn any blocking threads until they are needed due to calls to spawn_blocking. Since blocking tasks are not async, each blocking thread can work on only one spawn_blocking task at the time. More blocking threads are spawned if more spawn_blocking tasks are spawned at the same time, up to some upper limit (which is around 500 by default). Beyond the upper limit, tasks are put in a queue until one of the existing threads finish their work. The threads are shut down again if no new tasks are spawned within 10 seconds. All of these limits are configurable on the runtime builder.

Generally you do not gain any advantage from increasing the number of worker threads beyond the number of CPU cores, as that is the limit of parallelism on your machine. The one exception is if one of the async tasks is blocking the thread, but here the fix is to not block the thread, or to do it outside the worker thread (read the link for details). Spawning more worker threads to combat blocking is counterproductive.

[ilmoi]

This is very clear - thanks!

Related question - I've been struggling with some of the behavior of spawn_blocking as I describe on SO here. Any chance you can comment on what might be going on there?

Thanks in advance, you're a rockstar honestly.

[emogames]

I had the exact same question when reading the tutorial (https://tokio.rs/tokio/tutorial/spawning).
Great explanation!

This answer should be linked from the tutorial on some way :)