Relationship between tokio worker threads and threads in GDB #4159
-
|
Hi, I'm attempting to debug an issue with some code using tokio that spawns a number of non-blocking tasks (perhaps 10-100k within a short period of time). For concreteness, what I'm doing is downloading some large files from cloud storage (say, 100GB in total) in smaller chunks; the code I've written hits some scale issues somewhere between 50GB and 100GB total download size that I'm trying to dive into a bit more using GDB. The program appears to stall for several minutes, and what I'm attempting to do is attach to it using GDB during the stall and just answer the question "what is the program doing right now?". The confusing thing I'm seeing is that there are many, many more threads that show up in GDB as "tokio-runtime-w" than I requested (sometimes as many as 10x... it's not consistent). I built the tokio runtime with: I then block with: where the core of (and My understanding is that because I'm never using spawn_blocking, I should be limited to 8 threads, given how I'm setting up the runtime. However, when I attach to the process with GDB and run ... which is to say, many more than the 8 threads I requested. There are many other complicated factors here (including it's running inside a python interpreter as a native module, there's a progress bar whose updates I've elided, possible bugs in my surrounding code, etc.), so I'm not necesarily looking for a fix to the problem, but rather to try to understand: is this expected behavior? Should I expect that if I spawn a tokio runtime with Thank you! (For reference, I'm running tokio 1.11.0, rust 1.53.0, ubuntu 16.04.1 in a docker container.) |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Some IO operations have to be run using a blocking threadpool due to blocking syscalls being the only options for said calls, so tokio has a blocking thread pool for this purpose. Try using the |
Beta Was this translation helpful? Give feedback.
-
|
Specifically, this is for file system operations. So if you are writing the data to a file, then that will internally use spawn_blocking. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! That makes sense; I am writing each downloaded chunk to a file. Setting (For any future readers, reducing the number of max_blocking_threads did not in fact help with the problem of a long stall in my program, but I can at least now dive into what's going on here in gdb.) |
Beta Was this translation helpful? Give feedback.
Some IO operations have to be run using a blocking threadpool due to blocking syscalls being the only options for said calls, so tokio has a blocking thread pool for this purpose. Try using the
max_blocking_threads()method on the runtime builder, and see what that does. If this drops the thread count down, then you have your answer.