Creating too many threads

[JaskaranSM]

Hi, I m using this client for a telegram bot, the issue I am facing is a slow increase of threads, Here is the torrent that does the fastest increase of threads. Basically from the threadcreate profile, the thread count just keeps on increasing,
Here status of a download after 10 minutes of run, that TH is the thread count from threadcreate profile, GR is the NumGoroutines
Hajimete no Gal (BD 1080p FLAC) [Dual Audio] [FLE] - Downloading
[▏ ] 1.13% , 831.6 MB of 71.8 GB at 2.0 MB/s, ETA: 10h0m0s | P: 20 | S: 9
GID: LFdRazyJyiDEDZvv

Alloc: 40.9 MB | TAlloc: 2.3 GB | GC: 116 | GR: 139 | TH: 238

I m using this config
config := torrent.NewDefaultClientConfig() config.EstablishedConnsPerTorrent = 100 config.HTTPUserAgent = "qBittorrent/4.3.8" config.Bep20 = "-qB4380-" config.UpnpID = "qBittorrent 4.3.8" config.ExtendedHandshakeClientVersion = "qBittorrent/4.3.8" config.Seed = true config.ListenPort = 62642
I m running my app on a host that have a limit of 256 threads so I cannot afford to have this many threads, Is this expected Behavior from anacrolix/torrent?
goroutineTrace.txt
Anacrolix/Torrent version: v1.46.0

[anacrolix]

I don't currently have time to look into this in depth right now. Note that goroutines do not map to threads 1:1. Check the threads profile to see how many native threads you actually have.

[JaskaranSM]

I don't currently have time to look into this in depth right now. Note that goroutines do not map to threads 1:1. Check the threads profile to see how many native threads you actually have.

My app's progress status message does call threadcreate profile and gets the count (live), I have mentioned the thread count already as in the original issue here.
GR: 139 | TH: 238
The threadcount just keeps on increasing like something is blocking something (like IO perhaps), and go runtime is constantly trying to not find itself in a situation where the program hangs so the runtime just spins up a new thread. After some further investigation with top , I saw that most of the threads it created were sleeping most of the time. I am not sure how to debug it properly as anacrolix/torrent is very complex project for me. Have you ever encountered a situation where anacrolix/torrent has lead to creation of so many threads, I take you are running the client in production most of the time? I am using the file implementation of storage. Maybe is it because IO is messing up and that is causing creation of threads?

[anacrolix]

Yeah things that come to mind that could block, are related to C? Perhaps disabling uTP, piece completion using sqlite, any memory-mapping. But I'm pretty sure that none of those should cause this problem, for various reasons. Can you provide the thread profile? It sounds like it has nothing useful in it though?

[JaskaranSM]

I tried disabling uTP, using BoltDB instead of SQLite for piece completion, the issue remained. threadcreate profile is only useful to get the number of threads, no proper stack trace collection is there, reference
Output from threadcreate profile:
threadcreate profile: total 100
99 @
0x0

1 @ 0x524915 0x5250db 0x5253f0 0x521419 0x5520c1
0x524914 runtime.allocm+0xd4 /usr/local/go/src/runtime/proc.go:1745
0x5250da runtime.newm+0x3a /usr/local/go/src/runtime/proc.go:2090
0x5253ef runtime.startTemplateThread+0x8f /usr/local/go/src/runtime/proc.go:2167
0x521418 runtime.main+0x1b8 /usr/local/go/src/runtime/proc.go:229

[JaskaranSM]

Update!
Switched to mmap backend for storage and this issue went away, For now this issue seems to be happening because of file backend, I saw you are doing OpenFile call for every torrent write/read call. I have a doubt, wouldn't it do three syscalls for every write/read ? Open -> write/read -> Close? Would it be better to have the file handles open and read/write calls are done when data is received? the file handles can be closed when they are completed. (context: go runtime creates new threads when syscalls block and no free thread is available. this should decrease the amount of syscalls we do for every read/write call).

[anacrolix]

Thanks for resolving the issue more. I believe the open syscall is not reliably asynchronous on many platforms, so a dedicated thread may be necessary in the Go runtime. On top of that, the concurrency to chunk writes is proportional to the number of receiving connections, which you've artificially raised. Do you also have have a lot of torrents downloading? Each connection in each torrent could contribute in this way.

Currently the file backend closes file descriptors immediately: leaving them open requires one for every file, which gets very large for many files. Go recently raised the file descriptor soft limit by default, maybe it's now plausible to keep them open. Another fix is to start caching the file descriptors, or maintaining a cache. The last I can think of is to limit the write concurrency. I'm open to suggestions.

[anacrolix]

@JaskaranSM please reopen if you have feedback!