rTorrent reading 4x more data than sending

[ghost]

rTorrent seems to be reading 4-6x more data than it's seeding. In this case I'm seeing 100-160MB/s read from rTorrent while it's only seeding about 20MB/s worth of data.

--- IOTOP
Total DISK READ : 161.06 M/s | Total DISK WRITE : 27.28 M/s
Actual DISK READ: 161.06 M/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
42685 be/4 debian-d 161.06 M/s 27.28 M/s 0.00 % 70.79 % rtorrent

--- rTorrent
[Throttle off/off KB] [Rate 21780.5/27685.2 KB] [Port: 65300] [U 357/0] [D 141/0] [H 0/4096] [S 21/700/65023] [F 5339/32768]

Thoughts?

[ghost]

I've tried playing with the preload features to see if that causes it, I've tried disabling preloading and that doesn't help, nor does enabling it or making the memory cache larger. rTorrent doesn't seem to utilize the piece cache anyways even though i have it set to like 16G and have 300 torrents all seeding with active peers.

[Speeddymon]

I see you don't have throttling enabled.

Can you enable throttling for both upload and download, and see if you can replicate it with throttling enabled. Your throttle should be 80% of your total maximum speed from your ISP.

So if your upload speed is 20MB/s then set the upload throttle to around 17MB/s, and if your download speed is 100MB/s then set the download throttle to 80MB/s.

If you are able to replicate it with throttling enabled, then please set the download throttle to something very low and see if the issue still occurs (uploads drop down to 1/4 of your throttled download speed)

[ghost]

I did try this with throttling too.

It seems to happen with lots of uploading slots and lots of torrents. I think what's happening is a slow client requests a piece, rTorrent loads the entire piece into memory, and attempts to send it but in smaller blocks. In the meantime many many torrents and clients are doing this causing an internal cache thrashing. So a piece is having to be reloaded every time a peer requests the next block because the piece gets kicked out of the cache due to other pieces being loaded. So instead of reading the entire piece from disk, how about just the block the peer requests? I know this is less efficient for most setups but an option to change the behavior on how data gets loaded from disk would be nice.

[Speeddymon]

Hmm..

Preload options might help here.

pieces.preload.min_size.set
pieces.preload.min_rate.set
pieces.preload.type.set

min_size is set to 131072 (or 128kb)

min_rate appears to be set to 5192

Sadly the documentation is lacking and I'm not an expert at reading this code, so I can't tell if that's 5192 bytes or a factor of time or what have you. Looks like valid values for type are 0 and 1, but no idea what either of those means.

Try lowering the min_size in your rtorrent config and see if that helps any.

[ghost]

I've also tried with and without preload, doesn't make a difference.

[Speeddymon]

You confirmed it happens even with just one active torrent and all others stopped?

If you could, please test with just one active torrent, and the rest stopped, assuming you haven't already. ;-)

If it doesn't happen there, then that will probably confirm the theory about slow clients, because you're then only dealing with a small subset of the number of peers you normally are interacting with.

[ghost]

Here's the result of downloading a SINGLE torrent. With 0 upload speed.

It's reading an insane amount of data for only downloading a torrent at 2MB/s. I mean what's the point of the freaking disk process? I'm using the 0.9.6 published build.

TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
24180 be/4 rtorrent 8.37 M/s 2.03 M/s 0.00 % 7.69 % rtorrent
24181 be/4 rtorrent 3.29 M/s 0.00 B/s 0.00 % 3.15 % rtorrent [rtorrent disk]

[Throttle 200/2048 KB] [Rate 2.3/2071.4 KB] [Port: 12345]

[ghost]

Using a completely stock rtorrent and stock debian install (no .rtorrent.rc file)... downloading a SINGLE torrent with ZERO UPLOAD.

TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
24861 be/4 rtorrent 10.21 M/s 2.99 M/s 0.00 % 7.71 % rtorrent

[Throttle off/off KB] [Rate 3.5/3091.0 KB] [Port: 6944]

This is a huge performance impact for rTorrent. This renders it unusable for me and is likely impacting the entire userbase and they don't even know it.

[Speeddymon]

I can see it reading that much with hashing enabled but you have it off.

Can you try to reproduce this with deluge?

It uses libtorrent also, and will help to narrow down whether its a bug in libtorrent or rtorrent itself.

[ghost]

Deluge uses a different libtorrent actually - libtorrent-rasterbar.

I tried builds all the way back to 0.9.0 and they all are affected. I haven't tried prior versions since they won't compile under newer versions of the C/C++ compilers.

Deluge does not have this same problem. It also performs significantly less write op/s too. I get maybe an average of 30 IOops every few seconds when the write cache is being flushed. When I was using rTorrentI was seeing a constant 300-400 IOops/s with rTorrent.

TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
15868 be/4 deluge 0.00 B/s 2.49 M/s 0.00 % 0.99 % python /usr/bin/deluged

[chros73]

I can see it reading that much with hashing enabled but you have it off.

check_hash is on by default (e.g. when you don't use a config file). Although it should mean "Check hash for finished torrents.", but who knows, can you try to disable it at first with:

pieces.hash.on_completion.set = no

This is a huge performance impact for rTorrent. This renders it unusable for me and is likely impacting the entire userbase and they don't even know it.

What OS is this exactly? Is it run on a physical device or in virtual environment?
I have just checked on Ubuntu 14.04 laptop. These are the figures that I'm getting with sudo iotop -ok (interactive mode) checking different times and with this config (using my rtorrent-ps fork, based on 0.9.6, not on master):

[Rate 1697.0/1465.2 KB]
19812 be/4 chros73     990.25 K/s    2.53 M/s  0.00 %  2.06 % rtorrent
19812 be/4 chros73    1668.26 K/s 1144.28 K/s  0.00 %  1.87 % rtorrent
19812 be/4 chros73       2.79 M/s 1509.73 K/s  0.00 %  3.11 % rtorrent

[Rate 1469.0/2372.0 KB]
19812 be/4 chros73    1175.79 K/s    2.93 M/s  0.00 %  2.71 % rtorrent
19812 be/4 chros73       3.14 M/s 1419.14 K/s  0.00 %  4.33 % rtorrent
19812 be/4 chros73       2.18 M/s    2.71 M/s  0.00 %  2.94 % rtorrent
19812 be/4 chros73    1304.04 K/s 1591.24 K/s  0.00 %  1.71 % rtorrent

[Rate 1944.1/4556.1 KB]
19812 be/4 chros73     697.34 K/s    5.38 M/s  0.00 %  3.83 % rtorrent
19812 be/4 chros73    1716.96 K/s    4.05 M/s  0.00 %  0.10 % rtorrent
19812 be/4 chros73    1239.27 K/s    4.83 M/s  0.00 %  3.54 % rtorrent

[Rate 2071.3/  3.8 KB] 
19812 be/4 chros73    4846.16 K/s    0.00 K/s  0.00 % 16.97 % rtorrent
19812 be/4 chros73    7732.49 K/s    0.00 K/s  0.00 % 22.20 % rtorrent
19812 be/4 chros73    3456.99 K/s    0.00 K/s  0.00 %  7.55 % rtorrent
19812 be/4 chros73    3328.77 K/s    0.00 K/s  0.00 %  9.09 % rtorrent
19812 be/4 chros73    5752.25 K/s    0.00 K/s  0.00 % 27.22 % rtorrent

As you can see, I don't have that insane fluctuation like you do.
The last readings are interesting , sometimes it can be almost 4x times of data that it reads.
From the config I use these values (that can be interesting to this issue):

# Set the max amount of memory space used to mapping file chunks. This refers to memory mapping, not physical memory allocation. (max_memory_usage) This may also be set using ulimit -m where 3/4 will be allocated to file chunks.
pieces.memory.max.set = 2048M
# Adjust the send and receive buffer size for socket. Disabled by default (0), this means the default is used by OS (you have to modify the system wide settings!!!) (send_buffer_size, receive_buffer_size)
#   Increasing buffer sizes may help reduce disk seeking, connection polling as more data is buffered each time the socket is written to. It will result higher memory usage (not by rtorrent process!).
network.receive_buffer.size.set =  4M
network.send_buffer.size.set    = 12M
# Preloading a piece of a file. (Default: 0) Possible values: 0 (Off) , 1 (Madvise) , 2 (Direct paging). (https://github.com/rakshasa/rtorrent/issues/418)
pieces.preload.type.set = 2
# Check hash for finished torrents. (check_hash)
pieces.hash.on_completion.set = no

I mean what's the point of the freaking disk process?

I don't have rtorrent [rtorrent disk] process at all, just rtorrent. :)

Edit: I have to mention that I'm using modded 3.16.0-pf4-chros02 #4 SMP PREEMPT linux kernel with io tweaks, I don't know whether it matters or not.

[chros73]

downloading a SINGLE torrent with ZERO UPLOAD.

I've just tried it, just in case (with the above linked config): downloading only 1 with zero upload.

[Rate   4.5/4067.1 KB]
 9094 be/4 chros73       0.00 K/s 3968.74 K/s  0.00 %  0.00 % rtorrent

This part seems to be good on my system.

[chros73]

I made one more test:

• switch off pieces preloading (during runtime): pieces.preload.type.set = 0
• all other settings are the same as was before (network buffer size is still huge)
• just uploading with around 1130 KB (no download), and using sudo iotop -aokP accumulated output to see whats the sum after 20 sec:

[Rate 1125.2/  2.6 KB] 
 it should be around: 20x1130 = 22600.00 K
 sum is (reported by iotop) =56868.00 K

It's about 2.5x .

Can you try the similar test with deluge if it has any buffering option? I'm just curious.

[chros73]

And the last test, testing buffer sizes, setting them during runtime:

network.receive_buffer.size.set =  208K
network.send_buffer.size.set    = 208K
pieces.preload.type.set = 0

• all other settings are the same as was before
• just uploading with around 163 KB (no download), and using sudo iotop -aokP accumulated output to see what's the sum after 20 sec:

[Rate 162.4/  8.8 KB]
 it should be around: 20x163 = 3260.00 K
 sum is (reported by iotop) = 13292.00 K

It's about 4x .

So, it seems that bigger buffer sizes can help a bit with this but not really.

This is a huge performance impact for rTorrent. ... and is likely impacting the entire userbase and they don't even know it.

I tend to think that you're right :) Now, all we need is somebody who can fix this. :)
Thanks for mentioning this bug as well.

[mrvn]

Why does it read ANYTHING during download? Unless you download more chunks in parallel than the system can cache all downloads should stay in memory until they are no longer needed.

Only when doing the final hash check when the download is finished I would expect reads, unless the downloaded file is still in cache (likely for a single download test smaller than the ram)..

So on that single download without upload what was the chunk size and how many chunks where downloaded in parallel? How much memory was there?

[ghost]

The system had 8G of RAM which was way more than the size of the torrent. Also the torrent, I don't remember the chunk size, there were a few dozen seeds the box was connected to. rTorrent was allowed to use 1G of memory for cache. There's no reason it should be reading anyways during downloading.

I'm wondering if rTorrent is trying to be so aggressive with it's caching that it's actually cache thrashing. Deluge does not have any of the same problems.

Debian 8 system, virtualized, all measurements were taken inside the VM.

[mrvn]

How does rtorrent caching work at all? If it's just mmap()ing the file and then uses that as buffer to recv() then every page will be read in before being overwritten with the download. That would explain as much read as download at least. I hope this isn't the case.

[chros73]

@mrvn As I mentioned before:

• I can not reproduce reading during downloading 1 torrent
• I can reproduce the 3x-4x reading issue during seeding

Can you also try it out?

[mrvn]

I have not benchmarked it but I have seen a lot more reads than my upload limit would suggest since basically forever.

[chros73]

Maybe this issue is related in this way: #409

[ryandaniels]

Reproduced downloading only 1 torrent with everything else stopped, on RaspberryPi 2 / 3 (running fresh install Raspbian/Debian Jessie). Tried all above config settings with no change.
Issue seems to be related to non-local disks.
Easy to detect, since downloading is being limited due to 100Mbit/s ethernet limitation of Pi. This really limits the RaspberryPi !

For reference, this is my up/down limit setting:
download_rate = 3400
upload_rate = 2000

When downloading to local disk on RPi, no issue. Hit limit set in .rtorrent.rc config.
iotop - 17:20:16 2877 be/4 rtorrent 0.00 B/s 3.39 M/s 0.00 % 0.00 % rtorrent
ifstat - 17:20:16 28893.09 724.55

When downloading to NFS share, the issue happens. (Also tested with CIFS (samba) and still same issue). Tons of excessive reads when downloading 1 file!
iotop - 17:18:42 2877 be/4 rtorrent 5.01 M/s 1774.55 K/s 0.00 % 77.63 % rtorrent
iotop - 17:18:42 2879 be/4 rtorrent 2.36 M/s 0.00 B/s 0.00 % 37.71 % rtorrent [rtorrent disk]
ifstat - 17:18:42 88057.85 16187.82

FYI, when copying file from local disk to NFS share, file transfer is fast. So not an issue with NFS share. (Hitting max of RPi ethernet).
iotop - 17:57:31 8058 be/4 rtorrent 11.31 M/s 11.49 M/s 0.00 % 78.37 % cp -rp /NFSshare/ISO/ubuntu-16.04.1-desktop-amd64.iso /home/rtorrent
ifstat - 17:57:31 94198.44 4390.52
iotop - 18:04:02 8197 be/4 rtorrent 12.60 M/s 12.37 M/s 0.00 % 86.25 % cp -rp /home/rtorrent/ubuntu-16.04.1-desktop-amd64.iso /NFSshare/ISO
ifstat - 18:04:02 2599.28 97872.17

Note: ifstat is in bit per sec. iotop is in bytes per sec.
Commands and command headers for reference:
$ iotop -botqqq
TIME TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
17:18:42 2877 be/4 rtorrent 5.01 M/s 1774.55 K/s 0.00 % 77.63 % rtorrent
17:18:42 2879 be/4 rtorrent 2.36 M/s 0.00 B/s 0.00 % 37.71 % rtorrent [rtorrent disk]

$ ifstat -b -t -i eth0 1
Time eth0
HH:MM:SS Kbps in Kbps out
17:18:42 88057.85 16187.82

[chros73]

Thanks for the detailed report.
Can you also test 3x-4x seeding issue with local disk? That was the one that I could confirm. Thanks

[ryandaniels]

Upload limit 200KB. 1 torrent.
About 2.75x when seeding from NFS disk:
About 1.75x when seeding from local disk.

NFS disk:
HH:MM:SS KB/s in KB/s out
08:50:27 464.41 208.26
08:50:28 628.22 277.58
08:50:29 628.28 214.16

TIME TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
08:50:27 11070 be/4 rtorrent 403.75 K/s 0.00 B/s 0.00 % 6.65 % rtorrent
08:50:28 11070 be/4 rtorrent 550.84 K/s 0.00 B/s 0.00 % 3.27 % rtorrent
08:50:29 11070 be/4 rtorrent 550.73 K/s 0.00 B/s 0.00 % 3.58 % rtorrent

Local disk:
09:20:25 7.04 229.83
09:20:26 6.51 218.40
09:20:27 4.59 240.26

09:20:25 11070 be/4 rtorrent 352.54 K/s 0.00 B/s 0.00 % 0.63 % rtorrent
09:20:26 11070 be/4 rtorrent 352.40 K/s 0.00 B/s 0.00 % 0.60 % rtorrent
09:20:27 11070 be/4 rtorrent 352.57 K/s 0.00 B/s 0.00 % 0.00 % rtorrent

Note: both iotop and ifstat in bytes

[chros73]

Thanks, that's indeed confirms this seeding bug with a local disk as well.

[chros73]

Can this be related? rakshasa/libtorrent#83

[chros73]

Has anybody tried out this with master branch version?

[chros73]

I made test about the seeding issue again with local disk using 0.9.6: [Rate 2100.5 / 7500.6 KB] ... [U 65/300]
pidstat -p $PID -dl 20 5 (run it 5 times, gives average at every 20 secs, and after the last run)

Note:

• downloading isn't affected
• ratio depends on the used upload slots: the more slot is used the more unnecessary read is made

I. if pieces.preload.type.set=0 or pieces.preload.type.set=2

19:09:31      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
19:09:51     1000     23536  11553.82   7841.68      0.00  /opt/rtorrent/bin/rtorrent
19:10:11     1000     23536   9645.20   7442.00      0.00  /opt/rtorrent/bin/rtorrent
19:10:31     1000     23536   9087.00   6860.80      0.00  /opt/rtorrent/bin/rtorrent
19:10:51     1000     23536   9763.00   7477.20      0.00  /opt/rtorrent/bin/rtorrent
19:11:11     1000     23536   9218.80   6956.40      0.00  /opt/rtorrent/bin/rtorrent
Average:     1000     23536   9853.73   7315.67      0.00  /opt/rtorrent/bin/rtorrent

It's 4.6x more then it's needed.

II. if pieces.preload.type.set=1

19:07:31      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
19:07:51     1000     23536  14749.60   6930.60      0.00  /opt/rtorrent/bin/rtorrent
19:08:11     1000     23536  15739.80   7864.20      0.00  /opt/rtorrent/bin/rtorrent
19:08:31     1000     23536  16461.80   7450.60      0.00  /opt/rtorrent/bin/rtorrent
19:08:51     1000     23536  15243.00   7157.60      0.00  /opt/rtorrent/bin/rtorrent
19:09:11     1000     23536  15412.80   7462.20      0.00  /opt/rtorrent/bin/rtorrent
Average:     1000     23536  15521.40   7373.04      0.00  /opt/rtorrent/bin/rtorrent

It's 7.4x more then it's needed.

How can we debug this?

[chros73]

Has anybody tried out this with master branch version?

I checked, it suffers from the same issue.

... ratio depends on the used upload slots ...

It doesn't, as it turned out: I checked the same instance at a different time and I got better results: [Rate 2100.5 / 7500.6 KB] ... [U 162/300]

12:31:59      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
12:32:19     1000     23536   3342.60   8174.60      0.00  /opt/rtorrent/bin/rtorrent
12:32:39     1000     23536   3854.00   7937.60      0.00  /opt/rtorrent/bin/rtorrent
12:32:59     1000     23536   2982.80   7713.40      0.00  /opt/rtorrent/bin/rtorrent
12:33:19     1000     23536   2559.00   7169.80      0.00  /opt/rtorrent/bin/rtorrent
12:33:39     1000     23536   2736.20   8156.40      0.00  /opt/rtorrent/bin/rtorrent
Average:     1000     23536   3094.92   7830.36      0.00  /opt/rtorrent/bin/rtorrent

What interesting is the output of dstat -ctdD sdb:

----total-cpu-usage---- ----system---- --dsk/sdb--
usr sys idl wai hiq siq|     time     | read  writ
  8   4  63  17   0   8|27-08 12:34:07|2928k   21M
  2   1   5  86   0   6|27-08 12:34:08| 256k   61M
  6   4  43  45   0   3|27-08 12:34:09|1444k   62M
  3   2  46  43   0   6|27-08 12:34:10|1104k   89M
 12   5  55  24   0   3|27-08 12:34:11|7080k   18M
  7   3  78   8   0   4|27-08 12:34:12|2188k  176k
  5   2  83   5   0   5|27-08 12:34:13|2304k    0
  6   1  74  15   0   4|27-08 12:34:14|2816k    0
  6   3  81   6   0   5|27-08 12:34:15|2436k    0
  6   2  83   5   0   5|27-08 12:34:16|2448k    0

As we can see, writing incoming data is buffered but not reading.

Somebody has to take a look at this issue who is smarter than me. :) @rakshasa ? :)

[ghost]

It sounds/looks like rTorrent is reading larger chunks than needed for each piece request from each seed slot. That would suggest the problem is actually in libtorrent.

[chrcoluk]

Just want to chime in here, I was looking into this myself I disabled the preloading feature, and it barely had an impact, but I then noticed data was been read in 32k chunks by the daemon, but of course the default read_ahead in the scheduler is 128k, as soon as I dropped that down to 32k on the data drives I now have no read amplification. I hope this helps.

[AlexKMDev]

@chrcoluk can you please explain what needs to be changed, what scheduler do you mean?

[colinhd8]

@chrcoluk can you please explain what needs to be changed, what scheduler do you mean?

+1

[chrcoluk]

On the OS kernel scheduler.

check the value here /sys/block/sdb/queue/read_ahead_kb

substitute sdb for the drive label on your system.

[amlopin]

On the OS kernel scheduler.

check the value here /sys/block/sdb/queue/read_ahead_kb

substitute sdb for the drive label on your system.

Has anybody checked this? Does it work? Thanks

[eagle1maledetto]

Has anybody checked this? Does it work? Thanks

Yep, tested and not worked for me. Same read-amplification, crazy load average and a ton of I/O Wait

[ellidi]

Has anybody checked this? Does it work? Thanks

Yep, tested and not worked for me. Same read-amplification, crazy load average and a ton of I/O Wait

Same here. Or, the load average isn't bad, but reading from the disk being uploaded from is ruined for other programs.

[chrcoluk]

It could be my way of measuring is bad, I measured the amplification via proxmox (as my machine is a VM), and I had observed after decreasing the i/o read level, on the host machine dropped to about a 1.1 multiplier instead of around 4.6.

Or it also could be you guys have different causes, I will try and do more monitoring and testing and get back after.

[ellidi]

My DL drive is unusable while rtorrent is uploading faster than 1MB/s.
Is it time to abandon ship?

[defkev]

I can confirm @chrcoluk's finding's that lowering the readahead to match rtorrent's chunk size normalizes disk read.

Tho i am using a network storage (over SMB3) instead of a local disk on a 10GBit link as the backend so change rsize in /etc/fstab to 32k (rsize=32768) on the client.

I did notice this after limiting the bandwidth on the rtorrent attached switch port which rendered the rtorrent server sluggish and unresponsive after running into the burst limit will uploading.

Why ever this only affects uploads i have no idea.

Metrics were taken directly from the storage NIC so the measurement is fairly accurate.

[runaroundhome]

Anyone know how to make the read_ahead_kb changes on a RAID system?
Because the RAID drive, of which rtorrent reads/writes from is md2, which has a very high number compared to the RAID drives.

[sedago]

@runaroundhome
Did you manage to get rid of the problem?

[runaroundhome]

@sedago, nope, still got the issue. I switched over to qbittorrent, but it is much less aggressive at seeding/downloading compared to rtorrent. So I'm now running both about 20/80, with a total of ~600 torrents (20 being rtorrent). It seems to work kinda ok.

I feel like we are missing something, because allot of the big seedboxs around are still offering rtorrent, and they'll have 100s if not 1000s of torrents running at the same time, can't say they're just chucking SSDs at the issue.
Or maybe they've just managed to implement the read_ahead_kb workaround mentioned here. Or have fixed the issue in rtorrents code, but don't share it.

[sedago]

@runaroundhome
It really does not make sense, I think many seedboxes dedicated individual HDDs to users. Thanks for the update though!

[IvanShift]

Hello!
Set read_ahead_kb to 32 doesn't help for me. Is there some other options to change and test?

[sun1ight]

10Gbe up-link, i7-8700, XPEnology DSM 6.2.3, Btrfs RAID5 8x Ent. 6tb HDD 7200rpm with RAID0 3x1Tb SSD read cache, various popular docker containers with rtorrent+rutorrent gui. Maximum download and upload speed that I saw - 35-45Mb/s.
qBittorrent in a nearby docker container on the same server is capable of 250Mb/s+

PS: change read_ahead_kb didn't help me.

[zilexa]

The documentation send me here as it says:

Reduce disk usage
In theory, we can reduce disk i/o :) See this issue why: #443

I read through this issue.. but don't really see the solution. My goal: I have 32GB RAM, only 8-9GB is used in total, sometimes 12GB, excluding torrent client.
I would like rTorrent to use 8-10GB of RAM for all active downloads in total, before writing data to disk. Is this possible?
I use Ubuntu 20.10 and run everything in Docker.

WIth Transmission, setting disk cache to 8GB, I monitor via Netdata and notice it never uses more than 2GB. After little over 2GB, it writes to disk. Tested with a single 26GB torrent file (downloading) and everything else (downloads&seeds) on hold.

The documentation provides a few clues by explaining settings, but I don't see a clear path to get rTorrent to use that much RAM before writing to disk. Hope someone can be helpful here as I don't like Transmission and would like to switch.

I am a bit afraid to change network.receive_buffer.size.set to 8GB for example and network.send_buffer.size.set to 4GB as I do not see anyone writing about such high buffer sizes..

My internet speed is 500/500 but I am limiting my torrent client to 160/84 megabit (20/12MB).

[ghz-max]

@zilexa If you are under linux, you can play with the following options with sysctl

vm.dirty_background_ratio = 5 # you start to write to disk as soon as 5% of dirty cache
vm.dirty_ratio = 90 # you stop to accepting new writes until dirty cache is below of 90% of dirty cache

[IvanShift]

@zilexa
I set max_memory_usage to 8GB but it doesn't help. No changes in disc usage.

[jamiew0w]

hi @rakshasa

I understand you're busy, did you happen to have time to look into this issue or have any reccomendations with regards to a permanent fix?
rtorrent is fantastic but it'd be great to finally be rid of this bug.

[jamiew0w]

hi @rakshasa

we're rapidly nearing 6 years of this issue being open and from reading this issue, we don't really know what your plans are.
if you're not planning on fixing it, could it please be marked wont-fix or similar?

i think we can all appreciate your work through the years (i certainly do) and I really do hate to badger you, however it feels like i/we have no other option. you owe us nothing, but i think i can speak for everyone when i say that some communication on this would be hugely appreciated.

for example, if you need help with a certain issue it might spur people to contribute. if it's a monetary issue, people might donate. the silence really is deafening though - please do respond!

[andreygursky]

could it please be marked no-fix (or similar) and closed?

If an issue can be fixed but is not yet fixed, it should stay open. If the author cannot for whatever reason fix the issue, the issue can just be marked with label "help needed" / "pr welcome" or similar and that's it.

[Antorell]

I fixed the issue on my seedbox by switching to madvise instead of direct paging. Dell Virtual disk Raid 0 HDD drive.

Why I wanted to post was more as a warning after reading people wanting to experiment with read_ahead_kb. If you change the default value (to something too big?), you may end up with rtorrent misreporting uploaded data/ratio to the tracker. Don't think you can ratio cheat with it, the reported value/data is less than the actual uploaded data.

If you want to toy around with read_ahead_kb, you can decide what value to use for the setting/buffer using this command:

for V in 128 256 512 1024 2048 4096 8192 16384; do echo $V; echo $V > /sys/block/sda/queue/read_ahead_kb && hdparm -t /dev/sda | grep "Timing"; done
source: https://forums.gentoo.org/viewtopic-p-6798722.html?sid=f8eab13f1a18ccdc8add903ab70133d4#6798722

The command will leave read_ahead_kb with the last used value, 16384, so watch out for that.

[lps-rocks]

Can anyone confirm if the two PR's opened by @stickz do actually make a difference when the minimum speed set for preloading is higher than the slower seeds?

[jazha]

I'm not an expert at all, but I found this thread that I think could be atleast slightly relevant to the issue:

https://qbforums.shiki.hu/viewtopic.php?t=9243

The user describes the same type of problem, and fixes it by updating/reinstalling his libtorrent(with a newer version I believe).

I did not test this myself.

[lps-rocks]

I'm not an expert at all, but I found this thread that I think could be atleast slightly relevant to the issue:

https://qbforums.shiki.hu/viewtopic.php?t=9243

The user describes the same type of problem, and fixes it by updating/reinstalling his libtorrent(with a newer version I believe).

I did not test this myself.

That's a different client and libtorrent library that has nothing to do with rTorrent/libtorrent-rakshasa - There are 3 different libtorrent libraries that exist, the one that qBittorrent uses is https://www.libtorrent.org/

[jazha]

Thats too bad, thought maybe some of the information in the thread could be used, as it seems to be the same type of issue to me. Guess we'll just have to wait for a fix to come through here.