Settings for min 450mbit upload to 5 peers and only seed each piece once

[Falcosc]

Currently we use a huge cluster of servers to distribute our files to all mirrors in a client-server model. In average we have 450mbit/s of new data to distribute to all mirrors. And our bandwith requirements gets multiplied by the number of mirrors.

All mirrors have at least as many bandwidth for uploading as for downloading which gives us a great opportunity to test a p2p distribution model.

How could we get the best seeding performance if we want to use only a single host to seed the whole stream of new data?

First Goal: don't send data twice

Is it possible to combine strict_super_seeding with share_ratio_limit 1? And does super seeding still work if we use suggest_mode = suggest_read_cache?

Second Goal: distribute as fast as possible

Peers with a lot of bandwidth should be prioritized. I guess seed_choking_algorithm = fastest_upload would do this job

Third goal: very high upload rates per peer

How to calculate max_allowed_in_request_queue to achieve at least 450mbit/s across 1 to 5 peers?

Fourth goal: manageable disk I/O

We don’t want to get limited by disk performance, how should we setup the cache to deliver only to 5 peers? I take a look at the high performance seed settings pack, but it is optimized for a huge amount of slow peers instead of a very small amount of gigabit peers.
I guess we need to tweak the following settings
cache_size
use_read_cache
cache_buffer_chunk_size
read_cache_line_size
write_cache_line_size
send_buffer_low_watermark
send_buffer_watermark
send_buffer_watermark_factor
use_disk_cache_pool

I don’t understand the support_share_mode option, is it useful for our usecase?

I hope some of you have time to help us to get a configuration which is useful for our use case.

Kind Regards
Falco

[arvidn]

1. you want to use "super seeding" mode. libtorrent has two such modes, one strict and one relaxed. In the strict mode, a peer does not get another piece until the last piece that was sent to it has showed up at some other peer (demonstrating that it's not just leeching, but in fact sharing its pieces). The strict mode conflicts with goal 2 of uploading fast, so I would not recommend it.

2. You want to make sure you have large enough send buffers. you probably want to set a very large unchoke slot limit (to essentially not choke peers). The choking algorithm doesn't matter in that case, but if you would like to keep some peers choked, yes, fastest_upload is what you want.

3. My experience is that this is very difficult to achieve without controlling the downloading peers. Many, if not most, clients limit the number of outstanding requests they have to something that's appropriate for a normal home internet connection. If you have a fast connection, you need to keep a lot more requests outstanding to fill the bandwidth-delay-product. As the uploader, you can't make the client request more pieces. The only way you can control the upload rate is by (as much as possible) cut down on the latency responding. This could be achieved by keeping a very large disk cache in RAM for instance, or seed off of SSDs.

4. this goal conflicts a bit with the first one, since the best way to have scalable disk I/O is to get as much use out of each read as possible. i.e. send the data to as many peers as possible to amortize the cost of the disk read.

support_share_mode just controls whether the peer advertises support for share mode. share mode is a mode where a peer can passively participate in a swarm with the sole purpose of uploading more than it downloads (i.e. help out with distribution). I don't think this applies to your use case.

The best way, I think, to get a good understanding of which bottlenecks you're hitting and which knobs to turn, is to have good monitoring. in libtorrent you can enable stats logging, which posts alerts that you can print to a file and then parse with a python script under tools. This is one way to get a good understanding of peers' state and disk I/O performance.

[Falcosc]

Thank you very much for your answers, I appreciate it.
Our peers are all around 1Gbit/s and we want to distribute only 1-2 new file at same time managed by a queue on the master seeder (queue order based on peer count, because we have peers who want everything and peers who only want some files).

A little bit more about the purpose of the first goal:
For our master seeder we would reduce the load of old file requests. We want to priories new unknown files and don't want that it get disturbed by other requests.
Is there a way to priories it or do we need to stop torrents with scripts based on seeder count in swarm?
I like the idea of the share_ratio_limit but what does happen if the limit is reached? Does it stop the torrent completely or does it get a low priority?

libtorrent/src/torrent.cpp

Line 8395 in 192ef49

int torrent::seed_rank(aux::session_settings const& s) const

Looks like a priority handling but I don’t understand how the priority is handled? Based on connections or based on bandwidth or based on something else?
If we want to use a very low share ration limit like 1 or 1.5, do we need the strict super seed mode to prevent a prioritization drop of an uncomplete seed? Because no disturbances by old file request are more important than speed for a single file. We already have the problem of old file requests in our server-client based distribution.
What is a large enough send buffer and unchoke slot limit?

Kind Regards
Falco

[Falcosc]

Now I guess I understand the selected settings.

[arvidn]

please feel free to contribute documentation patches if you find improvements!

[Seeker2]

Falcosc said:

Third goal: very high upload rates per peer

arvidn said:

If you have a fast connection, you need to keep a lot more requests outstanding to fill the bandwidth-delay-product.

The libtorrent implementation of uTP seems unable to do this:
caisan/libtorrent#581
qbittorrent/qBittorrent#1387
qbittorrent/qBittorrent#4284
https://sourceforge.net/p/libtorrent/mailman/message/35568969/
https://qbforums.shiki.hu/index.php/topic,3982.msg24170.html#msg24170

Deluge using the ltConfig plugin can allow TCP peer/seed speeds to be decent without fully disabling uTP by doing what's described in this:
caisan/libtorrent#546
...but qBitTorrent can only disable uTP entirely.

For partial compatibility with uTP under Deluge with ltConfig plugin, you might allow uTP incoming and TCP incoming and outgoing.

As for very large disk cache in RAM, both Deluge and qBitTorrent 32bit versions will crash if their cache is over about 1400 MB even with very few torrents running. (So long as the torrents are active of course!)
Each 100 MB cache size increase in qBT will increase RAM use by qbittorrent.exe by about 120 MB.
If you don't have the cache set too large, you can also raise cache duration to 600 seconds (or more for Deluge).

[Falcosc]

Thanks, I will disable uTP as workarround because such high rates are only needed for a closed network.

We do have the 32bit in our documentation, too. The problem is that our performance documentation is split into 3 places:

• tuning page
• high performance settings pack comments
• settings pack documentation

@arvidn would it be ok to create references between the 3 documents? Or which of them should contain all information?

[arvidn]

I think it makes sense to keep the reference documentation as the main source of truth of the API. including documenting all the settings of settings_pack. Of course it would be good to link to the reference documentation from the tuning page. The tuning page is meant to collect all aspects of performance tuning. The high-performance settings preset could easily have its documentation moved, but I don't think it's necessarily very useful. The tuning page should attempt to describe the mechanisms in play that can cause bottlenecks

[Falcosc]

@Seeker2
I did read them all and now I have two question.
Why did you suggest disabling utp, doesn't mixed_mode_algorithm prefer_tcp work in this case?
Is it correct that we didn't have this issue at 1.0.x at all or is it only less slow there?

Would be nice to upload fast to utorrent clients. I did wonder why I have so many slow connections at year 2017.

But I am not sure if I could realy confirm this old bug. Currently I have a Deluge uTP connection with 500kbyte/s. I will have a look at fast peers from time to time.

@arvidn do we already have a ticket for the utp speed issue? Because I didn't find one. Or does we already have a confirmed fix?

[Seeker2]

Partial support for slow uTP peers is better than none, all things considering.
But the only way to do that and NOT cripple all TCP peers is mixed_mode_algorithm prefer_tcp.
A LAN-local uTP peer may not even run 1 MB/sec due to continuous packet failures and retransmits.
I saw roughly 10% failure rates in my tests.

If qBitTorrent/Deluge/libtorrent strongly favors making uTP peer connections over TCP peers, then it's a loss to try to partially use uTP peers because nearly every TCP peer will be auto "promoted" to being a uTP peer anyway.

The issue was in libtorrent 1.0.x as well ...to a lesser but still considerable extent, as I mentioned here:
qbittorrent/qBittorrent#4284

[Falcosc]

If qBitTorrent/Deluge/libtorrent strongly favors making uTP peer connections over TCP peers, then it's a loss to try to partially use uTP peers because nearly every TCP peer will be auto "promoted" to being a uTP peer anyway.

I don't understand this, why should they strongly favors making uTP peer connections over TCP? I mean, does a libtorrent client use tcp if seeder does set mixed_mode_algorithm prefer_tcp?

By the way, I do currently seed to a transmission peer over uTP with 800kbyte/s

[Seeker2]

Unless you've nearly saturated your upload bandwidth and/or the destination peers are slow/overloaded, you should be able to upload to faster uTP peers at >4,000 KiloBYTES/second each.

What's the top speed you've seen going to TCP peers?

[Falcosc]

Now I am confused.

you should be able to upload to faster uTP peers at >4,000 KiloBYTES/second each.

Does not match to your other comments

The libtorrent implementation of uTP seems unable to do this

And

A LAN-local uTP peer may not even run 1 MB/sec due to continuous packet failures and retransmits.

@Seeker2 Could you describe what is the difference between your last and your other comments is?

Thanks

I could not tell if there is a problem because my testhost has not enoght bandwith. I just refer to your comments and those does not match which does confuse me. And I did already manage to get higher uTP rates then you mentioned in the tickes. Is the uTP bug maybe already fixed?

[Seeker2]

Some versions of qBT and/or some conditions with qBT can allow uTP to exceed 1 MB/sec. I did see ~4 MB/sec briefly in early tests on older versions of qBT. So if you see better than that, it might allow more testing to narrow down the issues.

Sadly, your 2 examples:
"a Deluge uTP connection with 500kbyte/s."
"currently seed to a transmission peer over uTP with 800kbyte/s"
...Are both examples of <1 MB/sec to/from a single uTP seed or peer.

So if you get >4 MB/sec (or at least >1.5 MB/sec) to/from a single TCP peer or seed, then uTP may be working far worse than TCP.

[Falcosc]

Thanks for clarification. I did expect this kind of an answer :)

My test host is proberly to slow to get hit by the uTP bug if it's still there.

[Seeker2]

The other possible part of the uTP bug was the 10% retransmits and resulting speed-crippling effects that could have on TCP peers as well.

[arvidn]

would someone mind providing a summary of "the uTP bug"? Is it primarily a problem of sending or receiving?
If someone can reproduce it in a controller environment (i.e. with a single peer), it would be really helpful to get a uTP log. That requires rebuilding libtorrent with uTP logging enabled though (define TORRENT_UTP_LOG_ENABLE when building libtorrent, and each time you run there should be a utp.log written in the current working directory, so make sure that's writeable).

It doesn't have to be a long run, just long enough to demonstrate the performance issue.

[arvidn]

it might be worth to start a new topic/ticket for it actually

[Seeker2]

Deluge 1.3.13 could download without any uTP retransmits by uT 2.2.1 at speeds up to about 160-224 KB/sec. Then it started having progressively more uTP retransmits by uT 2.2.1 as speed increased and was unable to go faster than ~1100 KB/sec. Maybe 10-15% loss at worst.
qBT 3.3.10 was sometimes showing uTP retransmits even at 16 KB/sec! Retransmits much worse than Deluge at least up to ~500 KB/sec. About the same as Deluge on retransmits past about 800 KB/sec. Didn't seem to want to go higher than 800 KB/sec except as a brief spike up to maybe ~1200 KB/sec. Averaged closer to 600-700 KB/sec. No improvement with qBT 3.3.11...same behavior.
Tixati v2.53 had very low uTP retransmits from uT 2.2.1 (usually 0), but was very slow--in uT's no limits "firehose mode", Tixati's DL rate slowed below 50 KB/sec and only averaged about 30 KB/sec. Doesn't even run stable over 1000 KB/sec using TCP with limits!

[arvidn]

@Seeker2 thanks for having looked at this. This weekend I tried to reproduce this issue. I ran the latest version of uTorrent (Mac) seeding a large file over loopback to client_test (in libtorrent, RC_1_1 branch) and to transmission Mac.

client_test climbed to ~70 MiB/s and stabilized just above 50 MiB/s (I suspect disk I/O causing the fluctuations)
transmission did not get past 4 MiB/s

Are you suggesting that what you're seeing is a regression since libtorrent RC_1_0? Should I be trying with 1.0 too?

[Seeker2]

"Are you suggesting that what you're seeing is a regression since libtorrent RC_1_0? Should I be trying with 1.0 too?"

I thought as much earlier, but I retested a bunch using quite a few Deluge (v1.3.5 to v1.3.14) and qBitTorrent (v2.9.0 to v3.3.11) versions and only uTorrent 2.2.1 to them consistently seemed affected.
I've tested with libtorrent v1.1.1.0 as well (near the last post): qbittorrent/qBittorrent#4284
Others with similar slow uTP results: https://www.reddit.com/r/torrents/comments/42sgoz/qbitorrent_issues_utp_and_seeding_over_lan/

"client_test climbed to ~70 MiB/s and stabilized just above 50 MiB/s (I suspect disk I/O causing the fluctuations)"

There seems to be a triggering event, as only uT 2.2.1 uploading TO qBT is consistently slow, even qBT to uT tends to be immensely faster -- in excess of 30 MB/sec.
Deluge to/from qBT sometimes even slows to nothing randomly. I'm at a complete loss, as it's not something that happens often. Deluge 1.3.12.1 regularly disconnected with qBT 3.1.0, but Deluge 1.3.14 with qBT 3.3.11 seemed more stable.
This could be a negative synergy of Windows + ramdrive + multiple BT clients working over local loopback rather than anything intrinsically wrong with libtorrent, at least as far as the problems with qBT <-> Deluge are concerned.
Windows file caching going haywire at extreme speeds?

It's important to force the use of uTP in the tests, as when qBT allows uTP it doesn't stick with uTP sometimes.

"transmission did not get past 4 MiB/s"

Does the latest Transmission even have uTP? ...because the version I tested (v2.84) didn't use uTP.

Unrelated: A special case test using 16 KB piece size despite being a ~200 MB single file torrent seems to knock the speeds down a bunch. Everything was affected. The worst, Tixati started flailing and averaged <50 KB/sec.
Multi-file torrents with poor offsets (not on 16 KB boundaries) might be even worse.

[EDIT:] uT seems to be using a different packet size than libtorrent (in qBitTorrent and Deluge):
uT 2.2.1 uTP to qBT 3.3.11 = 1438 byte packet size with ~20-29 byte (ACK?) reply packets
qBT 3.3.11 uTP to uT 2.2.1 = 1472 byte packet size with ~20-37 byte (ACK?) reply packets
uT 2.2.1 uTP to Deluge 1.3.14 = 1438 byte packet size with ~20-59 byte (ACK?) reply packets
qBT 3.3.11 uTP to Deluge 1.3.14 = 1472 byte packet size with ~20-37 byte (ACK?) reply packets
Deluge 1.3.14 uTP to uT 2.2.1 = 1472 byte packet size with ~20-37 byte (ACK?) reply packets