Swarming effect in anonymous downloads: unit tests, performance analysis, and improvements

[synctext]

Swarms of thousand of peers are commons in the Bittorrent ecosystem. Numerous scientific studies investigated the swarming effect in relation to Internet locality.

Connecting to dozens of hidden seeds should be reliable and fast, as in the non-privacy case:

The current code includes anonymous Peer EXchange and a known security issue.

problem
No effective unit tests and performance analysis have been conducted of this feature. Swarming needs to be analysed, bottlenecks needs to be identified and removed. A key first step is creating a unit test where a swarm with 20 hidden seeders is created. An anonymous downloaders should see at least 80% of the anonymous swarm participants within 60 seconds.

A unit test is the primary issue, this should show the key issues. One possible outcome is that both PEX and DHT need improvement. Currently we use our own Python-based DHT implementation. Possible improvement is to find the first peers in a hidden swarm by using and expanding the Libtorrent API with DHT peer discovery for a given hash in combination with our own already implemented tunnel community PEX messages. However, benchmarks would first need to be made to determine if the Libtorrent DHT + torrent collecting has superior performance.

[synctext]

The latest V6.5 RC7 code can handle a lot of connections via exit nodes. That seems OK, 44 connections in picture below, giving 1.5 MByte/sec.

[synctext]

The n debug window shows nice PEX details:

16:52:39]: Extended circuit 161863140
[16:52:39]: DHT lookup for info_hash 64e895 resulted in peers: set([('82.193.98.125', 63096)])
[16:52:39]: Request key for info_hash 64e895 from ('82.193.98.125', 63096)
[16:52:41]: DHT lookup for info_hash 332270 resulted in peers: set([])
[16:52:50]: Joined an external circuit 2626678441 with 95.211.155.142:50226

[lfdversluis]

https://github.com/Tribler/pymdht does this dht do the PEX? Then I can build a test around this, rather than whole tribler

[rjruigrok]

@lfdversluis nope. The only PeX in the tunnel community is in the key-request and key-response which is used for setting up a hidden service. It's a bare, untested implementation written by me on a rainy sunday afternoon (see the commit mentioned by johan). It just takes a bunch of known candidates and appends them to the payload of key-request/response message. So only hidden services will benefit from this kind of pex. Maybe pex can be improved by doing pex in the ping/pong message?

[lfdversluis]

@rjruigrok Thanks for your explanation. Currently I am still stabilizing the devel branch with @devos50. once that branch is stable, I will start focusing on this topic among other things. Can I shoot you a question from time to time if I encounter any (maybe related to your implementation)?

[synctext]

@lfdversluis forgot about this one. Still, please focus on the async IO topic till end of July.

[lfdversluis]

Will do. I hope I can give some love to this though :)

[synctext]

#3183 #109

[qstokkink]

Seeing as this issue calls for unit tests and performance tests, I think it can now be closed. We've had performance tests for years, unit tests for even more years and we've even had huge performance improvements to the anonymous download speed already.