Abandoning collected torrent files

[vi]

In this comment of other issue whirm supposed the complete removal of torrent collection.

I think there should be a separate issue on Github for discussing this change.

[vi]

@whirm, Why there is collected_torrent_files in the first place? Is it a performance optimisation?

Do I understand correctly, that in

[magnet link] -> resolving -> [torrent file] -> downloading

scheme, if we can download, then we also can resolve; so if we can't resolve, we also can't download, therefore the torrent is "dead" and not worth be kept in the database anymore?

[LipuFei]

@vi Torrent collecting is mainly used by SearchCommunity. (Here is my understanding, it may not be correct) SearchCommunity uses it to popular your torrent database by collecting torrents so that you can search your local database for the torrents you want.

[vi]

What is a search community? A search feature in Tribler? A community of Tribler users who like searching?

[synctext]

The search community is our mechanism for "content discovery". This is the mechanism that makes websites redundant. We could start to gossip only magnet:// links and rely on BEP9 for torrentcollecting on-demand. But this would make things slower, we instead want to push things even further and move to a Youtube experience.

[synctext]

Just to document my thoughts at this point.

Would remove a lot of code. This is a big item on the roadmap and could dramatically improve Tribler performance. Every peer could locally store easily the metadata of 250000 swarms! However, when doing a search the penalty must be paid. Then you click on content and the .torrent must be fetched via BEP9.

This enables a dramatic improvement to channels. Currently channels do not scale well in number of items. Channels with over 15000 torrent are probably slow to sync. Due to the performance of the Bloom filters it should start to slow down. However, no measurements exist if this is really true.

Another point to consider is anonymous seeding. This might be especially sensitive to the .torrent fetching and precious few connectable peers. Possible to use the Rasterbar Libtorrent Merkle has feature to reduce .torrent size by removing all hashes.

We still have 2x DHTs and 2x .torrent fetching implementations. Libtorrent needs an API extension to just do DHT lookup or .torrent fetch.

[synctext]

Other issues also discuss Dispersy channel scalability plus #2039. Perhaps use the connection_tester of Libtorrent.

@xoriole
magnet-based channels
2018 edition. Scale to 40k magnet links, using the power of Libtorrent.
Scale to 40k magnet links (roughly 250 bytes each) per channel == 10MByte text file of Magnet links.
Use the Libtorrent core to sync channels, instead of Dispersy Bloom filters.
Each channel has 1 owner and 1 dynamic hash. Each new item added to the channel triggers a hash change. The channel is a text file with magnet links and item name.
Libtorrent can check hashes and efficiently re-seed a file with only a few modified bytes. When only a single line is added to a text file, the torrent block hashes should all remain the same, with only one exception.

ToDo: Python stand alone experiment with Libttorrent and Twisted for magnet-text-file to replace Dispersy channels. Key point is having full-speed download of channels. Download metadata at full saturated Internet Link speed. Test if Merkle hashes usage is useful or not.

[qstokkink]

This has grown into a duplicate of #3615 .

[Dmole]

My Tribler maintenance looks like this;

rm -r .Tribler/{collected_*,dlcheckpoints,logs,sqlite/tribler.sdb,sqlite/dispersy.db}

Which is just silly.

[ichorid]

Solved by #4090

[synctext]

It has been 4 hard years.

Closing a Jan 2015 issue. Quite profound that this ticket has been closed after all these years. We ran out of money, but found funding eventually. A milestone we almost never made. After all the hard and tedious work of the Tribler team the refactored code is now in excellent health. No more decode.py, timeline.py, and unsustainable AllChannel growth.

Now we will no longer need to make disruptive changes. No more framework changes, but instead stable Jenkins jobs. We have solid foundations and can focus mostly on the higher layers: trust, crowdsourcing, tokens, and our own micro-economy. Hopefully.

[vi]

Does all that micro-economy token thing mean that one will eventually need to store terabytes of blockchain and keep network on just to keep pace, like in Bitcoin?

Or there are mechanisms to prevent unlimited blockchain growth and/or ability to use Tribler just as a smarter Torrent client/database, without that micro-ecomony (albeit at maybe reduced efficiency).

[devos50]

@vi valid question. We use a blockchain that is fundamentally different from the blockchain as used by Bitcoin or Ethereum. Tribler selectively stores relevant records and avoid full network replication of all records ever created. You can find the details of our academic work in this paper.