Explore the compatibility of Tribler with libtorrent 2

[devos50]

It seems that libtorrent 2, with support for the BitTorrent 2 protocol was released a week ago. Unfortunately, the new protocol is not compatible with BitTorrent 1.x, due to the transition from SHA-1 to SHA-256.

The key changes are described in this blog post.

We should explore how much time it would take to integrate the new version of libtorrent in Tribler. Since there are quite some changes, I'm expecting that this will not be an easy task.

[ichorid]

The most important change for us is BitTorrent V2 requiring 32 bytes infohashes. This will require introducing a new format for Channels entries and changing all the relevant hardcoded "20 bytes" checks in our codebase.

Gonna be a headache...

[da2x]

For what it’s worth, Tribler is compatible with the v1 portion of BitTorrent v2+v1 compatible torrent files.

[drew2a]

Python binding that probably works without any additional tuning: https://pypi.org/project/libtorrent

[drew2a]

One of the "hidden" downsides of the migration to libtorrent 2.0 is the following: we actively use static environment for our infrastructure (builds, tests, experiments), which means such migration will touch not only codebase, but it will require a massive manual upgrade of affected machines.

(maybe it is also a good point to switch from Jenkins to Travis (or similar) as well)

[drew2a]

I tried to install libtorrent 2.0.5 and got the following error:

[PID:84481] 2022-03-30 14:05:01,511 - ERROR <tunnel_community:129> TriblerTunnelCommunity._poll_download_manager(): Error on polling Download Manager: No Python class registered for C++ class libtorrent::flags::bitfield_flag<unsigned char, libtorrent::connection_type_tag, void>
[PID:84481] 2022-03-30 14:05:01,711 - ERROR <rest_manager:57> tribler.core.components.restapi.rest.rest_manager.error_middleware(): No Python class registered for C++ class libtorrent::flags::bitfield_flag<unsigned char, libtorrent::connection_type_tag, void>
Traceback (most recent call last):
  File "/Users/<user>/Projects/github.com/Tribler/tribler/src/tribler/core/components/restapi/rest/rest_manager.py", line 50, in error_middleware
    response = await handler(request)
  File "/Users/<user>/Projects/github.com/Tribler/tribler/venv/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/Users/<user>/Projects/github.com/Tribler/tribler/src/tribler/core/components/libtorrent/restapi/downloads_endpoint.py", line 293, in get_downloads
    "availability": state.get_availability(),
  File "/Users/<user>/Projects/github.com/Tribler/tribler/src/tribler/core/components/libtorrent/download_manager/download_state.py", line 195, in get_availability
    peers = self.get_peerlist()
  File "/Users/<user>/Projects/github.com/Tribler/tribler/src/tribler/core/components/libtorrent/download_manager/download_state.py", line 222, in get_peerlist
    return self.download.get_peerlist()
  File "/Users/<user>/Projects/github.com/Tribler/tribler/src/tribler/core/components/libtorrent/download_manager/download.py", line 529, in get_peerlist
    'connection_type': peer_info.connection_type,
TypeError: No Python class registered for C++ class libtorrent::flags::bitfield_flag<unsigned char, libtorrent::connection_type_tag, void>

[drew2a]

The history of rolling back from libtorrent 2.x to libtorrent 1.2.x: qbittorrent/qBittorrent#17545

[xoriole]

Tried libtorrent 2.0.9 on the main branch. It works without any stacktraces. However, the memory consumption when downloading a torrent seems to linearly increase over time. Here is a screenshot of the Debug Pane.

The torrent that is being downloaded is Ubuntu 20.04. The download speed under 1 hop anonymity was between 1-2 MB/s.

[xoriole]

Still libtorrent 2.0.9.0. This is what happens when the download completes. It frees up the memory.

Here is a relevant section on Libtorrent documentation Disk I/O overhaul

In libtorrent 2.0, the default disk I/O uses memory mapped files, which means a lot more of what used to belong in the disk caching subsystem is now handled by the kernel. This greatly simplifies the disk code and also has the potential of making a lot more efficient use of modern disks as well as physical memory.

Comparing the same downloads with libtorrent 1.2.19.

Additionally, it appears there could be a memory leak on the GUI side.