Design for the refactor of the Libtorrent wrapping code

[synctext]

Design by @vandenheuvel after the work they did with @MaxVanDeursen on #2620.

This moves far towards a lock-free architecture, lock isolation (merely Libtorrent facing modules) and step towards async code architecture. Now code has more dependency (hell). We need discussion and consensus on what we would like as a result of a solid re-factor. on-demand streaming could also use a cleanup, not yet taken into account. Not yet the new idea/interface with Libtorrent in recent Arvid discussion regarding the tunnel matters

And if all this is not yet sufficient, it should be more or less Python3 compatible and perhaps allow us to one glorious year far far into the future to replace twisted with a leaner standard system lib

Goals:

• Solve problem with Tribler showing removed torrents scheduled for removal in libtorrent (from Download state are asynchronized in Tribler and libtorrent #3463).
• Solve problem with downloading extremely big torrents, like from this one (from v6.5.0-rc5 seems to be much slower than v6.4.3 with the big torrent file #1689).
• Solve problem with encodings and status of files in updated torrent (from Broken encoding while adding a torrent from file + no included files afterwards. #3017).

[arvidn]

If I may make some comments (this may be what you're thinking, I saw the word "async" in there).

I would highly recommend that if your GUI thread needs to render a list of torrents (or peers or file progress), it should not ask libtorrent for it on-demand, with blocking calls.

A better design would be to have your own list of torrents, holding all the current states, owned by your GUI thread. Periodically (with whatever update frequency you like) you call post_torrent_updates(). This will make libtorrent compile a list of all torrents that have had any metrics/attributes changed and post it back (asynchronously) in a state_update_alert. When you handle this alert, you can update your list of torrents and possibly re-render any that may be visible to the user.

The main benefit this buys you is to minimise the GUI thread stalls while waiting for the libtorrent network thread (which under heavy load quickly can become quite costly). But perhaps more importantly, you can have hundreds of thousands of torrents loaded, and as long as they're idle, they won't affect the performance of your updating and rendering of the active ones.

(there's a similar mechanism for polling session stats counters, post_session_stats).

There's an old blog post on this topic.

[devos50]

@arvidn thank you for your comment. This sounds like a good improvement.

The GUI and Tribler core (which makes the calls to libtorrent), run in different processes since Tribler 7. Since we're planning a redesign of the whole download engine, we will take your suggestion into consideration when designing it! 👍

[ichorid]

Related #3402

[vandenheuvel]

[ichorid]

I guess I could start by thoroughly reviewing the effects of #1410.
If removing all this locks introduces no new problems, we can move to refactor the structure of the wrapper.

[ichorid]

A note on general rules for API design.

[synctext]

RESTmanager is probably the first component to start and has overall control. Not session. Removal of LaunchManyCore.
(Idea from Developers meeting discussion)

[synctext]

@ichorid Maybe higher priority then Libpath work ? #4563

[ichorid]

@synctext , moving to Pathlib will make solving Libtorrent refactor considerably easier.

[ichorid]

Recent refactorings by @egbertbouman have put Libtorrent wrapper in a much better shape. So, moving this to backlog.

[arvidn]

also, feel free to reach out if there are any usability issues with the libtorrent API.

[ichorid]

@arvidn , thanks! We'll definitely will if it be of use for Libtorrent.

[synctext]

As a wise man said:

if your GUI thread needs to render a list of torrents (or peers or file progress), it should not ask libtorrent for it on-demand, with blocking calls.

What is the progress with lots of stuff mentioned above, like?

• post_torrent_updates()
• session locks, IPv8 class structure, RESTmanager
• etc

[egbertbouman]

@ichorid As far as I'm concerned, the refactoring of the download core has completed.

[synctext]

Seems most work has been done. Document changes here please and then close this ticket?

[ichorid]

Well, I don't think there are any changes to document, as nothing was documented initially... We cleaned up the stuff, removing duplicate functions, etc.
I don't think, we need any dedicated ticket for improving our current libtorrent wrapper. Instead, we can go with gradual changes.