structured performance suite?

[synctext]

in the far far future after anonymous downloading is completed and used by 1+ million people, we can focus on performance. A structured performance test for Tribler is very difficult. What is typical network traffic to process? How good is the UDP NAT puncturing doing? What peers am I connected to for search?

We need to evaluate various options, as we already have a mature unit testing framework using Nose, crafted Gumby, messed around with systemtap, and developed Jenkins AllChannels experiments to detect issue. Starting on yet another framework is wrong. It is nice to see what others are offering: http://speed.pyston.org/comparison/ ( code: http://github.com/tobami/codespeed )

[synctext]

This ticket describes what is known as 'performance regression tests'.
Excellent way to test the fresh new core, red flag if performance drops. For instance, below more than 2 standard deviations below the average of last 10 runs.

[synctext]

Next step on this issue is spend maximum a single day on listing the various performance parameters already in various issues (Tribler bootstrap time, maximum anon download speed, bootstrap time till first GUI appearance, bootstrap time till first remote query, Total time to disk check 100GByte of seeding). Then do a first throw away prototype in 1 week to learn from. Or another approach to go towards Github Actions integration. Define a plan.
Traditional graph-based benchmarking example.

[devos50]

Related work (GUI tester - runs every PR):

See here for all screenshots taken by the GUI tester.