LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes first

[cryvate]

Based upon this PR/branch, OP text quoted here:

In the real world, test scopes aren't equal: Some have just a few tests which take little time to run while others can have a large amount of tests taking longer to finish.
With the current implementation it can happen that the scheduler assigns a scope with a large amount of tests very late in the run, often we see a single worker still running tests of a large scope while all other workers are idle because all other scopes have finished running.

This PR inserts scopes into the workqueue sorted by the amount of tests. The idea is to begin testing with the biggest scopes so that small scopes can essentially be bin packed late in the run so that all workers finish processing roughly at the same time.

I am aware that the number of tests is not the best indicator for test runtime but it is better than random scope assignment.
We have developed a more sophisticated version sorting with timings from previous test runs but unfortunately it requires external tooling.

I have one reservation about this change and backwards compatibility when we have test_a.py and test_b.py:

• in the current release, no test from test_b.py is run before any tests from test_a.py in any runner
• after this patch, tests from test_b.py might run before any tests from test_a.py in a runner (if test_b.py contains more tests)

I don't know what the general pytest(-plugin)'s view and pytest-xdist's view in particular is on backwards compatibility in the face of, essentially, flaky/broken tests.

Long term I wonder whether we would want to expose the ordering of this queue somehow: it would be cool when e.g. running in CircleCI (which can seed based on previous running times) to be able to repeat the same within the runners, especially as they could have many cores/parallel runners and so there can be inefficiencies.

[joekohlsdorf]

Long term I wonder whether we would want to expose the ordering of this queue somehow: it would be cool when e.g. running in CircleCI (which can seed based on previous running times) to be able to repeat the same within the runners, especially as they could have many cores/parallel runners and so there can be inefficiencies.

That is exactly what I do in the "more sophisticated" approach I was talking about in the original PR. We sort scopes by previous runtime and then distribute among xdist workers according to this data. But I think it will be hard to get something like this into xdist because it depends a lot on your environment. In an ideal world xdist should provide an official extension mechanism which lets you hook in your own code easily, similar to how pytest itself is extensible.
I only submitted a very light version of this in the PR which should still be helpful for most people.

We had to do a huge monkey patch to get this done but the actual custom code is <50 lines.

FYI be careful with the CircleCI test splitter, my experience is that it doesn't correctly take timings into account even though it says so.

[nicoddemus]

Thanks @cryvate and @joekohlsdorf for the contribution. 👍

I will make a new release once this is merged.

[cryvate]

Thanks @nicoddemus!