Do not spawn more workers than testcases

[azmeuk]

I switched from a Core i5 (4 cores) to a Ryzen 7 (16 cores). The time pytest-xdist takes to spawn the workers is consequently greater, but it is expected.

Sometimes I run pytest for just several tests: sometimes 1 test, sometimes 1 file with a few tests. I thought it would save some time if pytest-xdist would only spawn the workers needed.

What do you think?

[RonnyPfannschmidt]

when xdist starts the workers, it has zero idea how many tests are there, so currently its not implementable

[Krilivye]

Should the collect phase start before launching the workers?

[nicoddemus]

I guess it would be possible to launch a single worker, collect the tests, and then based on the number of tests spawn the rest of the workers as needed, but this would probably require quite a bit of refactoring to accomplish.

[RonnyPfannschmidt]

@Krilivye the master does not run any collection, all collection is on the workers

[azmeuk]

Has it been designed willingly like this for technical reasons?

[nicoddemus]

Yes, because transferring tests and fixtures over the wire (between workers and the master node) is quite difficult/impossible, so every worker collects all tests and the master just schedules which test to run based on their node id (take a look at OVERVIEW for more details).

Some related issues: #270, #271