Ability to specify granularity of the tests to be distributed

[cfclrk]

As discussed in this post: http://stackoverflow.com/questions/4637036/is-there-a-way-to-control-how-pytest-xdist-runs-tests-in-parallel

When using distmode = 'load', it would nice if xdist provided a mechanism to allow users to specify an alternate test collection algorithm. As the StackOverflow post mentions, splitting up by TestSuite or module would be very useful.

[nicoddemus]

Hi @Ludachrispeed, thanks for the report.

This is a long requested feature, being duplicated by pytest-dev/pytest#175 and pytest-dev/pytest#738. I'm closing this as a duplicate then.

[wkerzendorf]

@nicoddemus I've tried to follow this feature request among the multiple issues - and they are seemingly all closed. I want to do something similar. The expensive calculation is all within the setup of a test class and the setup is parameterized. So is there a way to do this - even a hacky one?

[RonnyPfannschmidt]

@wkerzendorf currently the xdist internals are not in a shape that makes this easy

collection runs only on the workers, sheduling happens only on the controller, however only the only communicated metadata are test ids# not even fixtures and their scopes/ids

[wkerzendorf]

@RonnyPfannschmidt any other suggestion you have (even outside xdist)? One of our tests likely takes days. Is there a way to tell it not to split up the class, etc? thanks

[RonnyPfannschmidt]

@wkerzendorf just have a cli option to run exactly that thest and simply dont run it in a normal test run

if you have singluar end2end tests that take days - run them separate - in general - a general parallel test-runner is not thinking in terms of massive tests vs normal tests

[wkerzendorf]

@RonnyPfannschmidt we do this. So we have a parametrized setup of these long running tests (let's say A, B, C, D). Can I just say - run one parameter set of a parametrized test?
Thanks for helping me out here - I also don't want to spam your issue tracker - is there a chat room for pytest?

[nicoddemus]

I think @RonnyPfannschmidt means something like this:

1. Mark your slow/integration tests with a mark, say slow.
2. Run your regular tests using xdist: pytest -n auto -m "not slow"
3. Run your slow tests without xdist: pytest -m slow.

It should be simple to change your CI system to run pytest twice instead of only once. You can have even more layers of test sessions if you prefer.

[nicoddemus]

Btw there's a pytest channel at freenode where people answer questions.

[wkerzendorf]

@nicoddemus - we already do this.

[wkerzendorf]

But are there as helpful people as you and @RonnyPfannschmidt on this freenode channel ;-) I'll go there and give it a try.

[nicoddemus]

(btw I'm perfectly fine in using the issue tracker for this type of conversation. you can easily share code and the solution can be found by others later)

So that I understand, do you have one test which is parametrized, but is only slow for some of those parameters?