-
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
xdist: Different tests were collected #1075
Comments
the easy way could be, to declare the python hash seed, so the unordered parts of the collection have a high chance for having the same order the problem is, that when you use test items that have a indistinguishable source code location, order is left to mappings, and mapping order is undefined alternatively you could use a type whose items are put into a ordereddict |
As @RonnyPfannschmidt mentioned about the hash seed,
|
Im closing this one AS invalid, please reopened if a New Detail comes up |
This is still a problem for me, I have not been able to find a single project where I can successfully use more than 1 core without encountering a failure. I looked through the list of projects that use py.test and tried cloning a bunch of them and running py.test -n 2. For example tiddlyweb from here: [email protected]:tiddlyweb/tiddlyweb.git if that project is tested with py.test -n 2 test/ This happened on the pypy project and on the project I'm currently working on that I want to test faster (the test suite is currently 24 minutes long). So far xdist has been unusable for me, since specifying more than one core guarantees failure. Are there more special considerations to keep in mind if I want to parallelize with py.test -n 2 ? |
@trtg the project you linked uses a demonstrably broken approach to generate tests - thats pretty much its own fault |
Did the exploration of changing how tests are distributed discussed here: Pierre-yves Rofes changes that allow assigning all the tests in one class On Sep 13, 2016 11:57 PM, "Ronny Pfannschmidt" [email protected] @trtg https://github.com/trtg the project you linked uses a demonstrably — |
@trtg the discussionadvanced in the xdist issue tracker, but no implementation did take place |
For the record, the discussion took place in pytest-dev/pytest-xdist#18. |
@RonnyPfannschmidt My company ran into a similar issue today that I think is different. I wanted to check my understanding before opening a new issue.
Which lead to interesting test failures because of slight variations in the datetime.
Is this behavior covered by this issue? If it is would you be willing to expand on or link us to what good test generation looks like in this case? |
I believe docs on that need to be written Basically its an issue when your tests depend on dynamic values |
In your specific case it may be advisable to use "test-name" as I'd instead of part of the parameter set |
@RonnyPfannschmidt Which docs do you need. I'll go write a PR for em. |
@rawrgulmuffins thanks for signing up for that, i am currently not deeply aware of the state of the docs on what basically amounts to best practices wrt parameterize and declaring parameter-sets at all, i believe @pfctdayelise and @nicoddemus have a more detailed awareness and might be able to answer that question directly we might want to open up a followup issue to sort those details out as its not really fitting the issue at hand anymore |
@RonnyPfannschmidt definitely, created #4101 which explains the details. 👍 Thanks @rawrgulmuffins for volunteering. 🙇 |
In case someone is still having this issue. For me setting the PYTHONHASHSEED resulted one of my dict iterator still producing nondeterministic results, so I simply sorted the parametrized array after creating it. From @pytest.mark.parametrize(
'things',
[thing for thing in get_things().values()],
) To @pytest.mark.parametrize(
'things',
sorted([thing for thing in get_things().values()], key=lambda item: item.type),
) the things have unique |
Steps to reproduce the problem:
Create some test cases that look as follows:
Run
py.test -n 4
What is the expected behavior?
The tests should pass.
What went wrong?
This is the minimal test case I could break it down to. Looks like there's an issue if we're not
def
ing functions. Any help would be appreciated! 😃The text was updated successfully, but these errors were encountered: