Non-deterministic test collection order for deterministic fixtures makes xdist fail

[aldanor]

Issues that may be related: #437, #594, #596, #669.

pytest-xdist seems to fail sometimes because it orders parametrized fixtures differently -- e.g., when a fixture depends on two or more parametrized fixtures. Using the latest pytest and pytest-xdist on Python 3.4 with this test script:

import pytest

@pytest.fixture(params=['foo', 'bar'])
def fixture1(request):
    return request.param

@pytest.fixture(params=['baz', 'foo'])
def fixture2(request):
    return request.param

def test_1(fixture1, fixture2):
    pass

Sometimes it works:

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling

foo/test_1.py::test_1[2-3]
foo/test_1.py::test_1[1-3]
[gw0] PASSED foo/test_1.py::test_1[2-3]
[gw1] PASSED foo/test_1.py::test_1[1-3]
foo/test_1.py::test_1[1-4]
foo/test_1.py::test_1[2-4]
[gw0] PASSED foo/test_1.py::test_1[2-4]
[gw1] PASSED foo/test_1.py::test_1[1-4]

========================= 4 passed in 0.60 seconds ==========================

And sometimes it doesn't (because it chose to sort the fixtures by the second parameter first for some reason):

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling
collecting 0 items / 1 errors
================================== ERRORS ===================================
___________________________ ERROR collecting gw0 ____________________________
Different tests were collected between gw1 and gw0. The difference is:
--- gw1

+++ gw0

@@ -1,4 +1,4 @@

 foo/test_1.py::test_1[foo-foo]
+foo/test_1.py::test_1[foo-bar]
+foo/test_1.py::test_1[bar-bar]
 foo/test_1.py::test_1[bar-foo]
-foo/test_1.py::test_1[bar-bar]
-foo/test_1.py::test_1[foo-bar]
========================== 1 error in 0.65 seconds ==========================

[RonnyPfannschmidt]

interesting find, my initial guess is, this is a artifact of the python hash-seed and the fixture management is just sorted different in some mappings

it'll take a while to figure the details, and it'll happen after the next release of xdist (which i am working on)

[aldanor]

Yea, the test collection mechanism is a bit flaky – in a much larger example which I won't be able to reproduce here I was able to work around it by changing some fixtures from session-scoped to function-scoped, I have no clue why that helped. Also, this doesn't seem to happen on Python 2 (not that I've seen it fail, at least).

By the way, I wonder why should it care about the order at all if the sets are the same?

[RonnyPfannschmidt]

That goes a few years back, I recall its a workaround for duplicate test ids or something like that

[jstasiak]

@aldanor FYI I fail to be able to reproduce the failure using the example code you provided. My software: Python 2.7.9 and 3.5.1, pytest 2.9.1, xdist-1.14.

[nicoddemus]

Hmm perhaps we can pass master's PYTHON_SEED value to workers? This way dict hashing should be the same across master and workers. This does not actually fix the fact that xdist uses indices to distribute tests, but at least seems sensible and would solve this issue (and others related).

[hectorv]

I'm seeing this while trying to move away from nosetests on a legacy test suite:

$ py.test -sxv -rw -n4 common/tests

this is my environment:

Python 2.7.11
pytest==2.9.2
pytest-xdist==1.14

Edit: to be clear, this works fine:

$ py.test -sxv -rw common/tests

[renefritze]

I think we still see this in our project with pytest 3.2 and xdist 1.18.2:
https://travis-ci.org/pymor/pymor/jobs/260181761#L744

[RonnyPfannschmidt]

@renemilk the issue you show is a different problem that comes from your own self-made parametrization system

all your tests that differ in order may be in a different place based on pythonhashseed
as such pytest cant know the order itself since the methods share the order

[renefritze]

Ah, ok. I misunderstood then. Setting PYTHONHASHSEED=0 in env indeed fixes the ordering issue for me locally. Thanks.

[liiight]

I think I may have the same issue. I dynamically generate parameterization based on a txt file from S3:

def parametrize_tenant_from_s3_file(metafunc):
    file_path = metafunc.config.getoption('--tenant-list-file')
    profile_name = metafunc.config.getoption('--aws-profile-name')
    aws_session = AWSSession(profile_name=profile_name)
    s3 = S3(aws_session.session)
    parsed_link = parse.urlparse(file_path)
    file_name = Path.cwd() / 'tenant_list.txt'
    s3.download_file(bucket=parsed_link.netloc, key=parsed_link.path.lstrip('/'), path=file_name)
    metafunc.parametrize('tenant_id_from_s3_file', file_name.read_text().splitlines())

Running this without -n works:

Launching pytest with arguments -m test_marker --collect-only --aws-profile-name prod /Users/orcarmi/PycharmProjects/shield/tests in /Users/orcarmi/PycharmProjects/shield

============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.8.2, py-1.5.3, pluggy-0.8.0 -- /Users/orcarmi/PycharmProjects/shield/venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.6.5', 'Platform': 'Darwin-18.2.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.8.2', 'py': '1.5.3', 'pluggy': '0.8.0'}, 'Plugins': {'xdist': '1.22.2', 'rerunfailures': '4.2', 'repeat': '0.5.0', 'metadata': '1.7.0', 'lazy-fixture': '0.4.0', 'jira': '0.3.7', 'instafail': '0.4.0', 'html': '1.17.0', 'forked': '0.2', 'clarity': '0.1.0a1', 'allure-pytest': '2.5.2'}, 'Environment': 'prod-eu-2', 'SHIELD Version': '7.0.0'}
SHIELD version: 7.0.0
Environment: prod-eu-2
rootdir: /Users/orcarmi/PycharmProjects/shield, inifile: pytest.ini
plugins: xdist-1.22.2, rerunfailures-4.2, repeat-0.5.0, metadata-1.7.0, lazy-fixture-0.4.0, jira-0.3.7, instafail-0.4.0, html-1.17.0, forked-0.2, clarity-0.1.0a1, allure-pytest-2.5.2
collecting ... collected 5096 items / 4550 deselected
<Package '/Users/orcarmi/PycharmProjects/shield'>
  <Module 'test_tsd_migration.py'>
    <Function 'test_tsd_customers_migration[tenant_0003a1ef4dfa4360bdcdc175c858e7c6]'>
    <Function 'test_tsd_customers_migration[tenant_001dececa0164f6ba44a1514c6369b42]'>
...

With -n it does not:

Launching pytest with arguments -m test_marker -n auto --aws-profile-name prod /Users/orcarmi/PycharmProjects/shield/tests in /Users/orcarmi/PycharmProjects/shield

============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.8.2, py-1.5.3, pluggy-0.8.0 -- /Users/orcarmi/PycharmProjects/shield/venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.6.5', 'Platform': 'Darwin-18.2.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.8.2', 'py': '1.5.3', 'pluggy': '0.8.0'}, 'Plugins': {'xdist': '1.22.2', 'rerunfailures': '4.2', 'repeat': '0.5.0', 'metadata': '1.7.0', 'lazy-fixture': '0.4.0', 'jira': '0.3.7', 'instafail': '0.4.0', 'html': '1.17.0', 'forked': '0.2', 'clarity': '0.1.0a1', 'allure-pytest': '2.5.2'}, 'Environment': 'prod-eu-2', 'SHIELD Version': '7.0.0'}
SHIELD version: 7.0.0
Environment: prod-eu-2
rootdir: /Users/orcarmi/PycharmProjects/shield, inifile: pytest.ini
plugins: xdist-1.22.2, rerunfailures-4.2, repeat-0.5.0, metadata-1.7.0, lazy-fixture-0.4.0, jira-0.3.7, instafail-0.4.0, html-1.17.0, forked-0.2, clarity-0.1.0a1, allure-pytest-2.5.2
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
[gw0] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw1] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw2] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw3] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw4] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw5] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw6] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw7] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw0] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw1] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw2] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw3] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw4] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw5] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw6] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw7] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
gw0 [0] / gw1 [0] / gw2 [0] / gw3 [0] / gw4 [0] / gw5 [0] / gw6 [0] / gw7 [0]

scheduling tests via LoadScheduling

 generated xml file: /Users/orcarmi/PycharmProjects/shield/reports/jreport.xml -
 generated html file: /Users/orcarmi/PycharmProjects/shield/reports/report.html 
========================= no tests ran in 6.89 seconds =========================

I tried setting PYTHONSEEDHASH to no avail.
Please advise, thanks in advance

EDIT: Realised I wasn't on latest xdist and pytest, upgrade both to latest, still same issue.

[RonnyPfannschmidt]

@liiight based on what you show its a completely different issue please open new issues instead of recycling related looking but actually fixed issues

[liiight]

Will do.

…

On Mon, Jan 21, 2019, 18:30 Ronny Pfannschmidt ***@***.*** wrote: @liiight <https://github.com/liiight> based on what you show its a completely different issue please open new issues instead of recycling related looking but actually fixed issues — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#920 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AELANb2eTnC2O2rAXOrzTVTGSwIVmKkiks5vFesQgaJpZM4Fm6YP> .

[maratumba]

I just encountered this issue while collecting tests with parametrized fixtures (pytest==6.2.5) in Django. The order of tests changes in each collection.

# Naive datetime objects for edge cases
# values marked with fuzz are ignored by default, only ran in CI pipelines
naive_datetime_fuzz = [
    pytest.param(
        parse_datetime("2021-11-07T01:30:00"), marks=pytest.mark.fuzz
    ),  # Naive datetime during US DST end
    pytest.param(
        parse_datetime("2020-02-29T09:30:00"), marks=pytest.mark.fuzz
    ),  # Leap Year Feb29th
    pytest.param(
        parse_datetime("2021-11-07T00:00:00.000"), marks=pytest.mark.fuzz
    ),  # Exactly midnight
    pytest.param(faker_uk.past_datetime()),  # Recent time
]


@pytest.fixture(params=naive_datetime_fuzz)
def naive_datetime(request):
    return request.param

pytest.ini

[pytest]
addopts = 
    --ds=config.settings.test 
    --reuse-db 
    --strict-markers
    -m "not fuzz"
python_files = tests.py test_*.py
markers =
    slow: marks tests as slow (deselect with '-m "not slow"')
    unit
    e2e
    fuzz
    integration