pytest 8.0 sorting tests with multiple parameterization is broken #12008
Comments
bluetech
added
type: bug
|
Minimized example: @pytest.mark.parametrize('proto', ['a', 'b'], scope='class')
@pytest.mark.parametrize('unit', [1, 2], scope='class')
class Test:
def test(self, proto, unit):
passThe items are reordered by Setup plan in pytest 7, has 6 setups/teardowns: in pytest 8, has 5 setups/teardowns: The way import pytest
@pytest.fixture(params=[1, 2], scope='class')
def unit(request):
return request.param
@pytest.fixture(params=['a', 'b'], scope='class')
def proto(request):
return request.param
class Test:
def test(self, unit, proto):
passIn this framing, it is clearer why we want to minimize the setups/teardowns -- the fixtures can do real work, not just return the param. Both pytest 7 and 8 reorder this version to have 5 setups/teardowns. I can't say why before 09b7873 the reordering didn't happen, and if this was intentional or accidental. Will look into it more. |
|
@bluetech i believe the change by @sadra-barikbin fixed a bug in pytest wrt scope ordering it dates back to #519 and the fix corrects the test scope ordering based on values and fixture values and scopes for further validation we might want to add a variant of the example for #519 that mixes class scope in (in which case the old order is actually correct but the basic gist to my current understanding is that pytest no longer has ordering differences between sugared and de-sugared parameterize as now parameterize is expressed in pseudo fixtures all the way |
|
Technical details of the change: Before 09b7873, Metafunc will generate CallSpec's with separate After 09b7873, Metafunc handles the desugaring itself, and there is no more All of this happens before https://github.com/pytest-dev/pytest/blob/7.4.4/src/_pytest/fixtures.py#L168-L181 The important bit is ('proto', 0, <Function test[1-a]>)
('proto', 1, <Function test[1-b]>)
('proto', 2, <Function test[2-a]>)
('proto', 3, <Function test[2-b]>)
('unit', 0, <Function test[1-a]>)
('unit', 1, <Function test[1-b]>)
('unit', 2, <Function test[2-a]>)
('unit', 3, <Function test[2-b]>)After, there is no special handling of the param indexes of direct params, they are handled same as parametrized fixtures. This results in the following arg keys: ('proto', 0, <Function test[1-a]>)
('proto', 0, <Function test[2-a]>)
('proto', 1, <Function test[1-b]>)
('proto', 1, <Function test[2-b]>)
('unit', 0, <Function test[1-a]>)
('unit', 0, <Function test[1-b]>)
('unit', 1, <Function test[2-a]>)
('unit', 1, <Function test[2-b]>)Rephrasing the above in a way that may be clearer: Before, indexes for direct params were assigned in a sequential manner per
After, the param indexes are assigned as they would for the desugaring I gave above: import pytest
# param indexes: 0 1
@pytest.fixture(params=[1, 2], scope='class')
def unit(request):
return request.param
# param indexes: 0 1
@pytest.fixture(params=['a', 'b'], scope='class')
def proto(request):
return request.param |
|
@ShurikMen I wonder, is the example you gave realistic or do you use |
This is one of the simplest examples that are actually used in my projects. There are many examples with an even larger set of parameters including "mixed" scopes (class/function).
Not quite like that. Changing the order causes unwanted fixture calls. They are very expensive in terms of execution time. With scope class (same optimal): import pytest
@pytest.fixture(autouse=True, scope='class')
def some_fix(unit, proto):
yield
@pytest.mark.parametrize('proto', ['serial', 'telnet'], scope='class')
@pytest.mark.parametrize('unit', [1, 2], scope='class')
class TestA:
def test_one(self, unit, proto):
pass
def test_two(self, unit, proto):
passWith scope function import pytest
@pytest.fixture(autouse=True, scope='function')
def some_fix(unit, proto):
yield
@pytest.mark.parametrize('proto', ['serial', 'telnet'], scope='function')
@pytest.mark.parametrize('unit', [1, 2], scope='function')
class TestA:
def test_one(self, unit, proto):
pass
def test_two(self, unit, proto):
passClass params with function fixture import pytest
@pytest.fixture(autouse=True, scope='function')
def some_fix(unit, proto):
yield
@pytest.mark.parametrize('proto', ['serial', 'telnet'], scope='class')
@pytest.mark.parametrize('unit', [1, 2], scope='class')
class TestA:
def test_one(self, unit, proto):
pass
def test_two(self, unit, proto):
passHere I experimented a lot with various combinations of scope parameters and scope fixtures, including parameterization in fixtures (pytest.fixture(param=...)) to obtain the most optimal ways to perform fixtures and tests. |
|
Currently when parameterize is taken for consideration, compound dependent fixtures are not Id recommend having a single parameterset that creates the correct compound parameters to a single fixture so it will no longer be considered as independent parameters |
bluetech
changed the title
|
Here are the three approaches: |
|
Seems we have an appearance-efficiency trade-off, with @ShurikMen's suggestion and v7.4.4 approach yielding better appearance but having lower efficiency in setup-teardowns in comparison with v8.0.0. The three approaches could also be compared in terms of robustness to parametrization varieties which @ShurikMen 's suggestion for example performs better than v7.4.4 and solves the first example of #11257 but fails on the example below which v8.0.0 solves, unless the user be cautious and swap the order of parametrizations on # Here `test1["b",0]` and `test2[3]` would wrongfully have a common fixturekey in @ShurikMen 's method.
@pytest.mark.parametrize("arg2", [0, 1, 2], scope='module')
@pytest.mark.parametrize("arg1", ["a", "b"], scope='module')
def test1(arg1, arg2):
pass
@pytest.mark.parametrize("arg2", [0, 1, 2, 3], scope='module')
def test2(arg2):
passConsidering robustness alone, @ShurikMen method's gain over v7.4.4 seems remarkable but I'm not sure about that of v8.0.0 over @ShurikMen 's. |
|
Aside by the comments above,I'm for the notion of bug for current reordering in v8.0.0 as it has not been introduced by the intention to become more efficient which we discussed here about. It either has been an unnoticed side-effect of #11220 or something in my large initial PR, responsible for the first example of #11257 (which @ShurikMen 's method now solves as well). Sorry for inconvenience, mates! |
|
I have updated the pr. Unified indexing of parameters added through a marker and through fixtures. I think it's the right thing to do. Along the way, I corrected the tests related to parameterization through fixtures. |
|
@bluetech what about solving the problem on this issue? |
In #11220, an unintended change in reordering was introduced by changing the way indices were assigned to direct params. This PR reverts that change and reduces #11220 changes to just refactors. After this PR we could safely decide on the solutions discussed in #12008, i.e. #12082 or the one initially introduced in #11220 . Fixes #12008 Co-authored-by: Bruno Oliveira <[email protected]> Co-authored-by: Bruno Oliveira <[email protected]>
and others
Сontinue #11976
The solution from #11976 did not solve the problem with sorting tests with multiple parameters
Tests
Collecting items with pytest 7.4.4
Collecting items with pytest 8.0.1
The text was updated successfully, but these errors were encountered: