Customize the fixture ordering on the same scope level

[gavincyi]

Could we have one more parameter, e.g. priority (between 0 and 100), for the users the customize the cost of setup/teardown fixture? For example,

@pytest.fixture(scope="session", params=[1, 2, 3], priority=100)
def f1(request):
    pass

@pytest.fixture(scope="session", params=['a', 'b', 'c'], priority=50)
def f2(request):
    pass

def test_f(f1, f2):
    pass

should give

test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[3-a] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-c] PASSED

pytest now assumes an equal weighted cost of setup/teardown fixtures. So it reorders the product of fixture combination to optimize the count of setup/teardown time. For the code given before, the current behavior is

pytest_example.py::test_f[1-a] PASSED
pytest_example.py::test_f[1-b] PASSED
pytest_example.py::test_f[2-b] PASSED
pytest_example.py::test_f[2-a] PASSED
pytest_example.py::test_f[2-c] PASSED
pytest_example.py::test_f[1-c] PASSED
pytest_example.py::test_f[3-c] PASSED
pytest_example.py::test_f[3-b] PASSED
pytest_example.py::test_f[3-a] PASSED

The assumption is not pragmatic when

1. The setup/teardown cost of a fixture is much heavier than that of other fixtures
2. The current reordering can have more than 1 solution, while the user cannot determine which ordering will be used (e.g. the cost of ['1-a', '1-b', '2-b', '2-a'] is same as that of ['1-a', '2-a', '2-b', '1-b']).

This enhancement will allow the users to customize the ordering if they want and resolve #2846.

[pytestbot]

GitMate.io thinks possibly related issues are #538 (Fixture scope documentation), #805 (Fixture execution order ), #668 (autouse fixtures break scope rules), and #2846 (Unexpected order of tests using parameterized fixtures).

[nicoddemus]

@gavincyi thanks for opening up this issue. 👍

Seems doable, although I'm not sure adding a parameter to @pytest.fixture which would be used solely by session-scoped-parametrized-fixtures is a good idea, it might be confusing. Perhaps adding a mark instead, something like @pytest.mark.session_reorder_priority(100) or @pytest.mark.session_reorder_weight(100)? That mark then can be used by the reorder algorithm to reduce fixture instantiations of fixtures with higher priority/weight.

[nicoddemus]

cc @ceridwen and @cheezman34 because of #3161.

[RazerM]

I may be thinking about this wrong, but wouldn't this also be useful for non-session fixtures?

In my case what I'm trying is different function scoped fixtures (e.g. fix_pre1, fix_pre2, ...) that I always want to happen before a certain other fixture (e.g. fix_main). The problem is that fix_main can't depend on fix_pre because it's up to the tests to choose which fix_pre fixtures are in use.

I thought that the parameter order might help determine the fixture order, but that's only the case when I try a simple example like this:

import pytest

@pytest.fixture
def fix_main():
    pass

@pytest.fixture
def fix_pre():
    pass

def test_abc(fix_pre, fix_main):
    pass

In my real test suite, fix_main is always setup before fix_pre, and I can't see a way to change that. fix_main is provided by a plugin, so it's difficult to replace it if I wanted to make a different version for each combination of fix_preN that my tests use.

[Sup3rGeo]

Wow it took me a while but I finally found other people with the same problem as I am having now! Definitely not all fixtures for a same scope have equal weight.

Seems doable, although I'm not sure adding a parameter to @pytest.fixture which would be used solely by session-scoped-parametrized-fixtures is a good idea, it might be confusing.

I think this is valid for all the scopes higher than function - then the priority would select which fixtures should be preserved among all other fixtures at the same scope level.

Is there any workarounds that can be implemented locally to overcome this problem? I thought about implementing my own algorithm in pytest_collection_modifyitems but I am not sure if this would be too hard.

[Sup3rGeo]

So my proposal would be as follows:

1. Add a weight parameter to fixtures (even negative, considering the default priority could be 0)
2. The ordering would be based on fixture weight groups:

• higher weight fixtures should be preserved in detriment of more instantiations lower weight fixtures
• the current algorithm being applied for fixtures with the same weight.

[nicoddemus]

Is there any workarounds that can be implemented locally to overcome this problem?

One workaround, if you have access to the code of the fixtures, is to make one fixture depend on the other. For example:

@pytest.fixture
def foo():
    ...

@pytest.fixture
def bar():
    ...

If you want to make sure that bar always executes before foo, make foo depend on bar:

@pytest.fixture
def foo(bar):
    ...

@pytest.fixture
def bar():
    ...

About the original proposal, I'm still not entirely sure if we should really add a priority parameter to fixtures... perhaps we can reorganize the code a bit to allow for a plugin to change the priority instead? 🤔

[Sup3rGeo]

I have a working implementation of the sorting algorithm that takes weights/priorities into account, now I just need to figure the best way to define it in tests so that the algorithm can retrieve the weights for each parameter.

@nicoddemus you mentioned the idea of

Perhaps adding a mark instead, something like @pytest.mark.session_reorder_priority(100) or @pytest.mark.session_reorder_weight(100)? That mark then can be used by the reorder algorithm to reduce fixture instantiations of fixtures with higher priority/weight.

How could I access that mark? I basically want to access that mark in this function (or, to be more precise, my plugin's own implementation of that):

pytest/src/_pytest/fixtures.py

Lines 182 to 205 in 6663cb0

	def get_parametrized_fixture_keys(item, scopenum):
	""" return list of keys for all parametrized arguments which match
	the specified scope. """
	assert scopenum < scopenum_function # function
	try:
	cs = item.callspec
	except AttributeError:
	pass
	else:
	# cs.indices.items() is random order of argnames. Need to
	# sort this so that different calls to
	# get_parametrized_fixture_keys will be deterministic.
	for argname, param_index in sorted(cs.indices.items()):
	if cs._arg2scopenum[argname] != scopenum:
	continue
	if scopenum == 0: # session
	key = (argname, param_index)
	elif scopenum == 1: # package
	key = (argname, param_index, item.fspath.dirpath())
	elif scopenum == 2: # module
	key = (argname, param_index, item.fspath)
	elif scopenum == 3: # class
	key = (argname, param_index, item.fspath, item.cls)
	yield key

And I see we could access the FixtureDef through cs.metafunc._args2fixturedefs[argname]

[Sup3rGeo]

People interested in this, I made a plugin that implements a @parameter_priority decorator that can use to set strict order for parameters. The good thing is that this still uses the original algorithm, but priorities are not defined just by scope, but by (scope, priority).

Check how to use it in https://github.com/Sup3rGeo/pytest-param-priority and please provide feedback :) I did not test with classes yet.

The implementation possibly can be improved, and it would be quite easier if we could just add it to FixtureDefs somehow directly (check https://github.com/Sup3rGeo/pytest-param-priority/blob/master/pytest_param_priority.py)

[smarie]

Two comments on this:

• it seems to me that the ordering problem also applies on module-scope fixtures
• would it be useful also to enable users to declare that a session-scoped or module-scoped fixture should be unique (= setup and torn down only once per parameter in a given session/module)? For example @fixture(unique_in_scope=True). False would obviously be the default value to preserve the current behaviour.

[forestnew]

After so many years, this question remains relevant.
I ran into this problem and was also looking for a solution, I found this discussion.

People interested in this, I made a plugin that implements a @parameter_priority decorator that can use to set strict order for parameters. The good thing is that this still uses the original algorithm, but priorities are not defined just by scope, but by (scope, priority).

Check how to use it in https://github.com/Sup3rGeo/pytest-param-priority and please provide feedback :) I did not test with classes yet.

The implementation possibly can be improved, and it would be quite easier if we could just add it to FixtureDefs somehow directly (check https://github.com/Sup3rGeo/pytest-param-priority/blob/master/pytest_param_priority.py)

Your solution seems interesting to me. It's amazing that after all this time it still causes problems