Move `fixtures.py::add_funcarg_pseudo_fixture_def` to `Metafunc.parametrize` #11220

sadra-barikbin · 2023-07-17T01:34:33Z

To remove fixtures.py::add_funcargs_pseudo_fixture_def and add its logic i.e. registering funcargs as params and making corresponding fixturedefs, right to Metafunc.parametrize in which parametrization takes place.
~~To compute param indices smarter to have a better reordering experience.~~ Moved to Resolve param indices using param values, not parameterset index #11257
To remove funcargs from metafunc attributes as we populate metafunc params and make pseudo fixturedefs simultaneously and there's no need to keep funcargs separately.

RonnyPfannschmidt · 2023-07-19T18:38:35Z

src/_pytest/python.py

+            if name2pseudofixturedef is not None and argname in name2pseudofixturedef:
+                fixturedef = name2pseudofixturedef[argname]
+            else:
+                fixturedef = FixtureDef(


this particular usage indicates to me that we may want to investigate having a "constant" definitions that could be used to represent params both from parametrize, and also declaring fixtures that repressent sets of values

details are for a followup that may want to also integrate pytest-lazyfixtures

By "constant" you mean stateless? A fixturedef that just returns its input?

src/_pytest/python.py

bluetech

Hi @sadra-barikbin, I left some initial comments.

To remove fixtures.py::add_funcargs_pseudo_fixture_def and add its logic
To compute param indices smarter to have a better reordering experience

Are these two changes dependent on each other? It will help the review a lot if the two changes can be considered separately.

src/_pytest/python.py

src/_pytest/fixtures.py

src/_pytest/python.py

bluetech · 2023-07-27T20:55:09Z

src/_pytest/python.py

+
+        argnames = ["A", "B", "C"]
+        parametersets = [("a1", "b1", "c1"), ("a1", "b2", "c1"), ("a1", "b3", "c2")]
+        result = [(0, 0, 0), (0, 1, 0), (0, 2, 1)]


I might be misunderstanding, but shouldn't the last triplet be (0, 2, 2) (i.e. the index of c2 should be 2 not 1)?

It should be 1 as c2 is the second value in C unique values. In other words, its index is not determined by the index of the parameter set, but its place in the unique values of the arg.

Related to #11257

testing/python/metafunc.py

bluetech · 2023-07-27T21:23:07Z

src/_pytest/fixtures.py

@@ -1052,6 +973,9 @@ def __init__(
        self.cached_result: Optional[_FixtureCachedResult[FixtureValue]] = None
        self._finalizers: Final[List[Callable[[], object]]] = []

+        # Whether fixture is a pseudo-fixture made in direct parametrizations.


I understand this terminology is preexisting, but I don't like it much, "pseudo" is not very descriptive... Also, there's actually a PseudoFixtureDef already which is something else entirely and which only adds to the confusion :)

Anyway we can clean the terms later, you don't need to change it.

While we are at it, perhaps is_parametrized would be a good fit?

Also do we want this to be public, or prefix it with _ to be sure?

How is to introduce an IdentityFixture class and use isinstance instead of adding and checking is_pseudo?

class IdentityFixture(FixtureDef): def __init__(fixturemanager, argname, scope): super().__init__(fixturemanager, "", argname, lambda request: request.param, scope, None, _ispytest=True)

This also accounts for @RonnyPfannschmidt 's comment. Just we should remove @final from FixtureDef class.

Or simply introduce is_identity? @nicoddemus , Couldn't is_parametrized be confusing, since such a fixture is not really parametrized (using @pytest.fixture(params=) or @pytest.mark.parametrize(indirect=True) but the test is directly parametrized?

Hmm that's a good suggestion; my gut reaction is that I like this idea of having IdentityFixture (or some other name in case others have a suggestion). Let's see what others think.

i'd like to note that we do have a "spagetty-fied" pattern here about configuring a "dependency injection container" with either direct values or parameters to "factories"

if we introduce polymorphism, we might want to make "identity" and "from factory, possibly with parameters" siblings and check if we can get rid of those is_pseudo checks altogether

there is a rat-tail of discussions to be had around this, so we ought to consider it but without going down the rabbit hole too deep

As params arg of this very fixturedef is None, it's automatically ruled out in the only place (fixtures.py::pytest_generate_tests) where I used is_pseudo. So the need for is_pseudo attribute is eliminated.

bluetech

Thanks for splitting @sadra-barikbin. I left some more comments.

Overall the moving of add_funcarg_pseudo_fixture_def into Metafunc itself looks good to me, it's a good stepping-stone simplification.

I'd still like to understand the change in the issue519 test better.

Also, the removal of CallSpec2.funcargs is going to break some plugins, we'll need to analyze and mitigate and/or document it properly before merging. Since add_funcarg_pseudo_fixture_def always clears funcargs I'm not sure how useful it is, but maybe plugins hook to it before it is cleared.

src/_pytest/python.py

bluetech · 2023-07-29T07:48:46Z

src/_pytest/python.py

+            name2pseudofixturedef = node.stash.setdefault(
+                name2pseudofixturedef_key, default
+            )
+        arg_values_types = self._resolve_arg_value_types(argnames, indirect)


I think it would be helpful for clarity if (in separate PR) we change _resolve_arg_value_types like so:

Rename to _resolve_arg_directness - "arg value type" is pretty generic/non-specific and doesn't say much.

Change the Literal return value from Literal['params', 'funcargs'] to Literal['direct', 'indirect'].

Then the check becomes if arg_directness[argname] == "direct": continue which is pretty self-explanatory I think.

Note: we can do it later, you don't have to do it.

testing/python/metafunc.py

bluetech · 2023-07-29T09:00:25Z

testing/example_scripts/issue_519.py

@@ -22,13 +22,13 @@ def checked_order():
    assert order == [
        ("issue_519.py", "fix1", "arg1v1"),
        ("test_one[arg1v1-arg2v1]", "fix2", "arg2v1"),
-        ("test_two[arg1v1-arg2v1]", "fix2", "arg2v1"),


As I said in https://github.com/pytest-dev/pytest/pull/11231/files#r1278270549 IMO this change makes sense (though @RonnyPfannschmidt may still disagree).

But I would like to understand what exactly in this PR is causing this change? @sadra-barikbin do you think you can explain it?

i left another comment, i think we can choose to opt for the new ordering if we ensure to add the tools i mentioned there eventually

Current ordering stems from the way add_funcarg_pseudo_fixture_def assigns index to params. It assigns i to the index of all of the params in ith callspec of the metafunc. In our example, test_one and test_two, each have 4 callspecs and two params (arg1 and arg2). We assign index 0 to arg1 and arg2 of test_one[arg1v1,arg2v1] and also to those of test_two[arg1v1,arg2v1]. We assign index 1 to arg1 and arg2 of test_one[arg1v1,arg2v2] and also to those of test_two[arg1v1,arg2v2] and so on. As a result, ith callspecs of the two metafuncs would have common fixture keys. In our example, kth test_one pulls kth test_two towards itself, so we would have a one-two-one-two... pattern. Also test_one (also test_two ones) items would have no common fixture key because their index of arg1 and arg2 is 0,1,2 and 3.

The new way assigns i to the index of param j in all callspecs that use ith value of j. So the decision is made on a per-param basis. In our example, we would assign index 0 to arg1 of the items that use arg1v1 and index 1 to arg1 of the items that use arg1v2 and so on. As a result, if two callspecs have the ith value of the param j, they have a common fixture key.

The new way is improved in determining what ith value of the param is when there are multiple parametersets in #11257 .

After reading your comment and looking more closely at the issue_519.py file, I think I understood some things:

First, I'm really not used to the pytest_generate_tests hook, so I rewrote the test using pytest.mark.parametrize. I also changed some of the names to make things clearer to me:

Rewritten issue_519.py

import pprint from typing import List from typing import Tuple import pytest @pytest.fixture(scope="session") def checked_order(): order: List[Tuple[str, str, str]] = [] yield order pprint.pprint(order) @pytest.fixture(scope="module") def fixmod(request, a, checked_order): checked_order.append((request.node.name, "fixmod", a)) yield "fixmod-" + a @pytest.fixture(scope="function") def fixfun(request, fixmod, b, checked_order): checked_order.append((request.node.name, "fixfun", b)) yield "fixfun-" + b + fixmod @pytest.mark.parametrize("b", ["b1", "b2"], scope="function") @pytest.mark.parametrize("a", ["a1", "a2"], scope="module") def test_one(fixfun, request): print() # print(request._pyfuncitem.nodeid, request._pyfuncitem.callspec) @pytest.mark.parametrize("b", ["b1", "b2"], scope="function") @pytest.mark.parametrize("a", ["a1", "a2"], scope="module") def test_two(fixfun, request): print() # print(request._pyfuncitem.nodeid, request._pyfuncitem.callspec)

This made me realize that the apparent niceness of the previous ordering, which orders test_one[arg1v1-arg2v1] next to test_two[arg1v1-arg2v1], thus saving a setup, is pretty arbitrary. This can be shown by making the values of the function-scoped parameter (arg2/b) different in test_one and test_two:

-@pytest.mark.parametrize("b", ["b1", "b2"], scope="function") +@pytest.mark.parametrize("b", ["b11", "b12"], scope="function") @pytest.mark.parametrize("a", ["a1", "a2"], scope="module") def test_one(fixfun, request): print() # print(request._pyfuncitem.nodeid, request._pyfuncitem.callspec) -@pytest.mark.parametrize("b", ["b1", "b2"], scope="function") +@pytest.mark.parametrize("b", ["b21", "b22"], scope="function") @pytest.mark.parametrize("a", ["a1", "a2"], scope="module") def test_two(fixfun, request): print()

The ordering stays the same, which now doesn't make sense.

I also figured that the entire param_index thing was much more useful/natural in the pre-parametrize times, when only pytest_generate_tests/metafunc was used for parametrizing. With pytest_generate_tests you often use the same parameter set for different tests, as is done in the issue_519.py file. But with parametrize, which is mostly done for a single test-function only, it is much less common, I think.

bluetech

(I still need to grok https://github.com/pytest-dev/pytest/pull/11220/files#r1280394308 but some comments in the meantime)

src/_pytest/python.py

bluetech · 2023-08-02T06:02:28Z

testing/python/metafunc.py

@@ -1503,6 +1548,31 @@ def test_it(x): pass
        result = pytester.runpytest()
        assert result.ret == 0

+    def test_parametrize_module_level_test_with_class_scope(


Thanks, I think this test is good now.

I suggest adding a docstring:

Test that a class-scoped parametrization without a corresponding Class gets module scope, i.e. we only create a single FixtureDef for it per module.

By the way, Is it ok that the scope attribute of the created fixturedef would be Module/Class depending on whether the parametrized module-level test functions come first or such a class-less test?

I'm pretty sure that the fixturedef.scope needs to be class in this case, for things like the "can't request lower scope from higher scope" check, introspection, and probably other stuff. It seems safer to me to keep the original scope and do the "find the actual scope" dance only where needed.

I mean if we swap test_1 and test_2, the scope attribute of x fixturedef would be module as test_2 is collected earlier then. Is this OK?

However, It does not affect reordering since we use callspec._arg2scope for creating fixture keys.

I mean if we swap test_1 and test_2, the scope attribute of x fixturedef would be module as test_2 is collected earlier then. Is this OK?

It's dubious, I'd say it's a bug (preexisting of course).

testing/python/metafunc.py

bluetech · 2023-08-08T22:41:23Z

src/_pytest/python.py

+                    argname=argname,
+                    func=get_direct_param_fixture_func,
+                    scope=scope_,
+                    params=None,


This is now params=None, where before it was params=valuelist. This made me confused somewhat.

My mental model for before was the basically desugar this:

@pytest.mark.parametrize("x", [1]) def test_it(x): pass

to this:

@pytest.fixture(params=[1]) def x(request): return request.param def test_it(x): pass

but with params=None that makes less sense now. And indeed, if we change to params=None in the current code, all tests still pass, i.e. it wasn't needed even before.

I guess then that the correct mental model for the desugaring is indirect parametrization of the x fixture? I.e. this:

@pytest.fixture() def x(request): return request.param @pytest.mark.parametrize("x", [1], indirect=["x"]) def test_it(x): pass

Ran out of time to look into it today but will appreciate your insights :)

You're right. It is the test that holds param values in its callspec and the fixtures that it depends on, use them at test execution time. The only usage of fixturedef.params is in fixtures.py::pytest_generate_tests that the directly parametrized fixture hand the params to the test that uses it. By directly parametrized fixture, I mean this:

@pytest.fixture(params=[1]) def x(request): return request.param

bluetech · 2023-08-08T22:41:41Z

testing/example_scripts/issue_519.py

@@ -22,13 +22,13 @@ def checked_order():
    assert order == [
        ("issue_519.py", "fix1", "arg1v1"),
        ("test_one[arg1v1-arg2v1]", "fix2", "arg2v1"),
-        ("test_two[arg1v1-arg2v1]", "fix2", "arg2v1"),


After reading your comment and looking more closely at the issue_519.py file, I think I understood some things:

First, I'm really not used to the pytest_generate_tests hook, so I rewrote the test using pytest.mark.parametrize. I also changed some of the names to make things clearer to me:

Rewritten issue_519.py

import pprint from typing import List from typing import Tuple import pytest @pytest.fixture(scope="session") def checked_order(): order: List[Tuple[str, str, str]] = [] yield order pprint.pprint(order) @pytest.fixture(scope="module") def fixmod(request, a, checked_order): checked_order.append((request.node.name, "fixmod", a)) yield "fixmod-" + a @pytest.fixture(scope="function") def fixfun(request, fixmod, b, checked_order): checked_order.append((request.node.name, "fixfun", b)) yield "fixfun-" + b + fixmod @pytest.mark.parametrize("b", ["b1", "b2"], scope="function") @pytest.mark.parametrize("a", ["a1", "a2"], scope="module") def test_one(fixfun, request): print() # print(request._pyfuncitem.nodeid, request._pyfuncitem.callspec) @pytest.mark.parametrize("b", ["b1", "b2"], scope="function") @pytest.mark.parametrize("a", ["a1", "a2"], scope="module") def test_two(fixfun, request): print() # print(request._pyfuncitem.nodeid, request._pyfuncitem.callspec)

This made me realize that the apparent niceness of the previous ordering, which orders test_one[arg1v1-arg2v1] next to test_two[arg1v1-arg2v1], thus saving a setup, is pretty arbitrary. This can be shown by making the values of the function-scoped parameter (arg2/b) different in test_one and test_two:

-@pytest.mark.parametrize("b", ["b1", "b2"], scope="function") +@pytest.mark.parametrize("b", ["b11", "b12"], scope="function") @pytest.mark.parametrize("a", ["a1", "a2"], scope="module") def test_one(fixfun, request): print() # print(request._pyfuncitem.nodeid, request._pyfuncitem.callspec) -@pytest.mark.parametrize("b", ["b1", "b2"], scope="function") +@pytest.mark.parametrize("b", ["b21", "b22"], scope="function") @pytest.mark.parametrize("a", ["a1", "a2"], scope="module") def test_two(fixfun, request): print()

The ordering stays the same, which now doesn't make sense.

I also figured that the entire param_index thing was much more useful/natural in the pre-parametrize times, when only pytest_generate_tests/metafunc was used for parametrizing. With pytest_generate_tests you often use the same parameter set for different tests, as is done in the issue_519.py file. But with parametrize, which is mostly done for a single test-function only, it is much less common, I think.

sadra-barikbin · 2023-08-09T08:17:52Z

By pre-parametrize times, you mean such a use-case

def pytest_generate_tests(metafunc):
    metafunc.parametrize("arg", ["v1","v2","v3"])

in which user aims to parametrize all/many of the tests in a module/class with the same param? If yes, you say param_index is more meaningful in such case because for example arg_index=0 means the same thing (v1) for all the tests?

bluetech · 2023-08-09T15:23:00Z

Yes exactly, my guess is that it was much more common for parametrizations with the same argname to have the exact same parameter sets because it was just difficult to do otherwise. From that POV the param_index concept made some sense. Nowadays with pytest.mark.parameterize, less so.

…unc-parametrize

bluetech

OK, I admit I am not fully 100% on all of the implications but we covered everything I could figure out.

One thing I'm not very happy about is that previously Metafunc was "pure" in the sense that it did not have external side-effects, but now metafunc.parametrize registers the pseudo-fixtures. I think it would be less bug-prone to have Metafunc only collect the fixturedefs but leave the actual registering of them to _genfunctions. But this is not a blocker, maybe something for later.

So LGTM, thanks @sadra-barikbin!

src/_pytest/python.py

sadra-barikbin · 2023-08-09T18:13:18Z

OK, I admit I am not fully 100% on all of the implications but we covered everything I could figure out.

One thing I'm not very happy about is that previously Metafunc was "pure" in the sense that it did not have external side-effects, but now metafunc.parametrize registers the pseudo-fixtures. I think it would be less bug-prone to have Metafunc only collect the fixturedefs but leave the actual registering of them to _genfunctions. But this is not a blocker, maybe something for later.

So LGTM, thanks @sadra-barikbin!

Regarding the external side effects, the good news is that it's welll-known; Only when the parametrization is direct and its scope is higher than function. (which stashes the new fixturedef to a class/module/package/session node). Otherwise, only the metafunc itself is changed which was also the case before this PR.

In #11220, an unintended change in reordering was introduced by changing the way indices were assigned to direct params. This PR reverts that change and reduces #11220 changes to just refactors. After this PR we could safely decide on the solutions discussed in #12008, i.e. #12082 or the one initially introduced in #11220 . Fixes #12008 Co-authored-by: Bruno Oliveira <[email protected]> Co-authored-by: Bruno Oliveira <[email protected]>

sadra-barikbin changed the title ~~"Move add_pseudo_funcarg_fixturedef to Metafunc.parametrize"~~ Move fixtures.py::add_funcarg_pseudo_fixture_def to Metafunc.parametrize Jul 17, 2023

sadra-barikbin marked this pull request as ready for review July 19, 2023 00:39

sadra-barikbin mentioned this pull request Jul 19, 2023

Make high-scope fixtures teardown at last dependent test #10771

Closed

RonnyPfannschmidt reviewed Jul 19, 2023

View reviewed changes

src/_pytest/python.py Show resolved Hide resolved

bluetech self-requested a review July 20, 2023 08:35

sadra-barikbin mentioned this pull request Jul 20, 2023

Some possible improvements in fixtures module #11234

Open

bluetech reviewed Jul 27, 2023

View reviewed changes

bluetech mentioned this pull request Jul 28, 2023

fixtures: fix crash when parametrize(scope="package") is used without a Package #11256

Merged

Apply comments, rebase and a few improvements

96be57a

sadra-barikbin force-pushed the Improvement-move-add-pseudo-funcarg-to-metafunc-parametrize branch from 59a848c to 96be57a Compare July 28, 2023 23:41

sadra-barikbin mentioned this pull request Jul 28, 2023

Resolve param indices using param values, not parameterset index #11257

Open

bluetech requested changes Jul 29, 2023

View reviewed changes

A few improvements

40b2c09

sadra-barikbin force-pushed the Improvement-move-add-pseudo-funcarg-to-metafunc-parametrize branch from 9ac1f5c to 40b2c09 Compare August 1, 2023 10:15

bluetech reviewed Aug 2, 2023

View reviewed changes

Apply comments

8b59725

bluetech reviewed Aug 8, 2023

View reviewed changes

Merge branch 'main' into Improvement-move-add-pseudo-funcarg-to-metaf…

2853331

…unc-parametrize

bluetech approved these changes Aug 9, 2023

View reviewed changes

src/_pytest/python.py Outdated Show resolved Hide resolved

Update src/_pytest/python.py

c7f4f91

bluetech enabled auto-merge (squash) August 9, 2023 16:55

bluetech merged commit 09b7873 into pytest-dev:main Aug 9, 2023
23 checks passed

bluetech mentioned this pull request Aug 9, 2023

python: use clearer terminology for _resolve_arg_value_types #11301

Merged

sadra-barikbin mentioned this pull request Sep 5, 2023

Remove prune_dependency_tree and reuse getfixtureclosure logic #11243

Open

bluetech mentioned this pull request Feb 19, 2024

pytest 8.0 sorting tests with multiple parameterization is broken #12008

Closed

RonnyPfannschmidt mentioned this pull request Feb 21, 2024

fixture scope is ignored when using metafunc.parametrize() #519

Closed

sadra-barikbin mentioned this pull request Jun 27, 2024

Revert the unintended change in tests reordering from #11220 #12542

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move `fixtures.py::add_funcarg_pseudo_fixture_def` to `Metafunc.parametrize` #11220

Move `fixtures.py::add_funcarg_pseudo_fixture_def` to `Metafunc.parametrize` #11220

sadra-barikbin commented Jul 17, 2023 •

edited

Loading

RonnyPfannschmidt Jul 19, 2023

sadra-barikbin Jul 19, 2023

RonnyPfannschmidt Jul 20, 2023

bluetech left a comment

bluetech Jul 27, 2023

sadra-barikbin Jul 29, 2023 •

edited

Loading

sadra-barikbin Aug 1, 2023

bluetech Jul 27, 2023

nicoddemus Jul 27, 2023

nicoddemus Jul 27, 2023

sadra-barikbin Jul 28, 2023

nicoddemus Jul 28, 2023

RonnyPfannschmidt Jul 28, 2023

sadra-barikbin Jul 29, 2023 •

edited

Loading

bluetech left a comment

bluetech Jul 29, 2023

bluetech Jul 29, 2023

RonnyPfannschmidt Jul 29, 2023

sadra-barikbin Aug 1, 2023 •

edited

Loading

bluetech Aug 8, 2023

bluetech left a comment

bluetech Aug 2, 2023

sadra-barikbin Aug 2, 2023 •

edited

Loading

bluetech Aug 2, 2023

sadra-barikbin Aug 2, 2023 •

edited

Loading

sadra-barikbin Aug 2, 2023

bluetech Aug 2, 2023

bluetech Aug 8, 2023

sadra-barikbin Aug 9, 2023

bluetech Aug 8, 2023

sadra-barikbin commented Aug 9, 2023

bluetech commented Aug 9, 2023

bluetech left a comment

sadra-barikbin commented Aug 9, 2023 •

edited

Loading

Move fixtures.py::add_funcarg_pseudo_fixture_def to Metafunc.parametrize #11220

Move fixtures.py::add_funcarg_pseudo_fixture_def to Metafunc.parametrize #11220

Conversation

sadra-barikbin commented Jul 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bluetech left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadra-barikbin Jul 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadra-barikbin Jul 29, 2023 • edited Loading

Choose a reason for hiding this comment

bluetech left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadra-barikbin Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bluetech left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadra-barikbin Aug 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadra-barikbin Aug 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadra-barikbin commented Aug 9, 2023

bluetech commented Aug 9, 2023

bluetech left a comment

Choose a reason for hiding this comment

sadra-barikbin commented Aug 9, 2023 • edited Loading

Move `fixtures.py::add_funcarg_pseudo_fixture_def` to `Metafunc.parametrize` #11220

Move `fixtures.py::add_funcarg_pseudo_fixture_def` to `Metafunc.parametrize` #11220

sadra-barikbin commented Jul 17, 2023 •

edited

Loading

sadra-barikbin Jul 29, 2023 •

edited

Loading

sadra-barikbin Jul 29, 2023 •

edited

Loading

sadra-barikbin Aug 1, 2023 •

edited

Loading

sadra-barikbin Aug 2, 2023 •

edited

Loading

sadra-barikbin Aug 2, 2023 •

edited

Loading

sadra-barikbin commented Aug 9, 2023 •

edited

Loading