Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion: add new 'parametrization' fixture scope #1552

Open
csaftoiu opened this issue May 6, 2016 · 10 comments
Open

Suggestion: add new 'parametrization' fixture scope #1552

csaftoiu opened this issue May 6, 2016 · 10 comments
Labels
status: help wanted developers would like help from experts on this topic topic: parametrize related to @pytest.mark.parametrize type: backward compatibility might present some backward compatibility issues which should be carefully noted in the changelog type: enhancement new feature or API change, should be merged into features branch type: feature-branch new feature or API change, should be merged into features branch type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature

Comments

@csaftoiu
Copy link

csaftoiu commented May 6, 2016

Currently, a fixture can be scoped as function, class, module, or session.

I propose a fixture can also be scoped per parametrization.

Consider:

import time
@pytest.fixture
def slow_setup():
    time.sleep(1)
    return 'foo'

If I have a group of tests which I am parametrizing, but that would share this value, it'll do the setup for each one (as expected):

# takes 4 seconds
@pytest.mark.parametrize('bar', [1, 2, 3, 4])
def test_parametrized(slow_setup, bar):
    assert slow_setup == 'foo'

If I wanted to run the same group of tests but sharing the fixture, I only really have one option (considering that other tests in the same module need the fixture to be re-set up each time, so I can't use module or session scope): set the fixture scope to class, and put the tests inside a class:

@pytest.fixture(scope='class')
def slow_setup_cls():
    time.sleep(1)
    return 'foo'

# takes 1 second
class TestParametrized(object):
    @pytest.mark.parametrize('bar', [1, 2, 3, 4])
    def test_parametrized(self, slow_setup_cls, bar):
        assert slow_setup_cls == 'foo'

However, the meaning I really intend to convey is "this fixture should be shared across all parametrizations of a function". I propose the following:

@pytest.fixture(scope='parametrization')
def slow_setup_pmt():
    time.sleep(1)
    return 'foo'

# takes 1 second
@pytest.mark.parametrize('bar', [1, 2, 3, 4])
def test_parametrized(self, slow_setup_pmt, bar):
    assert slow_setup_pmt == 'foo'

This fixture scope conveys, to me: "All parametrizations of this test share the same setup and teardown with regards to this fixture."

@RonnyPfannschmidt
Copy link
Member

i like the idea

currently the proposal has 2 semantical problems

  1. there is no collection element for the function that is not parameterized (scoping selects the collection tree element based on a type belonging to the
  2. parameterization can happen on more than one scope ( class/module/session scoped fixtures may be parameterized as well

@RonnyPfannschmidt RonnyPfannschmidt added type: enhancement new feature or API change, should be merged into features branch status: help wanted developers would like help from experts on this topic type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature topic: parametrize related to @pytest.mark.parametrize type: backward compatibility might present some backward compatibility issues which should be carefully noted in the changelog type: feature-branch new feature or API change, should be merged into features branch sprint-candidate labels May 31, 2016
@SalmonMode
Copy link
Contributor

Why not just have parametrized fixtures that have autouse as True (factoring in dependencies, of course) make multiple versions of the scopes they are designated for?

For example, currently, given the following module:

import pytest


@pytest.fixture(scope="class", autouse=True)
def before_param():
    print("in before_param")


class TestIt():

    @pytest.fixture(scope="class", autouse=True, params=["a", "b"])
    def param_fix(self, before_param, request):
        print("in param_fix")
        return request.param

    @pytest.fixture(scope="class", autouse=True)
    def other_fix(self, param_fix):
        print("in other_fix")

    def test_stuff(self, other_fix):
        assert True

You get the following flow:

test_module (module)
└── TestIt (class)
    └── before_param
        ├── param_fix["a"]
        |   └── other_fix
        |       └── test_stuff (function)
        └── param_fix["b"]
            └── other_fix
                └── test_stuff (function)

But since autouse is True, the whole of that scope should be impacted. That, to me, would mean that I want to go through that scope, in it's entirety, once for each param set, as each set should impact the whole scope on its own. Here's the flow I would expect in this case:

test_module (module)
├── TestIt["a"] (class)
|   └── before_param
|       └── param_fix
|           └── other_fix
|               └── test_stuff (function)
└── TestIt["b"] (class)
    └── before_param
        └── param_fix
            └── other_fix
                └── test_stuff (function)

In this case, param_fix made two versions of the TestIt class, one for each param set.

If autouse is not True, then just only make multiple versions of the individual tests that depend on it.

This covers @csaftoiu's example with pytest.mark.parametrize without adding a need to use another argument or alternate scope options.

@SalmonMode
Copy link
Contributor

So after doing some digging into pytest, it looks like my initial idea of having multiple versions of each scope would pretty much require a total rework of test collection/execution. But I did find what I think is a good place to make a small change so that autouse parametrized fixtures cause other fixtures for that scope to re-execute as well.

I'm writing up a pull request now to put in this change. If it's decided to not go this route, and instead add a new argument/scope to trigger this kind of behavior, I think the changes I've made so far can be easily modified to do that instead.

@toby-w
Copy link

toby-w commented Dec 14, 2018

i like the idea

currently the proposal has 2 semantical problems

  1. there is no collection element for the function that is not parameterized (scoping selects the collection tree element based on a type belonging to the
  2. parameterization can happen on more than one scope ( class/module/session scoped fixtures may be parameterized as well

Hi @RonnyPfannschmidt, I am not clear on the two semantical issues that adding a parametrization scope would raise. Could you provide an example of how this would be a problem?

@RonnyPfannschmidt
Copy link
Member

conftest.py:

@pytest.fixtur(scope="param", params=["x", "y"])
def param_fix(fun):
   pass

test_a.py:

@pytest.fixture(scope="session", params=[1,2])
def fun():
  pass

@pytest.mark.parametrize("b", ["a", "b"], scope="module")
def test_param(fix_param, b):
  pass

what should be the order of tests and the setup/teardown in that case

@Sup3rGeo
Copy link
Member

If I understand it correctly, it means that in terms of levels, it should be session > module > class > parameterization > function?

Then the example would not be valid @RonnyPfannschmidt , as test_param has a module scope, depending on a parametrization scope.

--

Actually the function now is in fact every single test case, and this parametrization scope would be in fact one per function definition.

In fact I think it would be more accurately defined as:

  • session (same fixture instance if its the same pytest run)
  • module (same fixture instance if its the same python module)
  • class (same fixture instance if its the same python class)
  • function (same fixture instance if its the same python function) - this would be the proposed here, although of course the name is already used to mean the following lower level.
  • None (always recreate the fixture instance) - the current pytest function scope

@SalmonMode
Copy link
Contributor

@toby-w @Sup3rGeo I think what @RonnyPfannschmidt is referring to, is the fact that you can apply a parametrization within multiple scopes (i.e. session/package/module/class/function), and having a "param" scope doesn't specify which level of parametrization you would want to target.

For example:

@pytest.fixture(scope="class", params=["a", "b"], autouse=True)
def some_class_lvl_fix(request):
    return request.param

@pytest.fixture(scope="param", autouse=True)
def some_param_lvl_fix():
    return something

class TestSomething():
    @pytest.fixture(scope="class", params=["x", "y"], autouse=True)
    def another_cls_lvl_fix(self, request):
        return request.param
    @pytest.fixture(scope="function", params=[1, 2])
    def some_test_param_fix(self, request):
        return request.param
    def test_param(self, some_test_param_fix):
        # test something while parametrized by some_test_param_fix
    def test_not_param(self):
        # test something else while not parametrized by some_test_param_fix

The question is what level of parametrization are you targeting with some_param_lvl_fix? Parametrization is happening at several points, and at multiple levels, but a scope of param is not specific enough to say where you want to actually have this fixture reside in the order of operations. You're right that the intention with what @csaftoiu wanted was just to address that weird scope for functions where parametrization doesn't behave the same way as larger scopes do (i.e. parametrization happening after a fixture of the same scope doesn't trigger re-execution of the prior fixture), but "param"/"parametrization" just isn't specific enough of a term.

I also don't think @csaftoiu's proposal would actually solve the problem, because a scope of "param"/"parametrization" would suggest that it happens for each parameter set, meaning it would still run the slow running fixture more than once. Granted, the "param" (or whatever it would be called) scope of that type could be used to go back to the current behavior, while the standard "function" level could be made to operate like the larger scopes. But that would likely break a lot of previously written tests, so it's a lose-lose situation.

@Sup3rGeo
Copy link
Member

Sup3rGeo commented Mar 16, 2019

@SalmonMode yep, so what I wrote was my understanding of the feature @csaftoiu described in this issue, which I am convinced is different from what @RonnyPfannschmidt understood.

As I understand, the OP wants to have this:

@pytest.fixture
def slow_setup():
    time.sleep(1)
    return 'foo'

@pytest.mark.parametrize('bar', [1, 2, 3, 4])
def test_parametrized_one(slow_setup, bar):
    assert slow_setup == 'foo'

@pytest.mark.parametrize('bar', [5, 6, 7, 8])
def test_parametrized_two(slow_setup, bar):
    assert slow_setup == 'foo'

With only two fixture invocations:

  • one for test_parametrized_one (and all its [1] [2] [3] [4] instances);
  • and also one for test_parametrized_two (and all its [5] [6] [7] [8] instances);

Currently, if the slow_setup scope is function, you are going to have 8 invocations. If it is module, then you are going to have only 1 invocation.

Thats why he uses the workaround of having classes:

@pytest.fixture(scope="class")
def slow_setup():
    time.sleep(1)
    return 'foo'

class one:
    @pytest.mark.parametrize('bar', [1, 2, 3, 4])
    def test_parametrized_one(self, slow_setup, bar):
        assert slow_setup == 'foo'

class two:
    @pytest.mark.parametrize('bar', [5, 6, 7, 8])
    def test_parametrized_two(self, slow_setup, bar):
        assert slow_setup == 'foo'

Then we get the desired two invocation scenario.

@maxrothman

This comment was marked as off-topic.

@Wenzel

This comment was marked as spam.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: help wanted developers would like help from experts on this topic topic: parametrize related to @pytest.mark.parametrize type: backward compatibility might present some backward compatibility issues which should be carefully noted in the changelog type: enhancement new feature or API change, should be merged into features branch type: feature-branch new feature or API change, should be merged into features branch type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature
Projects
None yet
Development

No branches or pull requests

7 participants