metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations

[kamichal]

Both, python 2.7.13 and 3.5.3, pytest 3.6.1, no additional pytest plugins.

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}
def pytest_generate_tests(metafunc):
    if 'tested_branch' in metafunc.fixturenames:
        branches = KNOWN_BRANCHES.keys()
        metafunc.parametrize("tested_branch", branches, scope="session")

@pytest.fixture(scope="session")
def repo_preparation(tested_branch):
    """ If it would have a "session" scopoe then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[tested_branch] += 1
    return KNOWN_BRANCHES[tested_branch]

And the simplest reproduction:

# test_parametrization.py
import pytest

def test_nothing(repo_preparation):
    assert repo_preparation == 1  # that passes

@pytest.mark.parametrize("_", ["param_1", "param_2"])
def test_basic_parametrization(repo_preparation, _):
    # that causes parametrization confisuon
    assert repo_preparation == 1, "Repo prepared more than once."

Gives such an result:

$ pytest -v test_parametrization.py 
================================== test session starts ==================================
platform linux -- Python 3.5.3, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/kamichal/ecws/pyplayground, inifile:
collected 6 items                                                                       

test_parametrization.py::test_nothing[branch_1] PASSED                            [ 16%]
test_parametrization.py::test_basic_parametrization[branch_1-param_1] PASSED      [ 33%]
test_parametrization.py::test_nothing[branch_2] PASSED                            [ 50%]
test_parametrization.py::test_basic_parametrization[branch_1-param_2] PASSED      [ 66%]
test_parametrization.py::test_basic_parametrization[branch_2-param_1] FAILED      [ 83%]
test_parametrization.py::test_basic_parametrization[branch_2-param_2] FAILED      [100%]

======================================= FAILURES ========================================
_____________________ test_basic_parametrization[branch_2-param_1] ______________________

repo_preparation = 2, _ = 'param_1'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 2 == 1

test_parametrization.py:12: AssertionError
_____________________ test_basic_parametrization[branch_2-param_2] ______________________

repo_preparation = 3, _ = 'param_2'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 3 == 1

test_parametrization.py:12: AssertionError
========================== 2 failed, 4 passed in 0.03 seconds ===========================

Setting indirect=True in the metafunc.parametrize call fixes that problem, but there is no information that it's required for larger scopes to work properly. There is only an advice to do so, but...

[pytestbot]

GitMate.io thinks possibly related issues are #634 (metafunc.parametrize overwrites scope), #519 (fixture scope is ignored when using metafunc.parametrize()), #570 (indirect=True in parametrize breaks fixture scopes), #244 (parametrize fails when values are unhashable), and #1111 (pytest.mark.parametrize fails with lambdas).

[RonnyPfannschmidt]

that looks like a bug

note that you can use fun workaround

since you already know your parameters,

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}

@pytest.fixture(scope="session", param=KNOWN_BRANCHES.keys())
def repo_preparation(request):
    """ If it would have a "session" scope then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[request.param] += 1
    return KNOWN_BRANCHES[request.param]

might do (untested)

[kamichal]

Thank you Ronny. I know the "workaround" very well.
The point is that I would like to take the tested_branch parameters from parser.addoption.
So, contents of that list is variable in my case, that's why I took pytest_generate_tests hook.

[RonnyPfannschmidt]

@kamichal i see - in that case - there is a deeper underlying issue that i wont be investigating in near future - i have no idea if any of the other core devs has other plans

[winklerrr]

I came across the same bug, I think...
When using metafunc.parametrize(scope=session) in combination with parametrized tests, all of my session scoped fixtures teared down after each test.

This behavior can be demonstrated very vividly with --setup-show:

SETUP    S session_fixture1 # <-- This fixture is generated with pytest_generate_tests
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test1()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2 
TEARDOWN S session_fixture1 # <-- Session scoped fixture gets tear downed
SETUP    S session_fixture1 
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test2()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2
TEARDOWN S session_fixture1

Workaround

@kamichal For me, the workaround with indirect=True also works in combination with parameters received from metafunc.config.getoption.

Just define a real fixture with the same name (in the example below my_fixture) and use the request fixture to return the indirect parametrized value:

def pytest_addoption(parser):
    parser.addoption("--my_parser_option")
    
def pytest_generate_tests(metafunc):
    if "my_fixture" in metafunc.fixturenames:
        metafunc.parametrize("my_fixture", metafunc.config.getoption("my_parser_option"), scope="session", indirect=True)

@pytest.fixture(scope="session")
def my_fixture(request):
    return request.param

def test_fixture(my_fixture):
    assert my_fixture == "world"

Then you can call it with pytest --my_parser_option hello, which will result in "hello" != "world"
I hope this helps someone.

Regards
Winklerrr

[RonnyPfannschmidt]

thanks for providing the workaround,
this also means we can have a better way of understanding what is wrong

as its limited to freestanding parameterization

[PShiw]

I came across the same bug, I think...
When using metafunc.parametrize(scope=session) in combination with parametrized tests, all of my session scoped fixtures teared down after each test.

This behavior can be demonstrated very vividly with --setup-show:

SETUP    S session_fixture1 # <-- This fixture is generated with pytest_generate_tests
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test1()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2 
TEARDOWN S session_fixture1 # <-- Session scoped fixture gets tear downed
SETUP    S session_fixture1 
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test2()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2
TEARDOWN S session_fixture1

Workaround

@kamichal For me, the workaround with indirect=True also works in combination with parameters received from metafunc.config.getoption.

Just define a real fixture with the same name (in the example below my_fixture) and use the request fixture to return the indirect parametrized value:

def pytest_addoption(parser):
    parser.addoption("--my_parser_option")
    
def pytest_generate_tests(metafunc):
    if "my_fixture" in metafunc.fixturenames:
        metafunc.parametrize("my_fixture", metafunc.config.getoption("my_parser_option"), scope="session", indirect=True)

@pytest.fixture(scope="session")
def my_fixture(request):
    return request.param

def test_fixture(my_fixture):
    assert my_fixture == "world"

Then you can call it with pytest --my_parser_option hello, which will result in "hello" != "world"
I hope this helps someone.

Regards
Winklerrr

Worked perfectly in my case. Thanks for the workaround.

[rassie]

I'm sorry to be nagging, but is this something that is going to be addressed someday? Is this confirmed as a bug?

[The-Compiler]

I'm sorry to be nagging, but is this something that is going to be addressed someday?

As usual with open source projects, it will be addressed once someone feels like working on it (or someone pays someone to work on it).

Is this confirmed as a bug?

@RonnyPfannschmidt (a pytest core maintainer) said "that looks like a bug" above.