pytest.mark.parametrize should not trigger multiple pytest instances

[lephuongbg]

Environment data

• VS Code version: 1.60.0
• Extension version: v2021.9.1191016588
• OS and version: KDE Neon 20.04
• Python version (& distribution if applicable, e.g. Anaconda): 3.8
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Value of the python.languageServer setting: pylance

Expected behaviour

Using the test runner, running test with parametrize mark should only use one pytest instance.

Actual behaviour

Test runner runs each param group in a separate pytest instance. In our specific case, since there is database fixture setup in the test, multiple pytest instances will call same queries at the same time, causing conflicts.

Steps to reproduce:

@pytest.mark.parametrize(
    "input_,expected",
    (
        (1, "AAAAAAAAAAAAC"),
        (706940086461138253, "BHHY4Q23AACU2"),
    ),
)
def test_parametrize_bug(input_, expected):
    pass

Logs

Output for Python Test Log: Note there are two sessions started in the log, each run one parameter group

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.9.0, pluggy-0.13.1 -- /var/app/api/.venv/bin/python
cachedir: .pytest_cache
rootdir: /var/app/api, configfile: pyproject.toml
plugins: xdist-2.3.0, forked-1.1.3, asyncio-0.15.1, anyio-3.2.1, Faker-8.12.1, cov-2.12.1
collecting ... ============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.9.0, pluggy-0.13.1 -- /var/app/api/.venv/bin/python
cachedir: .pytest_cache
rootdir: /var/app/api, configfile: pyproject.toml
plugins: xdist-2.3.0, forked-1.1.3, asyncio-0.15.1, anyio-3.2.1, Faker-8.12.1, cov-2.12.1
collecting ... collected 1 item

tests/migrations/test_base32.py::test_parametrize_bug[1-AAAAAAAAAAAAC] collected 1 item

tests/migrations/test_base32.py::test_parametrize_bug[706940086461138253-BHHY4Q23AACU2] PASSED [100%]

-------------- generated xml file: /tmp/tmp-45586HQIhfvtleXdn.xml --------------
============================== 1 passed in 0.84s ===============================
PASSED [100%]

-------------- generated xml file: /tmp/tmp-45586UGX3hxbELmeb.xml --------------
============================== 1 passed in 0.85s ===============================

[paulacamargo25]

Thanks for the bug report! We just wanted to quickly acknowledge we received it and we will triage this as soon as we can.

[zztalker]

The same issue, if parameters too much it may hang for a while (as in #17396)

[karthiknadig]

When you click the gutter to run, VS Code creates a run request with each parametrized test added as a separate item. This causes pytest to be triggered separately for each node, currently we don't reduce the number of nodes we run. See the (384) nodes in the include list when run from the gutter.

You can run the parent node for the parametrized tests from the test explorer and it should run only one instance, as a work around for now. See that there is only one node in the include list:

We can address this issue to reduce the number of things we run when implementing #16929 . Essentially if all the child nodes for a test are in the include list, then we should just run the parent of those nodes.

This issue is not related to #17396, which is fixed in the insiders version of VS Code. #17396 occurred when VC code was attempting to update the UI for large number of tests, after the python extension provided the results from running the tests.

[zztalker]

I've got it. Thank you. Do you have ETA for #16929 ? I have to stick to "v2021.8.1159798656" of the extension because we have a huge amount of parameterized tests in our repo.

[karthiknadig]

We are working on replacing the test adapter for pytest, we will be addressing #16929 and a bunch of other issues as a part of this. This work is planned as one of the next items to tackle to improve overall testing experience. It is on the current iteration plan, but we don't have ETA yet.

[MetRonnie]

This was causing me grief today. Not only does it trigger multiple pytest instances, it invokes conda run for each one. Not only that but I am using pytest-xdist and have specified using 5 processes in the pytest args setting. So for a test that has dozens of cases, it would spin up conda run and 5 xdist processes for each, absolutely tanking my computer

[james10424]

Also encountered this today, it's running multiple "sessions" in parallel, causing multiple session-scoped fixtures to run. My session fixtures are not able to run in parallel. Please provide a way to disable this behaviour.

[eleanorjboyd]

closing in preference of this new issue which I have engaged on recently: #21529. Please comment on that issue as we are trying to determine desired behavior. I have responded to both the concern regarding tests running in parallel and the individual run requests per parameterized tests. Thank you!!