Folders named 'py' result in "Test result not found"

[botsman]

Environment data

• VS Code version: 1.60.0
• Extension version (available under the Extensions sidebar): v2021.9.1230869389
• OS and version: MacOS 11.6
• Python version (& distribution if applicable, e.g. Anaconda): 3.9.1
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pyenv
• Relevant/affected Python packages and their versions: pytest==6.2.5
• Value of the python.languageServer setting: Pylance

Expected behaviour

This simple test should appear succeeded in UI :

def test_passing():
   assert 42 == 42

Actual behaviour

The test is failed (red circle) with this reason :

Steps to reproduce:

The issue is very similar to #17270 which is closed by #17275

The only difference that in my case folder name is not starting with "py", but the whole folder name is "py":

1. Have pytest installed and selected as a default test framework
2. Put tests inside folder named "py". For example, "./tests/py/"
3. Try to execute test
4. The test is executed correctly, but test UI shows error.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~/.pyenv/versions/3.9.1/envs/vscode_python_pytest/bin/python -m pytest --rootdir ~/projects/vscode_python_pytest --override-ini junit_family=xunit1 --junit-xml=/var/folders/0_/6w94lk1171vc_fpwbjqx02nw0000gn/T/tmp-14897brlMlEUuYALH.xml ./tests/py/test_something.py::test_passing
cwd: ~/projects/vscode_python_pytest

[karthiknadig]

This is a very specific case with folders named py. This is a problem due to the way the test runner generates test ids. We plan on addressing this in the test adapter re-write.

[botsman]

One more thing. It looks like parameter --rootdir in the python.testing.pytestArgs also affects behavior of test execution from the test explorer UI.

Just want remind that it should be taken into account.

1. If I use

"python.testing.pytestArgs": [
        "./tests/py"
    ]

then tests are discovered, but if I try to execute those, then I am getting a message in Python Test Log: ERROR: file or directory not found: ... and the test I clicked continues to be displayed as gray (not executed)

2. If I use

"python.testing.pytestArgs": [
        "--rootdir",
        "./tests/py"
    ]

then test are discovered, but I am getting an error described in the first message in this issue: Test result not found for: ... and the test I clicked is turning red (as on the screenshot in the first post)

[hmrd-forpeople]

Is there a workaround for this (other than renaming directories) until the fix/rewrite is released? Or are we stuck with a broken test runner until then?

[karthiknadig]

@hszw-forpeople I recommend using an older version of the extension 2021.8.*, if you cannot change the name.

[botsman]

Just a note, probably related to #17242

[ianjirka]

Can confirm - renaming my directory from 'py' to '_py' works around this issue.

[eleanorjboyd]

Hello! We have just finished our testing rewrite and are beginning the roll out to users. I have tested this issue with the re-write and I am not longer reproducing the bug! If you would like to try it yourself, you need to be on vscode insiders and then add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. We are in the processes of switching all users to the rewrite but are doing so incrementally so if you do not have insiders, watch our release notes to get updated on when it will begin to hit stable. Let me know if it doesn’t work for you and we can re-open this issue. Thanks!