Pytest test discovery over SSH does not work

[sergiovalmac]

Type: Bug

Behaviour

Pytest test discovery does not find any test inside the VS Code python extension when working on a remote machine with the Microsoft SSH Remote extension. No error is reported in the OUTPUT/PYTHON window. Indeed, when I copy and paste the command from OUTPUT/PYTHON in the terminal, it finds the tests just fine. Similarly, when I run pytest --collect-only from the terminal, it finds the test just fine.
I have double checked the configuration and that all subfolders have __init__.py . Indeed, I have tried moving the test files to different places, including right in the folder (not in any subfolder) configured in settings.json and still no test is discovered.

However, it works well without the SSH remote extension.

Expected vs. Actual

I expect the tests to be discovered, or an output error to be displayed in OUTPUT/PYTHON window. However, no test is discovered and no error is displayed, as described above.

Moreover, I would expect OUTPUT/PYTHON to display the same output as when I run the command in the terminal:
~/venvs/my_repo/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear my_src_folder_with_tests

Interestingly, when I force a bug in one of the tests (like adding a random character after the brakedts in the test definition), the error is catched and displayed in OUTPUT/PYTHON. But when I fix the bug, no error is displayed as expected, but the tests are not discovered.

Steps to reproduce:

1. Connect to the remote machine over ssh and open the folder with the repo.
2. Create test_fake.py file inside the src_with_test folder configured in "python.testing.pytestArgs".
3. Be sure src_with_test has an __init__.py file, and that test_fake.py has a valid test, like:

test_one():
    assert True

4. Configure pytest (ctrl + shift + p) or in settings.json:

{
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "humanmodelling"
    ]
}

5. Open the Testing panel and click refresh discovery.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.8.0
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Extension version: 2022.14.0
VS Code version: Code 1.71.2 (74b1f979648cc44d385a2286793c226e611f59e7, 2022-09-14T21:03:37.738Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Sandboxed: No
Remote OS version: Linux x64 5.4.0-1090-azure

System Info

Item	Value
CPUs	11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (8 x 2995)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	undefined
Memory (System)	31.84GB (16.53GB free)
Process Argv	--crash-reporter-id c7209560-d4e0-4b0b-af53-21d213e1c4a6
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: sergiov-vm-nc6sv3
OS	Linux x64 5.4.0-1090-azure
CPUs	Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz (6 x 2593)
Memory (System)	110.06GB (107.62GB free)
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30557514
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vsclangdc:30486549
c4g48928:30535728
hb751961:30553087
dsvsc012:30540252
azure-dev_surveyone:30548225
2144e591:30553903
fc301958:30567733

[eleanorjboyd]

Hello, we are aware of buggy behavior on pytest discovery and running in vscode. We are currently work on a rewrite of how pytest works in vscode and hope this rewrite will fix this and similar bugs. The main issue which will be working on is here: #17242, and progress will be updated on this issue. Thank you!

[bphunter1972]

I think I have the same issue.

I wonder if this is at all related to vscode reporting that both pylint and mypy are not installed in the venv on the remote host even though they are. I say this because the output view shows no responses in vscode, but if I run these same commands in my shell they all work just fine:

> ~/.cache/pypoetry/virtualenvs/....8/bin/python ~/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear --no-cov tests
cwd: .
> ~/.cache/pypoetry/virtualenvs/....8/bin/python ~/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/linter.py -m pylint --max-line-length=120 ./tests/common_test_code.py
cwd: .
> ~/.cache/pypoetry/virtualenvs/....8/bin/python ~/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/linter.py -m mypy ./tests/common_test_code.py
cwd: .

If it helps, I used redirection to determine that all of the output is going to stdout and nothing goes to stderr.

[eleanorjboyd]

Hello! We don’t actually install pytest by default- could this be the problem that you are experiencing?

[bphunter1972]

Hello! We don’t actually install pytest by default- could this be the problem that you are experiencing?

Nope. Pytest, pylint, and mypy are all installed in the poetry env that the project points to. Run the exact same command-lines from the shell and I see the list/dict of tests.

[kdbanman]

I have the identical issue to @sergiovalmac.

Project structure:

Local, tests properly displayed

A project's tests are discovered and displayed just fine on a local VS Code window.

Remote, discovered tests NOT displayed

But the same project on a remote SSH VS Code window does not display the tests, even though most of the test discovery code path appears to work fine:

Test adapter payload JSON, remote vs local

@eleanorjboyd I am trying to diagnose the specific problem and patch it, because I would really like to interact with my tests in VS Code. I see that the test discovery results are reported via pythonFiles/testing_tools/adapter/report.py, specifically the _send call at the very end, which just prints a JSON payload to stdout.

The JSON payload appears to be functionally identical between the local and remote windows, though there are some differences.

I would like to continue tracing this execution path to see why the remote window GUI doesn't display the tests. So what happens with that JSON payload? Is stdout being consumed by some javascript/typescript somewhere?

[eleanorjboyd]

Hello! Apologies for the delay but I am unable to replicate this current issue. I have followed the steps outlined in @sergiovalmac description and was able to find all my tests successfully. I tried on a Mac to Windows connection, does everyone seeing this issue has the same remote and local machine? @kdbanman what is your remote and local machine types?

[eleanorjboyd]

@kdbanman yes standard out is being consumed. It should be consumed by this file, src/client/testing/testController/common/server.ts and from there go to the pytestController.py and then the controller.py. Let me know if this helps. We are also in the middle of a re-write for pytest which should fix this problem as well but I am unable to give an estimate on when that may be release. Thanks

[omidmeh]

I'm seeing the same thing.

Tests discovered locally and in WSL, but not in the Linux remote.

The odd thing is that the Python log shows the following command being executed which when I run directly in the shell does indeed produce results.

./.venv/bin/python <my-home>/.vscode-server/extensions/ms-python.python-2023.2.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear -c ./pytest.ini tests

[eleanorjboyd]

@omidmeh can you create a separate issue for the problem you are facing? This will just be easier to keep the conversation straight and help you specifically. Thanks

[bphunter1972]

@eleanorjboyd: Don't try Mac to Windows. Try Mac to Linux. That's where I'm seeing it as well.

[eleanorjboyd]

Sorry for the delay- running into some blockers reproducing due to VM setup policy.

[eleanorjboyd]

Could someone get their system args on their remote machine as they are attempting this? Trying to see where the cwd might have gotten messed up since that seems to be the problem. For linux running printenv.

[bphunter1972]

I cannot provide all of my env variables as that might show some proprietary information. Are there some in particular, such as PWD, that you would be interested in?
Also, is the Remote spawning the preferred shell for the user (mine is zsh), or is it spawning a standard sh or bash?

[eleanorjboyd]

Hello! Yes the PWD would be of interest. Then for remote-ssh it is spawning the standard shell for the remote when it is installing the vscode server. After the vscode server is installed it will then use the preferred shell as listed in your vscode settings.

[bphunter1972]

I'm having a hard time figuring out the best way accomplish this.

After Developer: Reload Window, I see this in the Python output:

Experiment 'pythonPromptNewToolsExt' is active
> ~/projects/__virtualenv__/imgr--NvdR-yX-py3.8/bin/python -I ~/.vscode-server/extensions/ms-python.python-2023.6.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2023.6.0/pythonFiles/interpreterInfo.py
Python interpreter path: ~/projects/__virtualenv__/imgr--NvdR-yX-py3.8/bin/python
> . ~/projects/__virtualenv__/imgr--NvdR-yX-py3.8/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode-server/extensions/ms-python.python-2023.6.0/pythonFiles/printEnvVariables.py
shell: bash
> ~/projects/__virtualenv__/imgr--NvdR-yX-py3.8/bin/python ~/.vscode-server/extensions/ms-python.python-2023.6.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear --no-cov
cwd: .
Starting Pylance language server.

As you can see, the printEnvVariables.py isn't sending anything to the output (nor is the discover pytest step).
Of course it works just fine when I run this in the terminal, whether that happens in bash or my preferred zsh.

So, how would I inject a 'printenv' into the steps above?

[eleanorjboyd]

Hi! No need to inject it- just open your terminal once the connection is set up and then run python to print the variables. Thanks

[koujaz]

Just came across this issue and thanks to this #19864 (comment) I was able to get it to work.

In my case the payload adapter also provided strange paths such as:
"source": "../../../opt/project/src/resources_test/usage_test.py"

Then I realized I'm opening the project in a directory "/root/current/devel" which is a symbolic link to "/opt/project"
on different filesystem. The above relative path is correct but not what would you expect relative to project root.

Opening the project directly in "/opt/project/" on remote machine yielded correct relative paths:
"source": "./src/resources_tests/usage_test.py"

And all tests were correctly discovered.

[bphunter1972]

Hi! No need to inject it- just open your terminal once the connection is set up and then run python to print the variables. Thanks

The PWD once the window is loaded is, as I expected, the absolute path of the workspace folder.

[gtkramer]

@koujaz you are a life saver! Confirmed that this is a bug. Steps for reproduction:

1. On a Linux machine

ln -sf /some/file/server/drive ~/drives/dev123
mkdir ~/drives/dev123/test-prj

2. On a Windows machine, open VSCode and connect to the remote Linux machine over SSH
3. Open the ~/drives/dev123/test-prj folder in VSCode
4. Open the integrated terminal and confirm that pwd is ~/drives/dev123/test-prj. Then run pytest in a virtual environment to see that it is working correctly:

python -m venv .venv
source .venv/bin/activate
pip install pytest
mkdir tests
echo -e 'def test_print():\n    print("Hello, world!")' > tests/test_example.py
pytest tests

5. Set default Python interpreter path in .vscode/settings.json to ensure it is always used everywhere:

{
"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python"
}

6. ctrl + shift + p and type "python: clear cache" and reload window
7. Configure Pytest through the VSCode UI
8. See that it discovers 0/0 tests

But, reopen VSCode at the absolute path to this folder, and suddenly everything works! Discovery and test execution works as expected.

I was going crazy. I wiped the contents of .cache, .config, .vscode, and .vscode-server on the Linux machine, reinstalled the extensions on Linux, reinstalled VSCode on Windows, tried a zillion configuration settings for the user and the remote... I even was using what @eleanorjboyd was suggesting for others to get more debug information by running python scripts out of the extension directly. Nothing seemed to be working. All I had to do was change where exactly I opened the VSCode workspace folder!

I am using VSCode 1.78.2 (the current release) and the Python extension v2023.8.0 (also the current release). This issue happens irrespective of the Python version in use (I tried 3.9, 3.10, and 3.11). I would expect the extension to work irrespective of whether VSCode has opened a workspace folder on the remote at the absolute path or not.

Just upvoted this and now fingers are crossed :)

[bphunter1972]

Whoa! That's it! I never thought to use the absolute path. Symlinks are just so natural, it never occurred to me.

Thanks so much!

[eleanorjboyd]

Hi! We are wrapping up a rewrite on testing infrastructure which is aimed at improving buggy behavior and making pytest work in a more native way. I will be testing discovery in regard to many issues we have in this space (symlinks, discovery etc) on the rewrite to see what issues still exist. If you would like to help and test your issue out on the rewrite (steps explained below) that would be most helpful in understanding the ongoing shortcomings and helping me to determine and execute the next steps. Thanks!

To test you need to be on vscode insiders and then add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. This is currently what is needed but we are in the process of switching all users to the rewrite but are doing so incrementally so if you do not have insiders, watch our release notes to get updated on when it will begin to hit stable. Thanks!

[eleanorjboyd]

Consolidating all issues on testing and symlinks to issue #9347. Let me know if you do not think this other issue is representative of the bug you are facing. Thanks!