pytest rootdir should be where configured

[slobinger]

Environment data

• VS Code version: Version: 1.46.1
  Commit: cd9ea6488829f560dc949a8b2fb789f3cdc05f5d
  Date: 2020-06-17T21:13:08.304Z
  Electron: 7.3.1
  Chrome: 78.0.3904.130
  Node.js: 12.8.1
  V8: 7.8.279.23-electron.0
  OS: Linux x64 5.3.0-59-generic
• Extension version (available under the Extensions sidebar): 2020.6.89148
• OS and version: Ubuntu 18.04.4 LTS
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.6.9 installed via Ubuntu package repo
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Relevant/affected Python packages and their versions: pytest==5.1.1
• Relevant/affected Python-related VS Code extensions and their versions: None
• Value of the python.languageServer setting: "Microsoft"

Expected behaviour

If I have the following directory structure:

project_folder/
|----.vscode
|     |----settings.json
|----app_src/
|     |----config/
|     |      |----settings.yml
|     |----package_a/
|     |      |----python files ...
|     |----package_b/
|     |      |----python files....
|     |----tests/
|     |      |----test_a.py
|     |      |----test_b.py
|     |      |----pytest.ini
|     |----app.py
|----Dockerfile
|----README.md
|----other meta files....

And in the .vscode/settings.json the following config for pytest:

"python.testing.pytestArgs": [
        "-W ignore::DeprecationWarning:invoke.loader",
        "-s",
        "--rootdir=app_src/test"
],
"python.testing.pytestEnabled": true,
"python.testing.cwd": "app_src/test"

I would expect the working directory is app_src/test relative to project root, same for pytest rootdir.
I also expect the the extra arg for rootdiris not needed because the described behavior in pytest for the rootdir is, that it is there the pytest.ini is found.
With specified working directory I would expect that any tests found in test directory can manually be executed via sidebar.

Actual behaviour

But this is not the case. I get a file not found error:

============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.1.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/foo/workspace/project_folder, inifile: app_src/test/pytest.ini
plugins: flask-0.15.0
collected 0 items

-------------- generated xml file: /tmp/tmp-26489TSNlz2Q1AYNq.xml --------------
============================ no tests ran in 0.01s =============================
ERROR: file not found: ./app_src/test/test_a.py

Steps to reproduce:

create a project with the described structure and configuration and try to run single tests or testsfiles within sidbar.

suggestion

I would suggest that the pytest integration in vscode follows as exactly as possbile the behavior of pytest itself.
It should alway work to execute the discovered testfiles if absolute paths are used. For Test discovery it is a musst have to respect the --rootdir argument, so that the discovery does not fail because of errors in imports of no-test-modules.
If --rotdir is not respected it musst be documented, that it is useless to set this argument.

At the moment I have only to workarounds:

1. execute pytest from cli
2. restructure my project so that vscode can handle what pytest on cli already can.

I prefer 1. 😄 .

[ericsnowcurrently]

@slobinger, thanks for letting us know about this. We'll take a look at what we can do to make the pytest behavior more behavior when you have --rootdir and/or the "python.testing.cwd" setting. In the meantime, please provide the content of the "Python" and "Python Testing Log" output panels.

[ericsnowcurrently]

@luabud, this is consistently a point of confusion for users with less-common project structures.

[slobinger]

As requested, to bring a little more detail in, if configuration of a project is as follow:

//....
"python.testing.pytestArgs": [
        "-W ignore::DeprecationWarning:invoke.loader",
        "-s",
        "--rootdir=."
 ],
 "python.testing.unittestEnabled": false,
 "python.testing.nosetestsEnabled": false,
 "python.testing.cwd": "app_src/test"
//...

discover tests and run one of the founded tests via sidebar, I get the following Test Log:

python /home/foo/.vscode/extensions/ms-python.python-2020.6.91350/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /home/foo/workspace/project --cache-clear -W ignore::DeprecationWarning:invoke.loader -s .
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.8.0, pluggy-0.12.0
rootdir: /home/foo/workspace/project/app_src/test, inifile: ../pytest.ini
collected 0 items

-------------- generated xml file: /tmp/tmp-5151qPoVyE1nupA6.xml ---------------
============================ no tests ran in 0.02s =============================
ERROR: file not found: ./app_src/test/test_module_a.py::test_echo_test

python log:

> ~/workspace/project/venv/bin/python ~/.vscode/extensions/ms-python.python-2020.6.91350/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/workspace/project --cache-clear "-W ignore::DeprecationWarning:invoke.loader" -s .
cwd: ~/workspace/project/app_src/test
> ~/workspace/project/venv/bin/python ~/.vscode/extensions/ms-python.python-2020.6.91350/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/workspace/project --junit-xml=/tmp/tmp-5151edcU85seW1lZ.xml "-W ignore::DeprecationWarning:invoke.loader" -s --rootdir=. ./app_src/test/test_module_a.py::test_echo_test
cwd: ~/workspace/project/app_src/test

@ericsnowcurrently just a little OT question. When you say "less-common project structures", which project structure would you suggest?

[ericsnowcurrently]

Thanks. That helps. Note that the only thing the "python.testing.cwd" setting does is set the current working directory for the spawned Python process that will discover/run the tests. "--rootdir=." should provide the right value to pytest normally, though you really shouldn't need it. (See the pytest docs about the "rootdir".)

I expect part of the problem here is with the value you used for the "python.testing.cwd" setting. What happens if you set "python.testing.cwd" to "app_src" instead of "app_src/test" and drop "--rootdir=."? If that doesn't help, what about if you use "--rootdir=./test" (and keep "cwd" as "app_src")? Likewise with "--rootdir=."?

[ericsnowcurrently]

just a little OT question. When you say "less-common project structures", which project structure would you suggest?

I hope that didn't sound like a critique! 😄 It wasn't meant to. In the Python extension we expect (correctly, we believe) that most projects are relatively flat, such that the project root directory is the one that would go on sys.path, like this:

project_folder/
|----my_package/
|      |----python files ...
|----tests/
|      |----test_a.py
|      |----test_b.py
|---- ...

However, there's nothing wrong with the structure you have. It's just less common so we don't accommodate it as well in the extension. We'd like to improve that. Helping you with this issue actually helps that cause. 😄

[slobinger]

Thanks. That helps. Note that the only thing the "python.testing.cwd" setting does is set the current working directory for the spawned Python process that will discover/run the tests. "--rootdir=." should provide the right value to pytest normally, though you really shouldn't need it. (See the pytest docs about the "rootdir".)

Yes, that is, what I expact, too. And the reason I set the "python.testing.cwd" is that the configuration library we use (Dynaconf) requires that for our file structure. But sadly even if I omit --rootpath and change the working dir to /app_src I get the same error message on Test execution.

In contrast I can run pytest from CLI without any problem.

Maybe the fix for #6548 is not that good because it seems to force the rootdir to be the project path. meaning all relative test file paths musst be relative to project root. But that is not the case if one changes python.testing.cwd.

I hope that didn't sound like a critique!

Oh no, I just was intrested - have seen so many different file structures for python projects. The one mentioned by you is definitly a very common one. We just desided for our project that it would makes our Dockerfile more complicated 😄 .

[ericsnowcurrently]

I'm not sure that #6548 is exactly the same thing, though it is definitely at least closely related. Regardless, please provide the content of the "Python" and "Python test log" output panels. Both will offer some clues as to what's going on, like the exact command and CWD used to run pytest, and the pytest output text. Thanks!

[ericsnowcurrently]

FWIW, I expect there is a bug in the extension that we'll need to fix. We just need to narrow down the problem.

[brettcannon]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on then we will be happy to re-open this issue to pick up where we left off.

[hsharrison]

Sorry to resurrect an old issue, but I think I'm having the same problem.

My project structure is not the most common one, but not so unusual either, I think.

<project-root>/
|----src/
|    |----<package>/ 
|    |    |----__init__.py
|    |    |----...
|----tests/
|    |----__init__.py
|    |----conftest.py
|    |----test_xxxx.py
|    |----...
|----setup.cfg

setup.cfg contains the pytest settings such that simple running pytest from <project-root> works perfectly.

However, when I try to discover the tests in vscode, I get this error:

During handling of the above exception, another exception occurred:
/opt/conda/envs/<env-name>/lib/python3.7/site-packages/_pytest/config/__init__.py:446: in _importconftest
    mod = conftestpath.pyimport()
/opt/conda/envs/<env-name>/lib/python3.7/site-packages/py/_path/local.py:721: in pyimport
    raise self.ImportMismatchError(modname, modfile, self)
E   py._path.local.LocalPath.ImportMismatchError: ('tests.conftest', '/home/<user>/src/<another-project-dir>/tests/conftest.py', local('/home/<user>/src/<another-project-dir>/tests/conftest.py'))

It's trying to load conftest.py from another project!! I've confirmed that this project is not installed in the conda env or accidentally cached anywhere (everywhere I could think to check). The project I'm working on has no reason to know that this project exists.

From the output I found this command: python /home/<user>/.local/share/code-server/extensions/ms-python.python-2020.5.86806/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir <project-path> -s

If I manually run this command in my project root, it works perfectly. If I run it from my home directory, I replicate the problem where it tries to load the wrong project.

I tried setting python.testing.cwd to <project-path> but it didn't change anything.

Am I encountering the same problem as this issue or not? It's hard to tell. Any advice is appreciated.

[brettcannon]

@hsharrison can you open a new issue to make this easier to track?

[hsharrison]

@hsharrison can you open a new issue to make this easier to track?

Sure thing, thanks. Opened #14795 .