Pytest - Run test from test explorer fail with uppercase test folder name

[AlexWang-516]

Environment data

• VS Code version: 1.35.1
• Extension version (available under the Extensions sidebar): 2019.6.22090
• OS and version: Windows 10 Home Edition
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.6.8
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): global python 3.6.8
• Relevant/affected Python packages and their versions: pytest 4.6.3
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi

Expected behaviour

Test file could be run from test explorer in activity bar.

Actual behaviour

Test file could not run from test explorer in activity bar.

CodeLens adornments disappear from the test file.

Steps to reproduce:

1. enable pytest from D:\projectRoot.vscode\setting.json as following:

{
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "Test"
    ]
}

2. project folder structure is as following:

D:\projectRoot\
    .vscode\
    SRC\
        src.py
    Test\
        __init__.py
        test_src.py

3. Run Test file from test explorer in activity bar

Logs

Python Test Log say that:

============================= test session starts =============================
platform win32 -- Python 3.6.8, pytest-4.6.3, py-1.5.2, pluggy-0.12.0
rootdir: d:\projectRoot
collected 0 items / 1 errors

=================================== ERRORS ====================================
______________________ ERROR collecting test/test_src.py ______________________
ImportError while importing test module 'd:\projectRoot\test\test_src.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
ModuleNotFoundError: No module named 'test.test_src'
...

Additional Information

All the setting work for the vscode-python for previous release (2019.5.18875).

When I change the folder name containing the test file from Test to test, with python.testing.pytestArgs setting in setting.json to ["test"], test file could be run from test explorer. However, codeLens adornments still disappear.

It seems for the current release, vscode-python change the uppercase letter for folder name to lowercase before they import the test module. This change to letter case cause the problem.

[karrtikr]

I am not getting similar errors, but I can confirm the tests in directory with uppercase test folder name are not being run.
Similar #6347 (comment)
Please refer to #6303 for code lenses issue. It says renaming first letter of the first dir in the workspace path to lowercase solved the issue.

[PhilMacKay]

I have a very similar issue. After updating to 2019.6.22090, every capital letter in the test module's path will be turned to lowercase when displayed in the test explorer. A quick fix could be to avoid any capital letter in any source file and folder names, but that would go against our naming convention.

When using "Run All Tests" or "Debug All Tests" buttons in the test explorer, I see in the Python Test Log output window that tests are executed (and whether they pass or not), and the status bar icon reflects the correct result. However, pass/fail results are not reported in the test explorer.

However, when I try to run or debug tests using the green "Play" arrow next to the lowercase test module name, I get a test collection error. Interestingly, when I look at the log message, it correctly capitalizes paths outside of the current workspace, but not paths inside the current workspace. For example, a test module in a workspace located in C:\Sources\ProjectName would have the following path: C:\Sources\ProjectName\TestModules\test_SomeModule.py, but when called from the test explorer, the log displays that PyTest tries to import c:\Sources\ProjectName\testmodules\test_somemodule.py

The test explorer also returns an "all lowercase" version of the file path (from "c:" to ".py") when trying to access the code from the test explorer view. I can't have any code lens on tests if there is a capital letter anywhere in the test module's full path.

Reverting to 2019.5.18875 fixes all these issues.

VS Code version: 1.36.1
Extension version (available under the Extensions sidebar): 2019.6.22090
OS and version: Windows 10 Pro 64-bit, version 1809, build 17763.557
Python version : Python 3.7.3
Type of virtual environment used : N/A (also affects venv and virtualenv)
Relevant/affected Python packages and their versions: pytest 5.0.1
Jedi or Language Server?: Jedi (both are affected)

[AnneTheAgile]

Is this windows only ?
On mac I do not think I have this issue.

[angelo-peronio]

I observe the same behaviour reported by PhilMacKay above #6410 (comment)

• VS Code version: 1.36.1
• Extension version (available under the Extensions sidebar): 2019.6.24221
• OS and version: Windows 10 Pro 64bit, version 1809, build 17763.557
• Python version: Python 3.6.7
• Type of virtual environment used: conda
• Relevant/affected Python packages and their versions: pytest 5.0.1
• Jedi or Language Server?: Language Server

[PhilMacKay]

After thorough investigation, I realized this happens with pytest version 5.0.0 and upper, and the extension's TestCollector (in pythonFiles/testing_tools/adapter/pytest/_discovery.py) which applies os.path.normcase on every path.
According to pytest's change log https://docs.pytest.org/en/latest/changelog.html#id15, bugfix pytest-dev/pytest#2761 (comment) changes the way they deal with paths on case-insensitive systems.

Basically, the workarounds I found was:

• Use most recent version of pytest and VSCode Python extension version 2019.5.18875 --> everything works as it should.
  OR
• Use most recent version of VSCode Python extension and a pytest version below 5.0.0. In that case, we suffer from bug pytest tests marked as passed or failed if test name contains upper letters #6695 and capital letters in file paths are lost, but we can run any test individually.
  OR
• Use most recent version of both VSCode Python extension and pytest, but modify the file ~/.vscode/extensions/ms-python.python-2019.X.XXXXX/pythonFiles/testing_tools/adapter/pytest/_discovery.py line 64: add an exception on windows such that NORMCASE maps to os.path.normpath instead of os.path.normcase. It solves all my problems, but I suspect it could create a bunch of problems in other situations 😏.

[ericsnowcurrently]

This should be resolved now due to #6781 and #6877. Those changes are already in the insiders build of the extension and will be part of the September release.

[ericsnowcurrently]

@PhilMacKay indicated that this is resolved. See #6820 (comment)