Python Testing: Codelens above parametrized tests doesn't open dropdown for selecting tests

[phloose]

• VS Code version: 1.40.1
• Extension version: 2019.10.44104
• OS and version: Windows 10 Education 1909

Expected behaviour

Clicking on the codelens above a parametrized test function/method or a whole test class (Run Test (Multiple) | Debug Test (Multiple)) opens a dropdown where one can select the test to run/debug.

Actual behaviour

When clicking on the codelens above a parametrized test function/method or a whole test class (Run Test (Multiple) | Debug Test (Multiple)) nothing happens.

Steps to reproduce:

On windows 10 (!):
Take any python package repository that contains tests and uses pytest, discover the tests and click on the appearing codelens above a parametrized test.

Note

There is no error message displayed in the various output panels (Protocol, Python Testing Log or others). Simply nothing happens.... Very confusing.

BUT!

While trying to find out the source of the error (because i like this codelens feature for running/debugging a single test very much) i cloned the vscode-python repo and set it up to debug the extension. I found out that the current code can't find a match between fileName and item.name or item.fullPath in the method displayFunctionTestPickerUI from the class TestDisplay. In my case fileName had the value c:\somepath\some_test_file.py and item.fullPath C:\somepath\some_test_file.py. The mismatch was the drive letter. When the drive letter is set to uppercase in fileName it works as expected and the dropdown menu is shown. Otherwise testFile is undefined an the method returns.

Maybe this helps sorting out this bug. I did not find out exactly where the path from tests.TestFiles[]/item.fullPath is generated but after some investigation i assume that it comes from the discover python module in the folder pythonFiles.

Below the source code of that method:

public displayFunctionTestPickerUI(cmdSource: CommandSource, wkspace: Uri, rootDirectory: string, file: Uri, testFunctions: TestFunction[], debug?: boolean) {
        const tests = this.testCollectionStorage.getTests(wkspace);
        if (!tests) {
            return;
        }
        const fileName = file.fsPath;
        const testFile = tests.testFiles.find(item => item.name === fileName || item.fullPath === fileName);
        if (!testFile) {
            return;
        }
        const flattenedFunctions = tests.testFunctions.filter(fn => {
            return fn.parentTestFile.name === testFile.name &&
                testFunctions.some(testFunc => testFunc.nameToRun === fn.testFunction.nameToRun);
        });

        this.appShell.showQuickPick(buildItemsForFunctions(rootDirectory, flattenedFunctions, undefined, undefined, debug),
            { matchOnDescription: true, matchOnDetail: true })
            .then(testItem => testItem ? onItemSelected(this.commandManager, cmdSource, wkspace, testItem, debug) : Promise.resolve());
    }

Worth to mention that this works as expected when using vscode-python inside WSL.

[ericsnowcurrently]

@phloose, thanks for letting us know about this and for being so proactive about finding a solution. Just to be sure I've understood, the problem is that the code lens tries to run a test with the wrong casing (for the drive letter)? This isn't specific to parameterized tests, is it?

Also, popping up a picker UI for parameterized tests sounds more like a feature request. I recommend opening a separate issue for that one.

[DonJayamanne]

popping up a picker UI for parameterized tests sounds more

This is how it used to behave (displaying a pocket for parameterized tests). Not sure what had changed in the extension.

[phloose]

the problem is that the code lens tries to run a test with the wrong casing (for the drive letter)?

The problem i discovered is that the listed code tries to match a path (returned from Uri.fsPath) for an actual testfile, in which a codelens is used on a parametrized test, with a path for a testfile which was build during test collection. Thereby Uri.fsPath returns a lowercase drive letter for a windows path (like described in the api docs) while the path generated during test collection has an uppercase drive letter. The rest of both path is exactly the same.

This isn't specific to parameterized tests, is it?

This applies ONLY to parametrized tests/tests that have a codelens that shows Run Test (Multiple), other tests work as expected.

This is how it used to behave (displaying a pocket for parameterized tests). Not sure what had changed in the extension.

Yeah i also remember that it worked correctly. In WSL/Linux and MacOS it works as expected.

[ericsnowcurrently]

Ah, thanks for clarifying. I was able to reproduce the problem using parameterized tests (only on WIndows). This is a know issue with VS Code. In the meantime, I expect the extension code needs to use FileSystem.arePathsSame() (e.g. in displayFunctionTestPickerUI).

---

FWIW, here's what I tried, with the following code in a top-level file:

import pytest

@pytest.mark.parametrize('x,y,expected', [(1, 1, 2), (2, 3, 5)])
def test_add(x, y, expected):
    assert x + y == expected

Linux

Test discovery worked fine and the codelens showed up. When I clicked on Run Test (Multiple), a picker popped up with the 2 parameterized values as options. Selecting one of them ran the tests successfully.

Windows

Test discovery worked fine and the codelens showed up. When I clicked on Run Test (Multiple), nothing happened.

[phloose]

Test discovery worked fine and the codelens showed up. When I clicked on Run Test (Multiple), nothing happened.

Exactly as i discovered. This happens ONLY on Windows.

For a possible fix look at my pull request #8647. It doesn't use that method (which just uppercases all path elements for the comparison when the platform is windows) but cuts of only the drive letter and uppercases that. There i should also add that this should only be done if the path is NOT relative, otherwise strange thinks happen...

[ericsnowcurrently]

I've left a review on your PR.