[Test UI] Feature Request: Different Organization Options

[JustinGrote]

@connor4312
Related to: #126932

There may be different ways a user wants to view their tests, such as sorted by tag/function or sorted by unit/integration/acceptance/etc. These views may require that a test be put in multiple locations in a tree, since a test may have one or more tags for instance.

This seems it would require either:

1. Allow a test to have multiple parents (suites could probably still be restricted to only one parent)
2. Have an additional member that defines "secondary parents" for a visibility perspective.

And also allow this dropdown to be hooked into to provide custom test trees (e.g. "Group by Tag", "Group by Function", "Group by Type". If there are multiple test providers where their root objects are shown, maybe put the "group by" as a dropdown at that level instead

Ideally the extension can register and supply whatever additional trees it wants, maybe via a new root object but would need to reference all the unique IDs already present in the main root, or otherwise some sort of "view" class

This goes into the "nice to have" and "vNext" category, it's not a must for initial release.

[connor4312]

I think @matepek also requested test tags. Is a generic tags: string[] on test items enough to do everything you need here? Or maybe Record<string, string> to group by a key? How would it work if tests are given from multiple providers that might have different sets of tags or organization?

[JustinGrote]

I don't consider myself smart enough to propose the ideal API implementation, but what I know needs to work:

1. When things are organized by a tag, if the tag is clicked to "run/debug", I would expect the runTests to be called with all the tests associated with that tag.
2. If a test shows up under two different tags (e.g. "Unit" and "MyApp"), when the test is run it should update in both places in the UI. As long as it's still referenced by its unique ID that shouldn't be a problem.
3. I would assume the root header would still preserve, so if you have "Pester" and "Markdown" test providers, they would each have their own root headers still and the tags would be organized under each respectively. If only one provider is loaded then the root header is omitted as is the behavior today.

tags: string[] would be simple to implement but I feel like the test provider should still provide the "view" so as not to pigeon-hole organizing tests into whatever vscode decides to support (in the same way the current parent/addChild method allows extensions to build the test tree to whatever makes sense to their provider). I'm not sure the best way to do this, maybe a view that allows for a parent/child hierarchy of Record<string,string> to be supplied, or a function that accepts a root object organized a different way without requiring a complete "rediscovery".

[connor4312]

For this I've experimented with converting the test filter for a more Marketplace-like box with @ tags and autocomplete. I also moved the current directives to "show tests in current document" for instance to be @ -directives since I didn't want to mix text and non-text search. Likewise I wanted tags to be text-based because the alternative experience would be click on filter -> click on tags -> quickpick for tags you want, which is pretty deep. History is still supported:

https://memes.peet.io/img/21-08-f9d0cc7b-42c5-4fcc-a4b4-e1eb201ed7d3.mp4

[connor4312]

@jdneo I saw that you have an intended use for these too -- does the UI I showed in the video work for you, any suggestions? 🙂

[jdneo]

Thank you @connor4312 ! The video looks promising.

There is another scenario came up in my mind: In a lazy-load test explorer with some children haven't been resolved yet. Will users want to search all the items related to a tag?

I'm not sure if this will be a request from users in the future though.

[connor4312]

Yea, when dealing with tags it only filters to tags and items that we know about. We could trigger a global expansion of every node but that could result in a performance cost. Thoughts?

[JustinGrote]

@connor4312 sounds like something that should be a configurable option :). "Search undiscovered Tests" or something like that, should default to false.

In addition, maybe when searching have an additional dummy entry or some sort of clickable link that says "more..." or "search all test files" or something so that you can have both.

[connor4312]

Yea we could have some additional text at the end of the tree saying something like "There may be more undiscovered tests matching this filter, would you like to [discover all tests]?"

[jdneo]

we could have some additional text at the end of the tree saying something like "There may be more undiscovered tests matching this filter, would you like to [discover all tests]?"

Sounds good to me!

[connor4312]

Verification:

1. Have or create a test extension that uses test tags. You can use the sample: https://github.com/microsoft/vscode-extension-samples/tree/main/test-provider-sample
2. Verify you can filter them in the Test Explorer view using the @controller:tag syntax, and get appropriate completion for them

[JustinGrote]

@connor4312 I don't see tags in the latest sample or in the latest proposed/stable API as far as I can tell. Is it present on testItem or am I missing something?

[connor4312]

It's in the latest proposed API right now. It should be stable at the end of this iteration.

[JustinGrote]

@connor4312 I had removed useproposedAPIs so I just noticed my vscode.proposed.d.ts wasn't actually updating, my bad!