Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Option --list-tests should take into account --filter option #8643

Open
evil-shrike opened this issue Aug 22, 2017 · 16 comments
Open

Option --list-tests should take into account --filter option #8643

evil-shrike opened this issue Aug 22, 2017 · 16 comments
Labels
Area-DotNet Test enhancement
Milestone
Backlog

Comments

@evil-shrike
Copy link

I'm trying to execute one particular test so I specify a filter, but dotnet test executes several tests. So I need to understand what tests my filter captures. I would expect that --list-tests with --filter tells me. But it doesn't. --list-tests always shows all tests ignoring --filter option which is a pity.

Steps to reproduce

dotnet test Tests.csproj --list-tests --filter FullyQualifiedName~MyClass.MyMethod

lists only tests satisfying the filter

Expected behavior

lists only tests satisfying the filter

Actual behavior

lists all tests in assembly
@wli3
Copy link

wli3 commented Aug 23, 2017

Hi @evil-shrike thank you for the issue. You are correct, --list-tests only lists all the tests. We will discuss this enhancement.

Hi @Faizan2304 is there any existing vstest API can do that?

@Faizan2304
Copy link
Contributor

Hi @Faizan2304 is there any existing vstest API can do that?

Unfortunately --list-tests with --filter is not supported.

@msftgits msftgits transferred this issue from dotnet/cli Jan 31, 2020
@msftgits msftgits added this to the Backlog milestone Jan 31, 2020
@UyttenhoveSimon
Copy link

Hello,

Any update on this issue ?
It is quite useful to validate the filter before running the tests.

BR,
Simon

@kriwar
Copy link

kriwar commented Jun 17, 2021

Hello,

Any update on this issue ?
It is quite useful to validate the filter before running the tests.

BR,
Simon

Seconded

@jtorres
Copy link

jtorres commented Nov 10, 2021

Same here, would be helpfull this issue to be landed...

@btvanhooser
Copy link

Definitely would be very useful for our team

@jabracadabrah
Copy link

Useful for our team as well, any updates?

@steveharris57
Copy link

I just ran into the same issue. If this could be updated, it would be very much appreciated.

@Riverlution
Copy link

I actually want this for something other than verifying the correct tests have been selected by my filter, but this issue has forced me to use the NUnit console runner to get a list of test names for a given filter.

@Yuriy-Pelekh
Copy link

Need that filter for list of tests too.

@marek-ciz
Copy link

I'd also appreciate this feature

this isn't a complicated, unreal request - if you look and understand what all of us is trying to do, you'd know it's a very basic and practical use case

So I'm struggling to understand why it wasn't implemented in this way from the start, but what I really don't understand is why the only response from the owners/devs was this -> "is not supported"...

to me, that's sounds like the owners/devs/contributors are simply ignoring their community

@wli3 @Faizan2304

@dcdraper
Copy link

Yes, I really want this feature too, please! I assumed it was there but no :( it always lists ALL the tests. I have to actually run them to see which tests are under a give filter. @Riverlution thanks for the nunit idea!

@worldspawn
Copy link

When I use it it tells me no test matches the filter. If I drop --list-tests it starts finding and running tests.

@jpobst jpobst mentioned this issue Feb 22, 2023
Merged
jpobst added a commit to dotnet/android that referenced this issue Feb 27, 2023
@jpobst
[ci] Use AZDO built-in parallelization strategy. (#7804)
d447bff 
[Previously](#6360), we split our `MSBuildIntegration` unit tests to run across multiple CI test agents.  While a huge improvement, there are a few downsides to the approach we went with at the time:

- The number of agents is hardcoded by copy/pasting steps in the CI YAML.
- The tests must be manually load-balanced across agents with `[Category ("Node-X")]` in code which is cumbersome and hard to keep updated.

As we are at the point where our tests have expanded well past the 1 hour target we are faced with needing to increase our parallelization.

First, we can remove the YAML duplication by using AZDO's built-in `parallel` strategy, allowing us to control the number of agents to use with a single number:

```yaml
  - job: 
    strategy:
      parallel: 4
```

This leaves us the issue of automatically splitting our tests among an unknown number of test agents.  A clever solution is [provided in the AZDO docs](https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-any-test-runner?view=azure-devops): 
- Use `dotnet test --list-tests` to find all the tests
- Use a script to calculate which tests the agent should run, based on `$(System.JobPositionInPhase)` and `$(System.TotalJobsInPhase)`
- Pass those test names into `dotnet test` as a filter

Unfortunately there are issues with the provided approach:
- `dotnet test --list-tests` is not compatible with `--filter` so you have to run *all* tests in the test assembly which is not desirable for us. (dotnet/sdk#8643)
- Passing test names (including test parameters) on the command line hits limitations with escaping certain characters and argument limits.

So the approach is good, but we have to do all the work ourselves.  Enter [dotnet-test-slicer](https://github.com/jpobst/dotnet-test-slicer).

This dotnet global tool:
- Uses the `NUnit.Engine` NuGet package to find all tests in a test assembly using the desired filter query.
- Slices the test list for the current test agent.
- Outputs the desired tests into an NUnit-specific `.runsettings` file that can be passed to `dotnet test --settings foo.runsettings`, bypassing command line arguments.

Voila! Now we can automatically scale our test agents up or down as needed by changing a single variable in our YAML file.

Limitation: The tests are sliced in a round robin fashion.  This can produce uneven results if test durations are uneven.  A future optimization would be to store approximate test durations so tests can be load balanced more intelligently.
@shatulsky
Copy link

Workaround for this case using nunit3-console

choco install nunit-console-runner
nunit3-console.exe "C:\Projects\MyProject\Tests\bin\Debug\net7.0\Tests.dll" --where="cat==Smoke||cat==ProdHealthCheck" --explore

filters format for dotnet test and nunit3-console are different
In my case to convert my filer from TestCategory=Smoke|TestCategory=ProdHealthCheck to cat==Smoke||cat==ProdHealthCheck I had to make a few replaces

TestCategory -> cat
= -> ==
| -> ||

And as a result, the --where + --explore attributes output only the test list that matches the specified filter as expected.

@mikeblakeuk
Copy link

It should throw if filter and list-tests are passed.

@nohwnd
Copy link
Member

nohwnd commented Apr 5, 2023 •
edited
Loading

--filter and --list-tests is possible, and works to some extent. But unfortunately the implementation is dependent on the test adapter, so it differs for MSTest, NUnit and XUnit. As far as I can tell MSTest works fine. XUnit works only partially (when filtering based on traits xunit/visualstudio.xunit#205). And NUnit ignores the filter.

Fixing this needs to be done in the respective repos of those (and other) test frameworks/test adapters.

@andresrsanchez andresrsanchez mentioned this issue Jun 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area-DotNet Test enhancement
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests