Test name filtering does not allow disambiguating tests with overlapping names

[jonhoo]

Consider the following two tests:

#[test]
fn it_works() { /* ... */ }
#[test]
fn it_works_complexly() { /* ... */ }

There is currently no way to run only it_works, and not it_works_complexly:

$ cargo t it_works
running 2 tests
test it_works ... ok
test it_works_more ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

In Go, the filter is a regular expression, so one can use go test 'it_works$', but this not work in Rust (because the test harness presumably just uses simple substring matching):

$ cargo t 'it_works$'
running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out

[malbarbo]

Try cargo t it_works -- --exact.

[jonhoo]

Is that documented anywhere outside of cargo test -- --help?
I bet most users have never run that, and probably wouldn't even think to (like me :p).

Also, --exact is still a limited solution. It requires that I type out the entire test name, and means I can only run a single test this way. For example, I can't avoid running just the complexly test here:

#[test]
fn foo_works() { /* ... */ }
#[test]
fn foo_works_complexly() { /* ... */ }
#[test]
fn bar_works() { /* ... */ }

[cuviper]

You can also --skip complexly. But I agree it would be nice to have more controls -- if not full regex, maybe just globbing? Even just being able to specify multiple filters would help.

[jonhoo]

Both globbing and multiple filters seem like good solutions!

[jonhoo]

I might take a stab at implementing this. A couple of questions though:

• Is using glob inside of libtest fine?
• Making this the default is technically a backwards-incompatible change (arguments that didn't previously match any tests, like it*, may now match tests). However, given that it will only make patterns that previously didn't match match (?, *, [, and ] aren't allowed in test names) it is unlikely that anything relies on the current behaviour. How would we feel about making globbing the default?
  • EDIT: on second thought, since globs require matching the full string, whereas the current default is substring matches, this wouldn't be backwards compatible at all. Yet having to specify cargo test -- --glob pattern seems awfully inconvenient. How bad would making this non-backwards-compatible change be? Alternatively I suppose we could use glob either if --glob is passed or if a globbing character is included in the pattern (they can't appear in test names anyway)?
• Is globbing sufficient? It does address the original concerns outlined above as long as the glob is required to match the entire test name (which is how globs usually behave anyway), but is still more limited than full regexes.
• Should --exactly apply only to the next pattern, or to all patterns?
  • This is relevant only when/if we get support for multiple patterns.
• Which takes precedence if both --exact and --glob are given?
• If --glob is supplied, should --skip patterns also be considered glob patterns? What if a skip pattern contains a special character? Should it automatically be made a glob pattern?

Probably cc @rust-lang/tools and maybe @sfackler judging from #38181.

[jonhoo]

@cuviper see #46417 if you haven't already.

[steveklabnik]

Triage: no changes that I'm aware of.

[RalfJung]

Both globbing and multiple filters seem like good solutions!

Multiple filters already work. You can do cargo test -- filter1 filter2 --skip f3 --skip f4 to run all tests that contain (filter1 and neither f3 nor f4) or (filter2 and neither f3 nor f4).

For more precise filtering, there's a regex feature request at #53278.

[jonhoo]

In that case I suppose this issue can be closed, as the case from the original post can be achieved with

cargo test -- it_works --skip complexly