Skipped tests reported as passing

[aherlihy]

Hello, I'm using assume to skip tests dynamically based on what's described in the documentation here: https://scalameta.org/munit/docs/filtering.html#ignore-single-test-case-based-on-a-dynamic-conditions, however the tests that fail the assertion are reported as passing (even though nothing is run). I'm not sure if I've misunderstood the documentation, from reading through some issues it looks like sometimes the final test report includes a "skipped" and sometimes not. I would like it to report as "n tests skipped", or at least ignore them completely but not report them as passing. I use the IntelliJ test API but it also happens when running with sbt.

The test itself is here but if I simply run assume(false) in either test(...) or in beforeEach, it doesn't run the test but reports it as passed.

If I manually use test("description".ignore) then it is reported as skipped, both on IntelliJ and SBT.

[tgodzik]

Thanks for reporting! I think we should show is as ignored, although I am not sure if that was the initial intention.

You can also ignore a test based on a TAG and that shows up correctly. We have some examples in scalameta/metals repository around here https://github.com/scalameta/metals/blob/main/tests/cross/src/main/scala/tests/BasePCSuite.scala#L159

[aherlihy]

Thanks for the help! I couldn't quite figure out how to set a tag dynamically, is that possible? If the test gets skipped is something that is determined after the fixture beforeEach is called. So I need to have instantiated the test already, but not called any asserts. It looks like:

class Test extends munit.FunSuite {
  val graph: Fixture[TestGraph] = new Fixture[TestGraph]("Graph") {
    override def beforeEach(context: BeforeEach): Unit = <... initialize graph ... >
  }
  
  test(testdir) {
     val g = graph()
     // if g.skip contains some property that matches the execution context, then I need to skip the test
  }

I'm not quite sure how to get the tags to be set after I've already entered test, which is the same problem I had with using <testname>.ignore. Is there a workaround possible?

Thank you!

[tgodzik]

Maybe, you could just use test transforms, but not sure if it will be possible to mark anything as ignored there.

The best option for you is probably what you tried before https://scalameta.org/munit/docs/filtering.html#ignore-entire-test-suite but you will lose the information about it being ignored.

[aherlihy]

Ok, I will do some refactoring, thanks for the help!