Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support “Pending” test case Tag #839 #840

Merged
merged 3 commits into from
Oct 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,12 +124,13 @@ MUnit test suites can be executed from VS Code like normal test suites.
Test results are formatted in a specific way to make it easy to search for them
in a large log file.

| Test | Prefix |
| ------- | ------- |
| Success | `+` |
| Failed | `==> X` |
| Ignored | `==> i` |
| Skipped | `==> s` |
| Test | Prefix | Comment | See Also |
| ------- | ------- | --------- | ------------------------------------- |
| Success | `+` | | |
| Failed | `==> X` | | [Writing assertions](assertions.html) |
| Ignored | `==> i` | `ignored` | [Filtering tests](filtering.html) |
| Pending | `==> i` | `PENDING` | [Declaring tests](tests.html) |
| Skipped | `==> s` | | [Filtering tests](filtering.html) |

Knowing these prefixes may come in handy for example when browsing test logs in
a browser. Search for `==> X` to quickly navigate to the failed tests.
Expand Down
5 changes: 5 additions & 0 deletions docs/scalatest.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ If you only use basic ScalaTest features, you should be able to replace usage of
+ test("ignored".ignore) {
// unchanged
}

- test("pending") (pending)
+ test("pending".pending) {
+ // zero or more assertions
+ }
```

If you are coming from `WordSpec` style tests, make sure to flatten them, or your tests
Expand Down
49 changes: 49 additions & 0 deletions docs/tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,55 @@ what Scala version caused the tests to fail.
+ munit.ScalaVersionFrameworkSuite.foo-2.13.1
```

## Tag pending tests

Use `.pending` to annotate a work-in-progress test case with known-incomplete coverage.
Any assertions in the pending case must pass, but will be reported as Ignored instead of Success.
Any failures will be reported as Failures (this differs from `.ignore` which skips the test case entirely).
This tag is useful for documenting:

- **Empty placeholders** that lack any assertions (unless tagged pending, these are reported as success).
- **Incomplete placeholders** with too few assertions (unless tagged pending, these are reported as success).
- **Accurate placeholders** whose stable assertions must pass (regressions are not ignored).
- **Searchability** of your codebase for known-incomplete test cases.
- **Cross-references** between your codebase and issue trackers.

You can (optionally) include a comment for your pending test case, such as a job ticket ID:

```scala
// Empty placeholder, without logged comments:
test("time travel".pending) {
// Test case to be written yesterday
}

// Empty placeholder, with logged comments:
test("time travel".pending("requirements from product owner")) {
// Is this funded yet??
}

// Empty placeholder, tracked for action:
test("time travel".pending("INTERN-101")) {
// Test case to be written by an intern
}

// Incomplete (WIP) placeholder, tracked for action:
test("time travel".pending("QA-404")) {
assert(LocalDate.now.isAfter(yesterday))
// QA team to provide specific examples for regression-test coverage
}
```

If you want to mark a failed regression test as pending-until-fixed,
you combine `.ignore` before or after `.pending`, for example:

```scala
test("this test worked yesterday".ignore.pending("platform investigation")) {
assert(LocalDate.now.equals(yesterday))
}
```

This allows pending comments, reasons, or cross-references to be logged for ignored tests.

## Tag flaky tests

Use `.flaky` to mark a test case that has a tendency to non-deterministically
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import static munit.internal.junitinterface.Ansi.*;

import java.lang.annotation.Annotation;
import java.util.Arrays;
import java.util.Collections;
import java.util.Set;
Expand Down Expand Up @@ -133,10 +134,28 @@ public void testIgnored(Description desc) {
desc,
new InfoEvent(desc, Status.Ignored) {
void logTo(RichLogger logger) {
StringBuilder builder = new StringBuilder(1024);
boolean isPending = false;
for (Annotation annotation : desc.getAnnotations()) {
if (annotation instanceof Tag) {
Tag tag = (Tag) annotation;
if (tag instanceof PendingTag) {
isPending = true;
}
else if (tag instanceof PendingCommentTag) {
builder.append(" ");
builder.append(tag.value());
}
}
}
if (isPending) {
builder.insert(0, " PENDING");
}
builder.append(" ignored");
logger.warn(
settings.buildTestResult(Status.Ignored)
+ ansiName
+ Ansi.c(" ignored", SKIPPED)
+ Ansi.c(builder.toString(), SKIPPED)
+ durationSuffix());
}
});
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package munit.internal.junitinterface;

public interface PendingCommentTag extends Tag {}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package munit.internal.junitinterface;

public interface PendingTag extends Tag {}
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,17 @@ final class JUnitReporter(
}
}

def reportTestIgnored(method: String): Unit = {
log(Info, AnsiColors.c(s"==> i $method ignored", AnsiColors.YELLOW))
def reportTestIgnored(
method: String,
elapsedMillis: Double,
suffix: String
): Unit = {
val suffixed = if (suffix.isEmpty) "" else s" ${suffix}"
log(
Info,
AnsiColors.c(s"==> i $method$suffixed ignored", AnsiColors.YELLOW) + " " +
formatTime(elapsedMillis)
)
emitEvent(method, Status.Ignored)
}
def reportAssumptionViolation(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,21 @@ class MUnitRunNotifier(reporter: JUnitReporter) extends RunNotifier {
override def fireTestIgnored(description: Description): Unit = {
ignored += 1
isReported += description
reporter.reportTestIgnored(description.getMethodName)
val pendingSuffixes = {
val annotations = description.getAnnotations
val isPending = annotations.collect { case munit.Pending =>
"PENDING"
}.distinct
val pendingComments = annotations.collect {
case tag: munit.PendingComment => tag.value
}
isPending ++ pendingComments
}
reporter.reportTestIgnored(
description.getMethodName,
elapsedMillis(),
pendingSuffixes.mkString(" ")
)
}
override def fireTestAssumptionFailed(
failure: notification.Failure
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package munit.internal.junitinterface

trait PendingCommentTag extends Tag
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package munit.internal.junitinterface

trait PendingTag extends Tag
4 changes: 3 additions & 1 deletion munit/shared/src/main/scala/munit/MUnitRunner.scala
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ class MUnitRunner(val cls: Class[_ <: Suite], newInstance: () => Suite)
catch onError
result.map { _ =>
notifier.fireTestFinished(description)
true
!test.tags(Pending)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't we ignoring PendingComment here and below?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, PendingComment uses Pending (because def pending(comment: String): TestOptions = pending.tag(PendingComment(comment))), so we only need to check for Pending.

}
}

Expand Down Expand Up @@ -361,6 +361,8 @@ class MUnitRunner(val cls: Class[_ <: Suite], newInstance: () => Suite)
notifier.fireTestAssumptionFailed(new Failure(description, f))
case TestValues.Ignore =>
notifier.fireTestIgnored(description)
case _ if test.tags(Pending) =>
notifier.fireTestIgnored(description)
case _ =>
()
}
Expand Down
3 changes: 3 additions & 0 deletions munit/shared/src/main/scala/munit/TestOptions.scala
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ final class TestOptions(
def fail: TestOptions = tag(Fail)
def flaky: TestOptions = tag(Flaky)
def ignore: TestOptions = tag(Ignore)
def pending: TestOptions = tag(Pending)
def pending(comment: String): TestOptions =
pending.tag(PendingComment(comment))
def only: TestOptions = tag(Only)
def tag(t: Tag): TestOptions = copy(tags = tags + t)
private[this] def copy(
Expand Down
2 changes: 1 addition & 1 deletion munit/shared/src/main/scala/munit/TestValues.scala
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,6 @@ object TestValues {
with NoStackTrace
with Serializable

/** The test case was ignored. */
/** The test case was ignored (skipped). */
val Ignore = munit.Ignore
}
7 changes: 7 additions & 0 deletions munit/shared/src/main/scala/munit/package.scala
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
import munit.internal.junitinterface.{PendingCommentTag, PendingTag}

package object munit {
case class PendingComment(override val value: String)
extends Tag(value)
with PendingCommentTag

val Ignore = new Tag("Ignore")
val Only = new Tag("Only")
val Flaky = new Tag("Flaky")
val Fail = new Tag("Fail")
val Pending: Tag with PendingTag = new Tag("Pending") with PendingTag
val Slow = new Tag("Slow")
}
Loading