Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for async fixtures #430

Merged
merged 10 commits into from
Oct 16, 2021
24 changes: 21 additions & 3 deletions build.sbt
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
import com.typesafe.tools.mima.core.DirectMissingMethodProblem
import com.typesafe.tools.mima.core.ProblemFilters
import com.typesafe.tools.mima.core.MissingTypesProblem
import com.typesafe.tools.mima.core._
import sbtcrossproject.CrossPlugin.autoImport.crossProject
import sbtcrossproject.CrossPlugin.autoImport.CrossType
import scala.collection.mutable
Expand Down Expand Up @@ -85,6 +83,26 @@ lazy val mimaEnable: List[Def.Setting[_]] = List(
),
ProblemFilters.exclude[DirectMissingMethodProblem](
"munit.internal.junitinterface.JUnitComputer.this"
),
// Known breaking changes for MUnit v1
ProblemFilters.exclude[IncompatibleMethTypeProblem](
"munit.FunSuite.munitTestTransform"
),
ProblemFilters.exclude[MissingClassProblem]("munit.GenericAfterEach"),
ProblemFilters.exclude[MissingClassProblem]("munit.GenericBeforeEach"),
ProblemFilters.exclude[MissingClassProblem]("munit.GenericTest"),
ProblemFilters.exclude[DirectMissingMethodProblem](
"munit.MUnitRunner.createTestDescription"
),
ProblemFilters.exclude[IncompatibleMethTypeProblem](
"munit.Suite.beforeEach"
),
ProblemFilters.exclude[IncompatibleMethTypeProblem](
"munit.Suite.afterEach"
),
ProblemFilters.exclude[MissingClassProblem]("munit.Suite$Fixture"),
ProblemFilters.exclude[IncompatibleMethTypeProblem](
"munit.TestTransforms#TestTransform.apply"
)
),
mimaPreviousArtifacts := {
Expand Down
73 changes: 70 additions & 3 deletions docs/fixtures.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,12 @@ to reusable or ad-hoc fixtures when necessary.
## Reusable test-local fixtures

Reusable test-local fixtures are more powerful than functional test-local
fixtures because they can declare custom logic that gets evaluted before each
fixtures because they can declare custom logic that gets evaluated before each
local test case and get torn down after each test case. These increased
capabilities come at the price of ergonomics of the API.

Override the `beforeEach()` and `afterEach()` methods in the `Fixture[T]` trait
to configure a reusable test-local fixture.
Override the `beforeEach()`, `afterEach()` and `munitFixtures` methods in the
`Fixture[T]` trait to configure a reusable test-local fixture.

```scala mdoc:reset
import java.nio.file._
Expand Down Expand Up @@ -179,6 +179,73 @@ class MySuite extends munit.FunSuite {
}
```

## Asynchronous fixtures with `FutureFixture`

Extend `FutureFixture[T]` to return `Future[T]` values from the lifecycle
methods `beforeAll`, `beforeEach`, `afterEach` and `afterAll`.

```scala mdoc:reset
import java.nio.file._
import java.sql.Connection
import java.sql.DriverManager
import munit.FutureFixture
import munit.FunSuite
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

class AsyncFilesSuite extends FunSuite {

val file = new FutureFixture[Path]("files") {
var file: Path = null
def apply() = file
override def beforeEach(context: BeforeEach): Future[Unit] = Future {
file = Files.createTempFile("files", context.test.name)
}
override def afterEach(context: AfterEach): Future[Unit] = Future {
// Always gets called, even if test failed.
Files.deleteIfExists(file)
}
}

override def munitFixtures = List(file)

test("exists") {
// `file` is the temporary file that was created for this test case.
assert(Files.exists(file()))
}
}
```

## Asynchronous fixtures with custom effect type

First, create a new `EffectFixture[T]` class that extends `munit.AnyFixture[T]`
and overrides all lifecycle methods to return values of type `Effect[Unit]`. For
example:

```scala mdoc:reset
import munit.AfterEach
import munit.BeforeEach

// Hypothetical effect type called "Resource"
sealed abstract class Resource[+T]
object Resource {
def unit: Resource[Unit] = ???
}

abstract class ResourceFixture[T](name: String) extends munit.AnyFixture[T](name) {
// The main purpose of "ResourceFixture" is to help IDEs auto-complete
// the result type "Resource[Unit]" instead of "Any" when implementing the
// "ResourceFixture" class.
override def beforeAll(): Resource[Unit] = Resource.unit
override def beforeEach(context: BeforeEach): Resource[Unit] = Resource.unit
override def afterEach(context: AfterEach): Resource[Unit] = Resource.unit
override def afterAll(): Resource[Unit] = Resource.unit
}
```

Next, extend `munitValueTransforms` to convert `Resource[T]` into `Future[T]`,
see [declare async tests](tests.md#declare-async-test) for more details.

## Avoid stateful operations in the class constructor

Test classes may sometimes get initialized even if no tests run so it's best to
Expand Down
66 changes: 12 additions & 54 deletions docs/tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@ test("buggy-task") {
```

Since tasks are lazy, a test that returns `LazyFuture[T]` will always pass since
you need to call `run()` to start the task execution. Override `munitValueTransforms`
to make sure that `LazyFuture.run()` gets called.
you need to call `run()` to start the task execution. Override
`munitValueTransforms` to make sure that `LazyFuture.run()` gets called.

```scala mdoc
import scala.concurrent.ExecutionContext.Implicits.global
Expand Down Expand Up @@ -118,9 +118,10 @@ parallel test suite execution in sbt, add the following setting to `build.sbt`.
Test / parallelExecution := false
```

In case you do not run your tests in parallel, you can also disable buffered logging,
which is on by default to prevent test results of multiple suites from appearing interleaved.
Switching buffering off would give you immediate feedback on the console while a suite is running.
In case you do not run your tests in parallel, you can also disable buffered
logging, which is on by default to prevent test results of multiple suites from
appearing interleaved. Switching buffering off would give you immediate feedback
on the console while a suite is running.

```sh
Test / testOptions += Tests.Argument(TestFrameworks.MUnit, "-b")
Expand Down Expand Up @@ -199,9 +200,9 @@ bug report but don't have a solution to fix the issue yet.

## Customize evaluation of tests with tags

Override `munitTestTransforms()` to extend the default behavior for how test bodies are
evaluated. For example, use this feature to implement a `Rerun(N)` modifier to
evaluate the body multiple times.
Override `munitTestTransforms()` to extend the default behavior for how test
bodies are evaluated. For example, use this feature to implement a `Rerun(N)`
modifier to evaluate the body multiple times.

```scala mdoc
case class Rerun(count: Int) extends munit.Tag("Rerun")
Expand All @@ -228,9 +229,9 @@ class MyRerunSuite extends munit.FunSuite {
}
```

The `munitTestTransforms()` method is similar to `munitValueTransforms()` but is different in
that you also have access information about the test in `TestOptions` such as
tags.
The `munitTestTransforms()` method is similar to `munitValueTransforms()` but is
different in that you also have access information about the test in
`TestOptions` such as tags.

## Customize test name based on a dynamic condition

Expand Down Expand Up @@ -316,46 +317,3 @@ abstract class BaseSuite extends munit.FunSuite {
class MyFirstSuite extends BaseSuite { /* ... */ }
class MySecondSuite extends BaseSuite { /* ... */ }
```

## Roll our own testing library with `munit.Suite`

The `munit.FunSuite` class comes with a lot of built-in functionality such as
assertions, fixtures, `munitTimeout()` helpers and more. These features may not
be necessary or even desirable when writing tests. You may sometimes prefer a
smaller API.

Extend the base class `munit.Suite` to implement a minimal test suite that
includes no optional MUnit features. At its core, MUnit operates on a data
structure `GenericTest[TestValue]` where the type parameter `TestValue`
represents the return value of test bodies. This type parameter can be
customized per-suite. In `munit.FunSuite`, the type parameter `TestValue` is
defined as `Any` and `type Test = GenericTest[Any]`.

Below is an example custom test suite with `type TestValue = Future[String]`.

```scala
class MyCustomSuite extends munit.Suite {
override type TestValue = Future[String]
override def munitTests() = List(
new Test(
"name",
// compile error if it's not a Future[String]
body = () => Future.successful("Hello world!"),
tags = Set.empty[Tag],
location = implicitly[Location]
)
)
}
```

Some use-cases where you may want to define a custom `munit.Suite`:

- implement APIs that mimic testing libraries to simplify the migration to MUnit
- design stricter APIs that don't use `Any`
- design purely functional APIs with no publicly facing side-effects

In application code, it's desirable to use strong types avoid mutable state.
However, it's not clear that those best practices yield the same cost/benefit
ratio when writing test code. MUnit intentionally exposes types such `Any` and
side-effecting methods like `test("name") { ... }` because they subjectively
make the testing API nice-to-use.
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ trait ScalaCheckSuite extends FunSuite {
new TestTransform(
"ScalaCheck Prop",
t => {
t.withBodyMap[TestValue](
t.withBodyMap(
_.transformCompat {
case Success(prop: Prop) => propToTry(prop, t)
case r => r
Expand Down
8 changes: 7 additions & 1 deletion munit/js/src/main/scala/munit/internal/PlatformCompat.scala
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import sbt.testing.EventHandler
import sbt.testing.Logger
import scala.concurrent.Promise
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext

object PlatformCompat {
def executeAsync(
Expand All @@ -20,7 +21,12 @@ object PlatformCompat {
task.execute(eventHandler, loggers, _ => p.success(()))
p.future
}
def waitAtMost[T](future: Future[T], duration: Duration): Future[T] = {

def waitAtMost[T](
future: Future[T],
duration: Duration,
ec: ExecutionContext
): Future[T] = {
future
}

Expand Down
42 changes: 38 additions & 4 deletions munit/jvm/src/main/scala/munit/internal/PlatformCompat.scala
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,15 @@ import scala.concurrent.Future
import sbt.testing.Task
import sbt.testing.EventHandler
import sbt.testing.Logger
import scala.concurrent.Await
import scala.concurrent.duration.Duration
import scala.util.Try
import java.util.concurrent.Executors
import scala.concurrent.Promise
import scala.concurrent.ExecutionContext
import java.util.concurrent.TimeUnit
import java.util.concurrent.TimeoutException

object PlatformCompat {
private val sh = Executors.newSingleThreadScheduledExecutor()
def executeAsync(
task: Task,
eventHandler: EventHandler,
Expand All @@ -17,8 +21,38 @@ object PlatformCompat {
task.execute(eventHandler, loggers)
Future.successful(())
}
def waitAtMost[T](future: Future[T], duration: Duration): Future[T] = {
Future.fromTry(Try(Await.result(future, duration)))
@deprecated("use the overload with an explicit ExecutionContext", "1.0.0")
def waitAtMost[T](
future: Future[T],
duration: Duration
): Future[T] = {
waitAtMost(future, duration, ExecutionContext.global)
}
def waitAtMost[T](
future: Future[T],
duration: Duration,
ec: ExecutionContext
): Future[T] = {
if (future.value.isDefined) {
// Avoid heavy timeout overhead for non-async tests.
future
} else {
val onComplete = Promise[T]()
var onCancel: () => Unit = () => ()
future.onComplete { result =>
onComplete.tryComplete(result)
}(ec)
val timeout = sh.schedule[Unit](
() =>
onComplete.tryFailure(
new TimeoutException(s"test timed out after $duration")
),
duration.toMillis,
TimeUnit.MILLISECONDS
)
onCancel = () => timeout.cancel(false)
onComplete.future
}
}

def isIgnoreSuite(cls: Class[_]): Boolean =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import sbt.testing.Task
import sbt.testing.EventHandler
import sbt.testing.Logger
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext

object PlatformCompat {
def executeAsync(
Expand All @@ -18,7 +19,11 @@ object PlatformCompat {
task.execute(eventHandler, loggers)
Future.successful(())
}
def waitAtMost[T](future: Future[T], duration: Duration): Future[T] = {
def waitAtMost[T](
future: Future[T],
duration: Duration,
ec: ExecutionContext
): Future[T] = {
future
}

Expand Down
5 changes: 5 additions & 0 deletions munit/shared/src/main/scala/munit/AfterEach.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
package munit

class AfterEach(
val test: Test
) extends Serializable
37 changes: 37 additions & 0 deletions munit/shared/src/main/scala/munit/AnyFixture.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package munit

/**
* AnyFixture allows you to acquire resources during setup and clean up resources after the tests finish running.
*
* Fixtures can be local to a single test case by overriding `beforeEach` and
* `afterEach`, or they can be re-used for an entire test suite by extending
* `beforeAll` and `afterAll`.
*
* It's preferable to use a sub-class like `munit.Fixture` or
* `munit.FutureFixture` instead of this class. Extend this class if you're
* writing an integration a third-party type like Cats `Resource`.
*
* @see https://scalameta.org/munit/docs/fixtures.html
* @param fixtureName The name of this fixture, used for displaying an error message if
* `beforeAll()` or `afterAll()` fail.
*/
abstract class AnyFixture[T](val fixtureName: String) {

/** The value produced by this suite-local fixture that can be reused for all test cases. */
def apply(): T

/** Runs once before the test suite starts */
def beforeAll(): Any = ()

/**
* Runs before each individual test case. An error in this method aborts the test case.
*/
def beforeEach(context: BeforeEach): Any = ()

/** Runs after each individual test case. */
def afterEach(context: AfterEach): Any = ()

/** Runs once after the test suite has finished, regardless if the tests failed or not. */
def afterAll(): Any = ()

}
5 changes: 5 additions & 0 deletions munit/shared/src/main/scala/munit/BeforeEach.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
package munit

class BeforeEach(
val test: Test
) extends Serializable
Loading