Skip to content

Commit

Permalink
docs: ux fuzz invariant (#880)
Browse files Browse the repository at this point in the history
* Docs for in-line test configuration.

* Added description about base config inheritance

* Rewording

* Pages that describe "tests" now reference pages that describe how to config tests.

* Config in block comments explained
  • Loading branch information
0xMelkor authored May 11, 2023
1 parent f41642d commit e037147
Show file tree
Hide file tree
Showing 4 changed files with 92 additions and 1 deletion.
7 changes: 6 additions & 1 deletion src/forge/fuzz-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ There are different ways to run property-based tests, notably parametric testing

You might have noticed that fuzz tests are summarized a bit differently compared to unit tests:

- "runs" refers to the amount of scenarios the fuzzer tested. By default, the fuzzer will generate 256 scenarios, however, this can be configured using the [`FOUNDRY_FUZZ_RUNS`](../reference/config/testing.md#runs) environment variable.
- "runs" refers to the amount of scenarios the fuzzer tested. By default, the fuzzer will generate 256 scenarios, but this and other test execution parameters can be setup by the user. Fuzzer configuration details are provided [`here`](#configuring-fuzz-test-execution).
- "μ" (Greek letter mu) is the mean gas used across all fuzz runs
- "~" (tilde) is the median gas used across all fuzz runs

### Configuring fuzz test execution

Fuzz tests execution is governed by parameters that can be controlled by users via Forge configuration primitives. Configs can be applied globally or on a per-test basis. For details on this topic please refer to
📚 [Global config](../reference/config/testing.md) and 📚 [In-line config](../reference/config/inline-test-config.md).
7 changes: 7 additions & 0 deletions src/forge/invariant-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,15 @@ Invariant testing campaigns have two dimensions, `runs` and `depth`:
- `runs`: Number of times that a sequence of function calls is generated and run.
- `depth`: Number of function calls made in a given `run`. All defined invariants are asserted after each function call is made. If a function call reverts, the `depth` counter still increments.

These and other invariant configuration aspects are explained [`here`](#configuring-invariant-test-execution).

Similar to how standard tests are run in Foundry by prefixing a function name with `test`, invariant tests are denoted by prefixing the function name with `invariant` (e.g., `function invariant_A()`).

### Configuring invariant test execution

Invariant tests execution is governed by parameters that can be controlled by users via Forge configuration primitives. Configs can be applied globally or on a per-test basis. For details on this topic please refer to
📚 [Global config](../reference/config/testing.md) and 📚 [In-line config](../reference/config/inline-test-config.md).

## Defining Invariants

Invariants are conditions expressions that should always hold true over the course of a fuzzing campaign. A good invariant testing suite should have as many invariants as possible, and can have different testing suites for different protocol states.
Expand Down
1 change: 1 addition & 0 deletions src/reference/config/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
- [Project](./project.md)
- [Solidity Compiler](./solidity-compiler.md)
- [Testing](./testing.md)
- [In-line test configuration](./inline-test-config.md)
- [Formatter](./formatter.md)
- [Documentation Generator](./doc-generator.md)
- [Etherscan](./etherscan.md)
78 changes: 78 additions & 0 deletions src/reference/config/inline-test-config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
## In-line test configuration
Foundry users are enabled to specify overall test configurations, using a combination of ENV variables and config statements in the `foundry.toml`. Checkout the [📚 Testing reference](./testing) for a detailed description.

Despite this may work in the general case, some tests may need finer control over their configuration. For such reason Forge provides a way to specify per-test configs for invariant and fuzz testing scenarios.

Users can in-line test config statements directly in Solidity comments. This would affect the behavior of the `forge test` command for a specific test instance, as illustrated in the example below.

```solidity
contract MyTest is Test {
/// forge-config: default.fuzz.runs = 100
/// forge-config: ci.fuzz.runs = 500
function test_SimpleFuzzTest(uint256 x) public {
// --- snip ---
}
}
```

What we are asking here is to run our fuzzer `100` and `500` times for the `default` and `ci` profiles respectively. The interesting fact is that this would override any fuzz `runs` setup existing at a global level. All other configs would be inherited from the global context, making this acting as a fallback for all possible configurations.

### Block comments
In-line test configurations can also be expressed in block comments, as illustrated in the example.

```solidity
contract MyTest is Test {
/**
* forge-config: default.fuzz.runs = 1024
* forge-config: default.fuzz.max-test-rejects = 500
*/
function test_SimpleFuzzTest(uint256 x) public {
// --- snip ---
}
}
```

### In-line fuzz configs
Users can specify the configs described in the table. Each statement must have a prefix of the form `forge-config: ${PROFILE}.fuzz.`

| Parameter | Type | Description |
|-|-|-|
|`runs`|integer|The amount of fuzz runs to perform for this specific test case [📚 ref](./testing#runs).|
|`max-test-rejects`|integer|The maximum number of combined inputs that may be rejected before the test as a whole aborts [📚 ref](./testing#max_test_rejects).|

Fuzz config example
```solidity
contract MyFuzzTest is Test {
/// forge-config: default.fuzz.runs = 100
/// forge-config: default.fuzz.max-test-rejects = 2
function test_InlineConfig(uint256 x) public {
// --- snip ---
}
}
```

### In-line invariant configs
Users can specify the configs described in the table. Each statement must have a prefix of the form `forge-config: ${PROFILE}.invariant.`

| Parameter | Type | Description |
|-|-|-|
|`runs`|integer|The amount of invariant runs to perform for this specific test case [📚 ref](./testing#runs-1).
|`depth`|integer|The number of calls executed to attempt to break invariant in one run [📚 ref](./testing#depth).
|`fail-on-revert`|boolean|Fails the invariant fuzzing if a revert occurs [📚 ref](./testing#fail_on_revert).
|`call-override`|boolean|Overrides unsafe external calls when running invariant test [📚 ref](./testing#call_override).

Invariant config example
```solidity
contract MyInvariantTest is Test {
/// forge-config: default.invariant.runs = 100
/// forge-config: default.invariant.depth = 2
/// forge-config: default.invariant.fail-on-revert = false
/// forge-config: default.invariant.call-override = true
function invariant_InlineConfig() public {
// --- snip ---
}
}
```



0 comments on commit e037147

Please sign in to comment.