Add json schema for test output

[mattsse]

Component

Forge

Describe the feature you would like

the test --json output can be useful for better integration.

foundry/crates/forge/src/result.rs

Lines 193 to 206 in f0d9eec

	pub struct SuiteResult {
	/// Wall clock time it took to execute all tests in this suite.
	#[serde(with = "humantime_serde")]
	pub duration: Duration,
	/// Individual test results: `test fn signature -> TestResult`.
	pub test_results: BTreeMap<String, TestResult>,
	/// Generated warnings.
	pub warnings: Vec<String>,
	/// Libraries used to link test contract.
	pub libraries: Libraries,
	/// Contracts linked with correct libraries.
	#[serde(skip)]
	pub known_contracts: Arc<ContractsByArtifact>,
	}

We currently don't have a proper spec for this making it hard to generate TS bindings for example.

This could be useful here: https://docs.rs/schemars/latest/schemars/

TODO

generate json schema for test --json ouput

Additional context

No response

[StErMi]

It would be nice to also have an explanation for each of the attributes to understand what they mean. I'm looking at some JSON logs and without having a full understanding of the whole source code (which is difficult for people not familiar with the source code and rust) is pretty difficult if not impossible.

As a quick example, the JSON structure inside 'kind' changes completely between the "standard" test and the "fuzz" test, which IMHO is not good

{
        "kind": {
          "Standard": 257
        },
}

{
        "kind": {
          "Fuzz": {
            "first_case": {
              "calldata": "...",
              "gas": 21607,
              "stipend": 21204
            },
            "runs": 257,
            "mean_gas": 403,
            "median_gas": 403
          }
        },
}

[sambacha]

I have this documented here already
https://github.com/sambacha/gas-reporting

Gas Snapshot Format

.gas-snapshot

FooTest:testSetFoo() (gas: 64071)
BarTest:testFuzzCurrentBar(uint256) (runs: 256, μ: 408, ~: 423)

JSON-Schema definition

Each test object represents a test function and has the following properties:

key	boolean	value
name	(required)	The name of the test function.
gas	(optional)	The gas used by the test function. This property is required if the test is not a fuzz test.
runs	(optional)	The number of runs for the test function. This property is required if the test is a fuzz test.
avg	(optional)	The average gas used across all runs. This property is required if the test is a fuzz test.
median	(optional)	The median gas used across all runs. This property is required if the test is a fuzz test.

The oneOf constraint ensures that a test object must have either the gas property (for non-fuzz tests) or the runs, avg, and median properties (for fuzz tests).

This JSON Schema describes the structure of the .gas-snapshot file format used by Forge. It allows for both regular tests with a single gas value and fuzz tests with multiple runs and statistical data (average and median gas used).

gas-snapshot

The schema defines an object with a single property tests, which is an array of test objects.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "tests": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "description": "The name of the test function"
          },
          "gas": {
            "type": "integer",
            "description": "The gas used by the test function"
          },
          "runs": {
            "type": "integer",
            "description": "The number of runs for the test function (only applicable for fuzz tests)"
          },
          "avg": {
            "type": "integer",
            "description": "The average gas used across all runs (only applicable for fuzz tests)"
          },
          "median": {
            "type": "integer",
            "description": "The median gas used across all runs (only applicable for fuzz tests)"
          }
        },
        "required": [
          "name"
        ],
        "oneOf": [
          {
            "required": [
              "gas"
            ]
          },
          {
            "required": [
              "runs",
              "avg",
              "median"
            ]
          }
        ]
      }
    }
  },
  "required": [
    "tests"
  ]
}

[Tushar4059x]

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

I am an open-source enthusiast with a strong proficiency in Rust, JavaScript, TypeScript, and Solidity. Given my experience and skills, I believe I am well-suited to work on the task of generating a JSON schema for the forge test --json output. My approach would involve thoroughly analyzing the current JSON structure, defining an accurate and comprehensive schema using tools like Schemars, and integrating this schema generation seamlessly into the forge codebase. I would also ensure that the schema remains in sync with future updates and provide thorough documentation and tests to support the implementation.

This work will enhance the tool's usability, especially for those looking to integrate it with other systems or generate TypeScript bindings. I'm confident that my background in these technologies will allow me to deliver a solution that meets the project's needs.

How I plan on tackling this issue

To tackle the task of generating a JSON schema for the forge test --json output, I would proceed as follows:

Understand the Current Output
First, I would examine the existing implementation of the test --json output by reviewing the code in result.rs. My goal would be to fully understand the structure and components of the JSON output, including the data types and any nested structures.

Define the JSON Schema
With a clear understanding of the output structure, I would then draft a JSON schema that accurately represents it. To make this process more efficient, I would use the Schemars library, which allows automatic derivation of JSON schemas from Rust structs. This schema would capture all the fields, types, and structures present in the output.

Integrate Schema Generation
Next, I would integrate the schema generation into the forge codebase. I might add a new command to output the schema separately or include it as part of the documentation. The key here is ensuring that the schema stays in sync with any future changes to the JSON output format.

Test the Schema
After generating the schema, I would validate it against the current test --json output to ensure it accurately represents the data. I’d also use the schema to generate TypeScript bindings or other integrations, verifying that it functions as intended for downstream users.

Document and Submit
Finally, I would update the project documentation to include details about the new JSON schema, how to generate it, and its potential uses. I would then submit my changes in a pull request, including tests to ensure the schema remains accurate in future updates.

[bshasikiran]

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

Engineering student just a web3kid

How I plan on tackling this issue

use schemars::JsonSchema;
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, JsonSchema)]
struct TestResult {
success: bool,
name: String,
// Add other relevant fields based on your output
}

[PedroRosalba]

Can I take this from here?

[Eminivince]

Let me handle this issue!

[hossamanany]

Can I take this from here?

[zerosnacks]

Sure, feel free to open a PR