Loading test data from files

[akirillo]

To what component is your bug related to?

Forge

Feature Request

Overview

Being able to read in test data from an external source, e.g. even just a flat file, would enable a massive unlock in testing flexibility while allowing developers to avoid the overhead of running a devnet node / blockifier / starknet_in_rust.

Use cases

Testing against reference implementations

This is my current use case. I have some rather complex Cairo code (we're doing Bulletproofs verification) that I would like to test against a reference implementation in another language.

This is generally a good practice for cryptographic code, other examples can include testing equivalency of the implementation of a hashing function, or of cryptographic data structures such as Merkle trees.

Integration testing

Something else I'd like to do - integration testing across a product stack that spans across more than just a system of smart contracts. In my case, I want to test the end-to-end flow of generating a ZKP -> verifying it in my smart contract.

Quick iteration on unimplemented features

This can be used to work around testing features that are not yet implemented in Starknet-Foundry. Like fuzzing, for example - I could generate fuzzed test data using e.g. Rust, write it to a flat file, and read it into my tests.

Potential implementation paths

I think the best initial approach is the simplest one - read in test data that is already calldata-serialized from a flat file. Testing functions can accept parameters - honestly, even just a single parameter (it can be an array/tuple if multiple values are necessary) - into which the calldata is deserialized. Of course, params must implement Serde.

Obviously, I defer to the maintainers on the desired implementation paths should this issue be taken on, I have little context on how Starknet-Foundry was built.

I'm not familiar with Foundry so I don't know if it implements something like this, but if there are any similar patterns for test data generation there, seems like a good place to start :)

[akirillo]

Ah, looks like Foundry has a precedent for this with its file cheatcodes: https://book.getfoundry.sh/cheatcodes/fs

[ftupas]

Would be great to be able to use JSON based input for test data, as long as it's respecting Cairo context (range checks, string length, etc.).

[Arcticae]

Also, see:
foundry-rs/foundry#858

[cptartur]

Must be resolved before we start this #397