ethereum/execution-spec-tests is both a collection of test cases and a framework implemented in Python to generate tests for Ethereum execution clients.
The framework collects and executes the test cases in order to generate test fixtures (JSON) which can be consumed by any execution client to verify their implementation of ethereum/execution-specs. The fixtures, which define state transition and block tests, are generated by the framework using one of the t8n
command-line tools that are provided by most execution clients, see below for an overview of the supported t8n
tools.
---
title: Test Fixture Generation with execution-spec-tests
---
flowchart LR
style C stroke:#333,stroke-width:2px
style D stroke:#333,stroke-width:2px
style G stroke:#F9A825,stroke-width:2px
style H stroke:#F9A825,stroke-width:2px
subgraph "ethereum/go-ethereum, ..."
C[<code>evm t8n</code>\nexternal executable]
end
subgraph ethereum/solidity
D[<code>solc</code>\nexternal executable]
end
subgraph ethereum/EIPs
E(<code>EIPS/EIP-*.md</code>\nSHA digest via Github API)
end
subgraph "ethereum/execution-spec-tests"
A(<code>./tests/**/*.py</code>\nPython Test Cases)
B([<code>$ fill ./tests/</code>\nPython Framework])
end
subgraph Test Fixture Consumers
subgraph ethereum/hive
G([<code>$ hive ...</code>\nGo Test Framework])
end
H([Client executables])
end
C <-.-> B
D <-.-> B
A --> B
E <-.-> |retrieve latest spec version\ncheck tested spec version| B
B -->|output| F(<code>./fixtures/**/*.json</code>\nJSON Test Fixtures)
F -->|input| G
F -->|input| H
The generated test fixtures can be used:
- Directly by client teams' test frameworks, and,
- In the integration tests executed in the ethereum/hive framework.
The following transition tools are supported by the framework:
Client | "t8n" Tool | Tracing Support |
---|---|---|
ethereum/evmone | evmone-t8n |
Yes |
ethereum/execution-specs | ethereum-spec-evm |
Yes |
ethereum/go-ethereum | evm t8n |
Yes |
ethereumjs | ethereumjs-t8ntool.sh |
No |
hyperledger/besu | evm t8n-server |
Yes |
status-im/nimbus-eth1 | t8n |
Yes |
Generally, specific t8n
implementations and branches must be used when developing tests for upcoming EIPs.
We use named reference tags to point to the specific version of the t8n
implementation that needs to be used fill the tests.
All current tags, their t8n implementation and branch they point to, are listed in .github/configs/evm.yaml.
The tools provided by ethereum/execution-spec-tests use uv
(docs.astral.sh/uv) to manage their dependencies and virtual environment. uv
downloads Python for your target platform if one of the required versions (Python 3.10, 3.11 or 3.12) is not available natively.
uv
can be installed via curl (recommended; can self-update):
curl -LsSf https://astral.sh/uv/install.sh | sh
or pip (requires Python, can't self-update):
pip install uv
Clone execution-spec-tests and install its dependencies:
git clone https://github.com/ethereum/execution-spec-tests
cd execution-spec-tests
uv sync --all-extras
uv run solc-select use 0.8.24 --always-install
See Installation Troubleshooting in the online docs if you encounter issues.
By default, JSON test fixtures are generated from this repository's Python test cases using the Ethereum Execution Layer Specification (EELS) reference implementation. The resulting JSON fixtures can be executed against execution clients to verify consensus. The process of generating fixtures is often referred to as "filling".
-
Explore test cases via
--collect-only
and search for test cases that combinePUSH0
andDELEGATECALL
in the EVM functionality introduced in the Shanghai hard fork:uv run fill --collect-only -k "push0 and delegatecall" tests/shanghai/
The
fill
command is based onpytest
. The above command uses the optional pytest arguments:--collect-only
only collect test cases; don't execute them.-k <expression>
filter test cases by their test case ID based on the given expression.tests/shanghai
the directory containing the test cases (tellsfill
to only discover test cases in this directory; default:tests/
).
-
Fill
state_test
fixtures for these test cases:uv run fill -k "push0 and delegatecall" tests/shanghai/ -m state_test -v
where:
-m state_test
only fills test cases marked as astate_test
(see all available markers viauv run fill --markers
).-v
enables verbose output.
-
Verify the generated fixtures:
a. Check the corresponding fixture file has been generated:
head fixtures/state_tests/shanghai/eip3855_push0/push0/push0_contract_during_call_contexts.json
b. Open the generated HTML test using the link provided at the bottom of the console output. This is written to the output directory at:
./fixtures/.meta/report_fill.html
More information on how to obtain and consume the released test fixtures can be found in the documentation.
For further help with working with this codebase, see the online documentation:
- Learn useful command-line flags.
- Execute tests for features under development via the
--from=FORK1
and--until=FORK2
flags. - Optional: Configure VS Code to auto-format Python code and execute tests within VS Code.
- Implement a new test case, see Writing Tests.
The available test cases can be browsed in the Test Case Reference doc.
If you encounter issues during the installation process, please refer to the Installation Troubleshooting page.
Contributions and feedback are welcome. Please see the online documentation for this repository's coding standards and help on implementing new tests.
Care is required when adding PRs or issues for functionality that is live on Ethereum mainnet, please refer to the Security Policy for more information about reporting vulnerabilities and eligibility for the bug bounty program.
This project is licensed under the MIT License - see the LICENSE file for details.