Slitherin Benchmark

The Slitherin Benchmark is designed for running detectors on Ethereum Virtual Machine (EVM) compatible Smart Contracts' verified sources database.

Setup

Configuration

1. Rename example.config.py to config.py and customize the parameters if needed.

Databases

The benchmark comes with default databases for Mainnet and Arbitrum contracts. The Mainnet database contains a random slice of verified contracts.

Database Directory Structure

Each database contains a contracts.json file with a JSON line for each contract. Each JSON must have the following fields: address (contract address), chain_id (blockchain ID in hex format, e.g., 0x1), and compiler (Solidity compiler version).

The source code of each contract is stored in xx/address/, where xx is the first two symbols in the hex representation of the address.

Use loader.py to load the source code of verified contracts from Etherscan. The input file should contain a JSON with the required field address.

python loader.py -o [output_directory] -i [input_json_file] -c [chain_id]

Contracts duplicate matcher

python contract_matcher.py -i [contracts_directory]

Check all contracts in input directory for copies and similar contracts. Uses oz_hashes.json and most_common_hashes.json. oz_hashes.json contains hashes of sources of all versions of openzeppelin contracts. Generated by oz_loader.py. most_common_hashes.json contains hashes of the most frequently occurring files.

Solc Compiler Loader

Load Solidity compiler binaries from ethereum/solc-bin.

python solc_loader.py

Usage

python runner.py -o [output_file] -eo [extra_output_file] -i [contracts_directory] [other_options]
Options:

-o, --output: File to save results.
-eo, --extra-output: File to save extra results (address, detector name, lines).
-i, --input: Directory with contracts.
-sd, --skip-duplicates: Skip duplicate contracts(marked by contract_matcher).
-sl, --skip-libs: Skip lib contracts(marked by contract_matcher).
-nc, --new-contracts: Check only unchecked contracts.
-nd, --new-detectors: Check contracts only with unchecked detectors.
-t, --timeout: Stops benchmark after seconds.
-l, --limit: Stops benchmark after seconds.
-d, --detect: Comma-separated list of detectors (defaults to Slitherin detectors).
-p, --pool: Number of process pools (defaults to CPU count).

The runner uses SQLite to save information about which contracts were checked with which detectors. Use the --new-contracts and --new-detectors flags to skip already checked contracts.

Analyze results

Results file

Benchmark runs with the --output option save results to a CSV file. Columns represent detector names, and rows indicate statistical counters. Two count methods are available: 'by_contract' counts the number of contracts with at least one finding for the detector, while 'by_finding' tallies the total number of findings detected during the benchmark run for each detector.

Google sheets

You can upload results to google sheets table using save_sheet.py script. There must be list with names for each count method(list_by_contract and list_by_finding).

python save_sheet.py --help
Usage: save_sheet.py [OPTIONS]

Options:
  -i, --input TEXT               file with benchmark results  [required]
  -sa, --service-account TEXT    google service account json file  [required]
  -si, --sheet-id TEXT           google sheet id  [required]
  -ln, --list-name TEXT          google list name  [required]
  -sv, --slitherin-version TEXT  slitherin version, default value taken from
                                 slitherin --version command

Slitherin project's CI pipeline uploads data to google sheets

Grafana

We visualize the results of slitherin benchmark through the Grafana.