Testing

Ethereum stateless protocol changes cut deep into the protocol since it must simultaneously change many angles, such as the state tree, EVM gas costs, and opcodes functioning.

Execution spec tests

One strategy for testing is using devnets, where multiple clients join and can check if they reach a consensus on executing automatically generated transactions or letting users experiment with the new features. While they’re very useful, it’s pretty costly to spin up a devnet, mainly in the early to mid stages of protocol development for clients — mainly on the coordination front.

A more efficient approach is creating a comprehensive test suite runnable by every client. Clients can efficiently run these test suites while they’re in development, both manually and in CIs. Since this doesn’t require a devnet, there’s no required coordination or overheads, allowing us to make progress faster. Moreover, they allow us to precisely describe corner case scenarios, which would be hard to replicate on every development spin-up.

There’s currently an active branch in execution-spec-tests, which contains tests for:

  • EIP-4762
  • EIP-6800
  • EIP-7709
  • EIP-7612

The current test suite contains ~200 test fixtures, which have uncovered more than 15 consensus bugs throughout many EL clients, usually in intricate corner cases. It is still far from perfect, and there’s always ongoing effort to improve it. For example, if any bug is detected in a devnet, adding it as a new test is important.

Note that filling tests for such deep protocol changes involved more than creating the tests; it also involved making big changes in the testing framework and Geth’s evm t8n filling tools. This effort was made collaboratively between the Stateless Consensus and the STEEL teams at the EF.

Filling & running tests

The execution spec test documentation is an excellent resource for understanding the overall process. Next, we’ll provide concrete steps referencing which are the right branches to be used to fill the test properly:

  1. Install the required prerequisites to use the testing framework.
  2. Pull the main branch from this repo.
    1. Run go build -o evm ./cmd/evm
    2. Save the generated binary in PATH_A
  3. Pull the verkle/main branch from execution-spec-tests repo
    1. Run uv run fill --fork Verkle -v -m blockchain_test -n auto --evm-bin=<PATH_A> to fill the tests. You can use whatever extra flags are described in the testing framework documentation to filter fillings.
  4. In the fixtures folder, you’ll find the generated fixtures.

The command for running the test depends on your EL client. For example, in Geth, you should go run ./cmd/blocktest <json fixture path>.

CI

In the geth branch used for stateless development, there are GitHub Actions workflows:

These are run on every new PR, so there’s quick feedback if a new feature or bug fix has a regression — they can also be helpful for other EL teams to include in their pipelines. You can also semi-derive the steps mentioned in the Filling tests section by reading the CI steps.