Skip to content

refactor(fill,deps): forbid yul code in python tests; remove solc-select dependency #1779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 7, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ All commands use `uv run` prefix.

```bash
uv sync --all-extras
uv run solc-select use 0.8.24 --always-install
uvx pre-commit install
```

Expand Down
1 change: 0 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,6 @@ We welcome contributions via pull requests! This section will guide you through
uv python install 3.12
uv python pin 3.12
uv sync --all-extras
uv run solc-select use 0.8.24 --always-install
```

See [installation troubleshooting](https://eest.ethereum.org/main/getting_started/installation_troubleshooting) if you encounter issues.
Expand Down
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,6 @@ cd execution-spec-tests
uv python install 3.11
uv python pin 3.11
uv sync --all-extras
uv run solc-select use 0.8.24 --always-install
```

See [Installation Troubleshooting](https://eest.ethereum.org/main/getting_started/installation_troubleshooting/) in the online docs if you encounter issues.
Expand Down
11 changes: 11 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ Test fixtures for use by clients are available for each release on the [Github r

- Python 3.10 support was removed in this release ([#1808](https://github.com/ethereum/execution-spec-tests/pull/1808)).

#### πŸ’₯ Important Change for test contributors

- EEST no longer allows usage of Yul code in Python tests. From now on, please make use of our opcode wrapper. Yul code is now only allowed in the "static tests" located in `./tests/static/` (these are test cases defined by JSON and YAML files instead of Python test functions that were originally maintained in [ethereum/tests](https://github.com/ethereum/tests)).
- In order to fill the static tests (which is not the case by default), please ensure that `solc` is located in your `PATH`.

#### πŸ’₯ Important Change for `fill` Users

The output behavior of `fill` has changed ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)):
Expand All @@ -27,6 +32,12 @@ Users can select any of the artifacts depending on their testing needs for their

### πŸ› οΈ Framework

#### πŸ”€ Refactoring

- πŸ”€ Move `TransactionType` enum from test file to proper module location in `ethereum_test_types.transaction_types` for better code organization and reusability.
- ✨ Opcode classes now validate keyword arguments and raise `ValueError` with clear error messages.
- πŸ”€ This PR removes the `solc` requirement to fill Python test cases. Regular test contributors no longer need to concern themselves with `solc` and, as such, the `solc-select` dependency has been removed. The remaining tests that used Yul have been ported to the EEST opcode wrapper mini-lang and the use of Yul in Python tests is no longer supported. Maintainers only: To fill the "static" JSON and YAML tests (`./tests/static/`) locally, `solc` (ideally v0.8.24) must be available in your PATH.

#### `fill`

- ✨ Add the `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository ([#1590](https://github.com/ethereum/execution-spec-tests/pull/1590)).
Expand Down
6 changes: 2 additions & 4 deletions docs/dev/porting_legacy_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,16 +85,14 @@ Follow the hyperlinks for lost base coverage (`LBC`) to address coverage gaps. H

It's important to note that coverage helps identify missing test paths. If you believe the coverage loss is due to differences in "setup" code between frameworks and doesn't impact the feature you're testing, explain this in your PR. A team member can help with the review.

Also note that yul tests and possibly other tests used `CALLDATALOAD` that might no longer needed when designing a test with python. But we must always investigate if an opcode is not covered anymore to see if its okay.

For example, review the [discussion in this PR.](https://github.com/ethereum/execution-spec-tests/pull/975#issuecomment-2528792289)
For example, review the [discussion in this PR] to see an example of why and how coverage loss can occur.(https://github.com/ethereum/execution-spec-tests/pull/975#issuecomment-2528792289)

## Resolving Coverage Gaps from Yul Compilation

When porting tests from ethereum/tests, you may encounter coverage gaps that are false positives. This commonly occurs because:

- **Original tests** often used Yul to define smart contracts, and solc compilation introduces additional opcodes that aren't specifically under test
- **EEST ports** typically use the explicit EEST Opcode mini-language, which is more precise in opcode definition
- **EEST ports** use the explicit EEST Opcode mini-language, which is more precise in opcode definition

If coverage analysis shows missing opcodes that were only present due to Yul compilation artifacts (not the actual feature being tested), this can be resolved during PR review by adding the `coverage_missed_reason` parameter:

Expand Down
14 changes: 2 additions & 12 deletions docs/getting_started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,27 +26,17 @@ If installed via `curl`, `uv` will download Python for your target platform if o

Clone [execution-spec-tests](https://github.com/ethereum/execution-spec-tests) and install its dependencies. We recommend using Python 3.12, the following uses `uv` to download and configures 3.12 to be the Python version used in execution-spec-tests:

=== "All x86-64 platforms and ARM64 macOS"
=== "All platforms"

```console
git clone https://github.com/ethereum/execution-spec-tests
cd execution-spec-tests
uv python install 3.12
uv python pin 3.12
uv sync --all-extras
uv run solc-select use 0.8.24 --always-install
```

=== "ARM64 Linux"

```console
git clone https://github.com/ethereum/execution-spec-tests
uv python install 3.12
uv python pin 3.12
cd execution-spec-tests
uv sync --all-extras
```
Then follow [this guide](./installation_troubleshooting.md#problem-exception-failed-to-compile-yul-source) to build the `solc` binary from source and copy it to the expected location.
Static tests/maintainers only: To learn how to build the `solc` binary from source (optional) follow [this guide](./installation_troubleshooting.md#problem-exception-failed-to-compile-yul-source).

## Installation Troubleshooting

Expand Down
19 changes: 6 additions & 13 deletions docs/getting_started/installation_troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,11 @@ This page provides guidance on how to troubleshoot common issues that may arise

### Problem: `CERTIFICATE_VERIFY_FAILED`

!!! danger "Problem: `Failed to install solc ... CERTIFICATE_VERIFY_FAILED`"
When running either `uv run solc-select use 0.8.24 --always-install` or `fill` you encounter the following error:
!!! danger "Problem: `Failed to ... CERTIFICATE_VERIFY_FAILED`"
When running `fill` you might encounter the following error:

```bash
Exit: Failed to install solc version 0.8.24: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)>
Exit: Failed to ...: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)>
```

=== "Ubuntu"
Expand All @@ -66,15 +66,8 @@ This page provides guidance on how to troubleshoot common issues that may arise

### Problem: `Exception: failed to compile yul source`

!!! danger "Problem: `Running fill fails with tests that contain yul source code` on ARM platforms"
If you're using `x86_64`, try to manually run `solc-select` to install the required version of the `solc` binary:

```shell
uv run solc-select use 0.8.24 --always-install
```

Otherwise, this can happen when you're using an ARM64 OS but followed the x86-64 installation guide.
To resolve the issue you must build solidity from source (avoid 0.8.24 as it might require building z3 from source as well).
!!! danger "Problem: `Running fill on static_tests fails with tests that contain yul source code` on ARM platforms"
To resolve the issue you must acquire the `solc` binary e.g. by building solidity from source. The guide below installs v0.8.28 but you might prefer to choose a different version.

!!! success "Solution: Build solc from source"
The following steps have been tested on Ubuntu 24.04.2 LTS ARM64:
Expand All @@ -88,7 +81,7 @@ This page provides guidance on how to troubleshoot common issues that may arise
cp ./solc/solc $HOME/Documents/execution-spec-tests/.venv/bin/
chmod +x $HOME/Documents/execution-spec-tests/.venv/bin/solc
```
Running `uv run solc --version` should now return the expected version. Verify that `fill` works by running `uv run fill -m yul_test` in the execution-spec-tests root folder.
Running `uv run solc --version` should now return the expected version.

## Problem: `ValueError: unsupported hash type ripemd160`

Expand Down
4 changes: 2 additions & 2 deletions docs/running_tests/test_formats/blockchain_test_engine_x.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The key optimization is that **clients need only be started once per group** ins

Instead of including large pre-allocation state in each test fixture, this format references a pre-allocation groups folder (`pre_alloc`) which contains all different pre-allocation combinations organized by group.

A single JSON fixture file is composed of a JSON object where each key-value pair is a different [`BlockchainTestEngineXFixture`](#BlockchainTestEngineXFixture) test object, with the key string representing the test name.
A single JSON fixture file is composed of a JSON object where each key-value pair is a different [`BlockchainTestEngineXFixture`](#blockchaintestenginexfixture) test object, with the key string representing the test name.

The JSON file path plus the test name are used as the unique test identifier.

Expand Down Expand Up @@ -48,7 +48,7 @@ Each file in the `pre_alloc` folder corresponds to a pre-allocation group identi

## Consumption

For each [`BlockchainTestEngineXFixture`](#BlockchainTestEngineXFixture) test object in the JSON fixture file, perform the following steps:
For each [`BlockchainTestEngineXFixture`](#blockchaintestenginexfixture) test object in the JSON fixture file, perform the following steps:

1. **Load Pre-Allocation Group**:
- Read the appropriate file from the `pre_alloc` folder in the same directory
Expand Down
4 changes: 4 additions & 0 deletions docs/running_tests/test_formats/common_types.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ State allocation represented as a JSON object, where the keys are the addresses

Maps forks to blob schedule configurations as defined by [EIP-7840](https://eips.ethereum.org/EIPS/eip-7840).

### `Environment`

Contains blockchain-related configuration that provides the context in which a test is run.

### `ForkBlobSchedule`

A fork blob schedule as defined by [EIP-7840](https://eips.ethereum.org/EIPS/eip-7840) as a JSON dictionary with the following entries:
Expand Down
2 changes: 0 additions & 2 deletions docs/writing_tests/fork_methods.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,8 +151,6 @@ Methods for fork identification and comparison:
```python
fork.name() # Returns the name of the fork
fork.transition_tool_name(block_number=0, timestamp=0) # Returns name for transition tools
fork.solc_name() # Returns name for the solc compiler
fork.solc_min_version() # Returns minimum solc version supporting this fork
fork.is_deployed() # Returns whether the fork is deployed to mainnet
```

Expand Down
11 changes: 1 addition & 10 deletions docs/writing_tests/tutorials/blockchain.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ Before proceeding with this tutorial, it is assumed that you have prior knowledg
- Repository set-up, see [installation](../../getting_started/installation.md).and run an execution specification test as outlined in the .
- Able to run `fill`, see [Getting Started: Filling Tests](../../filling_tests/getting_started.md).
- Understand how to read a [blockchain test](https://ethereum-tests.readthedocs.io/en/latest/test_filler/blockchain_filler.html).
- Know the basics of [Yul](https://docs.soliditylang.org/en/latest/yul.html), which is an EVM assembly language.
- Familiarity with [Python](https://docs.python.org/3/tutorial/).
- Understand how to write an execution spec [state transition test](./state_transition.md).

Expand All @@ -26,15 +25,7 @@ A smart contract is defined that is called by each transaction in the test. It s
```python
contract_addr: Account(
balance=1000000000000000000000,
code=Yul(
"""
{
let next_slot := sload(0)
sstore(next_slot, number())
sstore(0, add(next_slot, 1))
}
"""
),
code=code,
storage={
0x00: 0x01,
},
Expand Down
Loading
Loading