Skip to content

refactor(fill,fixtures): rename "engine reorg" to "engine x"; improve pre-allocation group terminology #1760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 18, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ Users can select any of the artifacts depending on their testing needs for their

#### `fill`

- ✨ Add the `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository [#1590](https://github.com/ethereum/execution-spec-tests/pull/1590).
- ✨ Add a new pytest plugin, `ported_tests`, that lists the static fillers and PRs from `ported_from` markers for use in the coverage Github Workflow [#1634](https://github.com/ethereum/execution-spec-tests/pull/1634).
- ✨ Enable two-phase filling of fixtures with shared pre-allocation groups and add a `BlockchainEngineReorgFixture` format [#1606](https://github.com/ethereum/execution-spec-tests/pull/1706).
- ✨ Add the `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository ([#1590](https://github.com/ethereum/execution-spec-tests/pull/1590)).
- ✨ Add a new pytest plugin, `ported_tests`, that lists the static fillers and PRs from `ported_from` markers for use in the coverage Github Workflow ([#1634](https://github.com/ethereum/execution-spec-tests/pull/1634)).
- ✨ Enable two-phase filling of fixtures with pre-allocation groups and add a `BlockchainEngineXFixture` format ([#1706](https://github.com/ethereum/execution-spec-tests/pull/1706), [#1760](https://github.com/ethereum/execution-spec-tests/pull/1760)).
- πŸ”€ Refactor: Encapsulate `fill`'s fixture output options (`--output`, `--flat-output`, `--single-fixture-per-file`) into a `FixtureOutput` class ([#1471](https://github.com/ethereum/execution-spec-tests/pull/1471),[#1612](https://github.com/ethereum/execution-spec-tests/pull/1612)).
- ✨ Don't warn about a "high Transaction gas_limit" for `zkevm` tests ([#1598](https://github.com/ethereum/execution-spec-tests/pull/1598)).
- 🐞 `fill` no longer writes generated fixtures into an existing, non-empty output directory; it must now be empty or `--clean` must be used to delete it first ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)).
Expand Down
2 changes: 1 addition & 1 deletion docs/library/cli/extract_config.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ Since Hive doesn't directly expose container IDs, the tool uses a detection mech
The tool supports:

- Individual fixture JSON files (BlockchainFixture format)
- SharedPreStateGroup JSON files
- PreAllocGroup JSON files
- Directories containing multiple fixture files

## Troubleshooting
Expand Down
2 changes: 1 addition & 1 deletion docs/navigation.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
* [State Tests](running_tests/test_formats/state_test.md)
* [Blockchain Tests](running_tests/test_formats/blockchain_test.md)
* [Blockchain Engine Tests](running_tests/test_formats/blockchain_test_engine.md)
* [Blockchain Engine Reorg Tests](running_tests/test_formats/blockchain_test_engine_reorg.md)
* [Blockchain Engine X Tests](running_tests/test_formats/blockchain_test_engine_x.md)
* [EOF Tests](running_tests/test_formats/eof_test.md)
* [Transaction Tests](running_tests/test_formats/transaction_test.md)
* [Common Types](running_tests/test_formats/common_types.md)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
# Blockchain Engine Reorg Tests <!-- markdownlint-disable MD051 (MD051=link-fragments "Link fragments should be valid") -->
# Blockchain Engine X Tests <!-- markdownlint-disable MD051 (MD051=link-fragments "Link fragments should be valid") -->

The Blockchain Engine Reorg Test fixture format tests are included in the fixtures subdirectory `blockchain_tests_engine_reorg`, and use Engine API directives with optimized shared pre-allocation for improved execution performance.
The Blockchain Engine X Test fixture format tests are included in the fixtures subdirectory `blockchain_tests_engine_x`, and use Engine API directives with optimized pre-allocation groups for improved execution performance.

These are produced by the `StateTest` and `BlockchainTest` test specs when using the `--generate-shared-pre` and `--use-shared-pre` flags.
These are produced by the `StateTest` and `BlockchainTest` test specs when using the `--generate-pre-alloc-groups` and `--use-pre-alloc-groups` flags.

## Description

The Blockchain Engine Reorg Test fixture format is an optimized variant of the [Blockchain Engine Test](./blockchain_test_engine.md) format designed for large-scale test execution with performance optimizations.
The Blockchain Engine X Test fixture format is an optimized variant of the [Blockchain Engine Test](./blockchain_test_engine.md) format designed for large-scale test execution with performance optimizations.

It uses the Engine API to test block validation and consensus rules while leveraging **shared pre-allocation state** to significantly reduce test execution time and resource usage. Tests are grouped by their initial state (fork + environment + pre-allocation) and share common genesis states through blockchain reorganization.
It uses the Engine API to test block validation and consensus rules while leveraging **pre-allocation groups** to significantly reduce test execution time and resource usage. Tests are grouped by their initial state (fork + environment + pre-allocation). Each group is executed against the same client instance using a common genesis state.

The key optimization is that **clients need only be started once per group** instead of once per test (as in the original engine fixture format), dramatically improving execution performance for large test suites.

Instead of including large pre-allocation state in each test fixture, this format references a shared pre-allocation folder (`pre_alloc`) which includes all different pre-allocation combinations used for any test fixture group.
Instead of including large pre-allocation state in each test fixture, this format references a pre-allocation groups folder (`pre_alloc`) which contains all different pre-allocation combinations organized by group.

A single JSON fixture file is composed of a JSON object where each key-value pair is a different [`ReorgFixture`](#reorgfixture) test object, with the key string representing the test name.
A single JSON fixture file is composed of a JSON object where each key-value pair is a different [`BlockchainTestEngineXFixture`](#BlockchainTestEngineXFixture) test object, with the key string representing the test name.

The JSON file path plus the test name are used as the unique test identifier.

## Shared Pre-Allocation File
## Pre-Allocation Groups Folder

The `blockchain_tests_engine_reorg` directory contains a special directory `pre_alloc` that stores shared pre-allocation state file used by all tests in this format, one per pre-allocation group with the name of the pre-alloc hash. This folder is essential for test execution and must be present alongside the test fixtures.
The `blockchain_tests_engine_x` directory contains a special directory `pre_alloc` that stores pre-allocation group files used by all tests in this format, one per pre-allocation group with the name of the pre-alloc hash. This folder is essential for test execution and must be present alongside the test fixtures.

### Pre-Allocation File Structure
### Pre-Allocation Group File Structure

Each file in the `pre_alloc` folder corresponds to a pre-allocation hash to shared state groups:
Each file in the `pre_alloc` folder corresponds to a pre-allocation group identified by a hash:

```json
{
Expand All @@ -37,28 +37,28 @@ Each file in the `pre_alloc` folder corresponds to a pre-allocation hash to shar
}
```

#### SharedPreStateGroup Fields
#### Pre-Allocation Group Fields

- **`test_count`**: Number of tests sharing this pre-allocation group
- **`pre_account_count`**: Number of accounts in the shared pre-allocation state
- **`testIds`**: Array of test identifiers that use this shared state
- **`test_count`**: Number of tests in this pre-allocation group
- **`pre_account_count`**: Number of accounts in the pre-allocation group
- **`testIds`**: Array of test identifiers that belong to this group
- **`network`**: Fork name (e.g., "Prague", "Cancun")
- **`environment`**: Complete [`Environment`](./common_types.md#environment) object with execution context
- **`pre`**: Shared [`Alloc`](./common_types.md#alloc-mappingaddressaccount) object containing initial account states
- **`pre`**: Pre-allocation group [`Alloc`](./common_types.md#alloc-mappingaddressaccount) object containing initial account states

## Consumption

For each [`ReorgFixture`](#reorgfixture) test object in the JSON fixture file, perform the following steps:
For each [`BlockchainTestEngineXFixture`](#BlockchainTestEngineXFixture) test object in the JSON fixture file, perform the following steps:

1. **Load Shared Pre-Allocation**:
1. **Load Pre-Allocation Group**:
- Read the appropriate file from the `pre_alloc` folder in the same directory
- Locate the shared state group using [`preHash`](#-prehash-string)
- Extract the `pre` allocation and `environment` from the shared group
- Locate the pre-allocation group using [`preHash`](#-prehash-string)
- Extract the `pre` allocation and `environment` from the group

2. **Initialize Client**:
- Use [`network`](#-network-fork) to configure the execution fork schedule
- Use the shared `pre` allocation as the starting state
- Use the shared `environment` as the execution context
- Use the pre-allocation group's `pre` allocation as the starting state
- Use the pre-allocation group's `environment` as the execution context
- Use [`genesisBlockHeader`](#-genesisblockheader-fixtureheader) as the genesis block header

3. **Execute Engine API Sequence**:
Expand All @@ -70,13 +70,13 @@ For each [`ReorgFixture`](#reorgfixture) test object in the JSON fixture file, p
4. **Verify Final State**:
- Compare the final chain head against [`lastblockhash`](#-lastblockhash-hash)
- If [`postStateDiff`](#-poststatediff-optionalalloc) is present:
- Apply the state differences to the shared pre-allocation
- Apply the state differences to the pre-allocation group
- Verify the resulting state matches the client's final state
- If `post` field were present (not typical), verify it directly

## Structures

### `ReorgFixture`
### `BlockchainTestEngineXFixture`

#### - `network`: [`Fork`](./common_types.md#fork)

Expand All @@ -88,39 +88,39 @@ This field is going to be replaced by the value contained in `config.network`.

#### - `preHash`: `string`

Hash identifier referencing a shared pre-allocation group in the `pre_alloc` folder. This hash uniquely identifies the combination of fork, environment, and pre-allocation state shared by multiple tests.
Hash identifier referencing a pre-allocation group in the `pre_alloc` folder. This hash uniquely identifies the combination of fork, environment, and pre-allocation state that defines the group.

#### - `genesisBlockHeader`: [`FixtureHeader`](./blockchain_test.md#fixtureheader)

Genesis block header. The state root in this header must match the state root calculated from the shared pre-allocation referenced by [`preHash`](#-prehash-string).
Genesis block header. The state root in this header must match the state root calculated from the pre-allocation group referenced by [`preHash`](#-prehash-string).

#### - `engineNewPayloads`: [`List`](./common_types.md#list)`[`[`FixtureEngineNewPayload`](#fixtureenginenewpayload)`]`

List of `engine_newPayloadVX` directives to be processed after the genesis block. These define the sequence of blocks to be executed via the Engine API.

#### - `syncPayload`: [`Optional`](./common_types.md#optional)`[`[`FixtureEngineNewPayload`](#fixtureenginenewpayload)`]`

Optional synchronization payload used for blockchain reorganization scenarios. When present, this payload is typically used to sync the chain to a specific state before or after the main payload sequence.
Optional synchronization payload. When present, this payload is typically used to sync the chain to a specific state before or after the main payload sequence.

#### - `lastblockhash`: [`Hash`](./common_types.md#hash)

Hash of the last valid block after all payloads have been processed, or the genesis block hash if all payloads are invalid.

#### - `postStateDiff`: [`Optional`](./common_types.md#optional)`[`[`Alloc`](./common_types.md#alloc-mappingaddressaccount)`]`

State differences from the shared pre-allocation state after test execution. This optimization stores only the accounts that changed, were created, or were deleted during test execution, rather than the complete final state.
State differences from the pre-allocation group after test execution. This optimization stores only the accounts that changed, were created, or were deleted during test execution, rather than the complete final state.

To reconstruct the final state:

1. Start with the shared pre-allocation from the `pre_alloc` folder
1. Start with the pre-allocation group from the `pre_alloc` folder
2. Apply the changes in `postStateDiff`:
- **Modified accounts**: Replace existing accounts with new values
- **New accounts**: Add accounts not present in pre-allocation
- **Deleted accounts**: Remove accounts (represented as `null` values)

#### - `config`: [`FixtureConfig`](#fixtureconfig)

Chain configuration object to be applied to the client running the blockchain engine reorg test.
Chain configuration object to be applied to the client running the blockchain engine x test.

### `FixtureConfig`

Expand All @@ -138,8 +138,7 @@ Engine API payload structure identical to the one defined in [Blockchain Engine

## Usage Notes

- This format is only generated when using `--generate-shared-pre` and `--use-shared-pre` flags
- This format is only generated when using `--generate-pre-alloc-groups` and `--use-pre-alloc-groups` flags
- The `pre_alloc` folder is essential and must be distributed with the test fixtures
- Tests are grouped by identical (fork + environment + pre-allocation) combinations
- The format is optimized for Engine API testing (post-Paris forks)
- Reorganization scenarios are supported through the `forkChoiceUpdate` mechanism
8 changes: 4 additions & 4 deletions src/cli/extract_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from ethereum_test_fixtures import BlockchainFixtureCommon
from ethereum_test_fixtures.blockchain import FixtureHeader
from ethereum_test_fixtures.file import Fixtures
from ethereum_test_fixtures.shared_alloc import SharedPreStateGroup
from ethereum_test_fixtures.pre_alloc_groups import PreAllocGroup
from ethereum_test_forks import Fork
from pytest_plugins.consume.hive_simulators.ruleset import ruleset

Expand Down Expand Up @@ -111,9 +111,9 @@ def create_genesis_from_fixture(fixture_path: Path) -> Tuple[FixtureHeader, Allo
alloc = fixture.pre
chain_id = int(fixture.config.chain_id)
else:
shared_alloc = SharedPreStateGroup.model_validate(fixture_json)
genesis = shared_alloc.genesis # type: ignore
alloc = shared_alloc.pre
pre_alloc_group = PreAllocGroup.model_validate(fixture_json)
genesis = pre_alloc_group.genesis # type: ignore
alloc = pre_alloc_group.pre

return genesis, alloc, chain_id

Expand Down
Loading
Loading