all: Support ethereum/tests StateTest format

.github/workflows/fixtures.yaml

@@ -19,21 +19,11 @@ jobs:
             fill-params: ''
             solc: '0.8.21'
             python: '3.11'
-          - name: 'fixtures_hive'
-            evm-type: 'main'
-            fill-params: '--enable-hive --from=Merge'
-            solc: '0.8.21'
-            python: '3.11'
           - name: 'fixtures_develop'
             evm-type: 'develop'
             fill-params: '--until=Cancun'
             solc: '0.8.21'
             python: '3.11'
-          - name: 'fixtures_develop_hive'
-            evm-type: 'develop'
-            fill-params: '--enable-hive --from=Merge --until=Cancun'
-            solc: '0.8.21'
-            python: '3.11'
     steps:
       - uses: actions/checkout@v3
         with:

README.md

@@ -68,10 +68,11 @@ The following transition tools are supported by the framework:
 
 ### Upcoming EIP Development
 
-Generally, specific `t8n` implementations and branches must be used when developing tests for upcoming EIPs (last updated 2023-10-19):
+Generally, specific `t8n` implementations and branches must be used when developing tests for upcoming EIPs.
 
-- Cancun related EIPs (4844, 4788, 1153, 6780) - [lightclient/go-ethereum@devnet-10](https://github.com/lightclient/go-ethereum/tree/devnet-10)
-- EOF tests - [ethereum/evmone@master](https://github.com/ethereum/evmone)
+We use named reference tags to point to the specific version of the `t8n` implementation that needs to be used fill the tests.
+
+All current tags, their t8n implementation and branch they point to, are listed in [evm-config.yaml](evm-config.yaml).
 
 ## Getting Started
 

docs/CHANGELOG.md

@@ -9,6 +9,27 @@ Test fixtures for use by clients are available for each release on the [Github r
 ### 🧪 Test Cases
 
 - 🔀 Add reentrancy suicide revert test ([#372](https://github.com/ethereum/execution-spec-tests/pull/372)).
+- 🔀 BlockchainTests converted to StateTest (also automatically generate BlockchainTest) ([#368](https://github.com/ethereum/execution-spec-tests/pull/368), [#370](https://github.com/ethereum/execution-spec-tests/pull/370)):
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_normal_gas
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_insufficient_balance_blob_tx
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_tx_blob_count
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_blob_hash_versioning_single_tx
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_tx_attribute_opcodes
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_tx_attribute_value_opcode
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_tx_attribute_calldata_opcodes
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_tx_attribute_gasprice_opcode
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_type_tx_pre_fork
+  - tests/cancun/eip4844_blobs/test_point_evaluation_precompile_gas.py::test_point_evaluation_precompile_gas_usage
+  - tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py::test_valid_precompile_calls
+  - tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py::test_invalid_precompile_calls
+  - tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py::test_point_evaluation_precompile_external_vectors
+  - tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py::test_point_evaluation_precompile_calls
+  - tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py::test_point_evaluation_precompile_gas_tx_to
+  - tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py::test_point_evaluation_precompile_before_fork
+  - tests/cancun/eip7516_blobgasfee/test_blobgasfee_opcode.py::test_blobbasefee_before_fork
+  - tests/shanghai/eip3860_initcode/test_initcode.py::test_contract_creating_tx
+  - tests/shanghai/eip3860_initcode/test_initcode.py::TestContractCreationGasUsage
+- 🐞 Fixed `tests/cancun/eip4844_blobs/test_blob_txs.py:test_invalid_tx_max_fee_per_blob_gas` to account for extra gas required in the case where the account is incorrectly deduced the balance as if it had the correct block blob gas fee ([#370](https://github.com/ethereum/execution-spec-tests/pull/370)).
 
 ### 🛠️ Framework
 
@@ -19,16 +40,25 @@ Test fixtures for use by clients are available for each release on the [Github r
 - 🔀 Locally calculate the transactions list's root instead of using the one returned by t8n when producing BlockchainTests ([#353](https://github.com/ethereum/execution-spec-tests/pull/353))
 - ✨ Fork objects used to write tests can now be compared using the `>`, `>=`, `<`, `<=` operators, to check for a fork being newer than, newer than or equal, older than, older than or equal, respectively when compared against other fork ([#367](https://github.com/ethereum/execution-spec-tests/pull/367))
 - 🐞 Storage type iterator is now fixed ([#369](https://github.com/ethereum/execution-spec-tests/pull/369))
+- 💥 Removed `--enable-hive` parameter, now all test types are generated by default ([#358](https://github.com/ethereum/execution-spec-tests/pull/358))
+- 💥 `StateTest`, spec format used to write tests, is now limited to a single transaction per test ([#361](https://github.com/ethereum/execution-spec-tests/pull/361))
+- ✨ `StateTestOnly`, spec format is now available and its only difference with `StateTest` is that it does not produce a `BlockchainTest` ([#368](https://github.com/ethereum/execution-spec-tests/pull/368))
 
 ### 🔧 EVM Tools
 
 ### 📋 Misc
 
 - 🔀 Docs: Update `t8n` tool branch to fill tests for development features in the [readme](https://github.com/ethereum/execution-spec-test) ([#338](https://github.com/ethereum/execution-spec-tests/pull/338)).
+- 🔀 Filling tool: Updated filling tool to go-ethereum@master ([#368](https://github.com/ethereum/execution-spec-tests/pull/368))
 
 ## Breaking Changes
 
-1. In this release the pytest node ID is now used for fixture names (previously only the test parameters were used), this should not be breaking. However, "=" in both node IDs and therefore fixture names, have been replaced with "_", which may break tooling that depends on the "=" character.
+1. Fixture output, including release tarballs, now contain subdirectories for different test types:
+   - `blockchain_tests`: Contains BlockchainTest formatted tests
+   - `blockchain_tests_hive`: Contains BlockchainTest with Engine API call directives for use in hive
+   - `state_tests`: Contains StateTest formatted tests
+2. `StateTest`, spec format used to write tests, is now limited to a single transaction per test.
+3. In this release the pytest node ID is now used for fixture names (previously only the test parameters were used), this should not be breaking. However, "=" in both node IDs and therefore fixture names, have been replaced with "_", which may break tooling that depends on the "=" character.
 
    Pytest node ID example:
 

docs/getting_started/debugging_t8n_tools.md

@@ -22,38 +22,39 @@ will produce the directory structure:
 
 ```text
 📂 /tmp/evm-dump
-└── 📂 berlin__eip2930_access_list__test_acl__test_access_list
-    └── 📂 fork_Berlin
-        ├── 📂 0
-        │   ├── 📄 args.py
-        │   ├── 📂 input
-        │   │   ├── 📄 alloc.json
-        │   │   ├── 📄 env.json
-        │   │   └── 📄 txs.json
-        │   ├── 📂 output
-        │   │   ├── 📄 alloc.json
-        │   │   ├── 📄 result.json
-        │   │   └── 📄 txs.rlp
-        │   ├── 📄 returncode.txt
-        │   ├── 📄 stderr.txt
-        │   ├── 📄 stdin.txt
-        │   ├── 📄 stdout.txt
-        │   └── 📄 t8n.sh
-        └── 📂 1
-            ├── 📄 args.py
-            ├── 📂 input
-            │   ├── 📄 alloc.json
-            │   ├── 📄 env.json
-            │   └── 📄 txs.json
-            ├── 📂 output
-            │   ├── 📄 alloc.json
-            │   ├── 📄 result.json
-            │   └── 📄 txs.rlp
-            ├── 📄 returncode.txt
-            ├── 📄 stderr.txt
-            ├── 📄 stdin.txt
-            ├── 📄 stdout.txt
-            └── 📄 t8n.sh
+└── 📂 blockchain_tests
+    └── 📂 berlin__eip2930_access_list__test_acl__test_access_list
+        └── 📂 fork_Berlin
+            ├── 📂 0
+            │   ├── 📄 args.py
+            │   ├── 📂 input
+            │   │   ├── 📄 alloc.json
+            │   │   ├── 📄 env.json
+            │   │   └── 📄 txs.json
+            │   ├── 📂 output
+            │   │   ├── 📄 alloc.json
+            │   │   ├── 📄 result.json
+            │   │   └── 📄 txs.rlp
+            │   ├── 📄 returncode.txt
+            │   ├── 📄 stderr.txt
+            │   ├── 📄 stdin.txt
+            │   ├── 📄 stdout.txt
+            │   └── 📄 t8n.sh
+            └── 📂 1
+                ├── 📄 args.py
+                ├── 📂 input
+                │   ├── 📄 alloc.json
+                │   ├── 📄 env.json
+                │   └── 📄 txs.json
+                ├── 📂 output
+                │   ├── 📄 alloc.json
+                │   ├── 📄 result.json
+                │   └── 📄 txs.rlp
+                ├── 📄 returncode.txt
+                ├── 📄 stderr.txt
+                ├── 📄 stdin.txt
+                ├── 📄 stdout.txt
+                └── 📄 t8n.sh
 ```
 
 where the directories `0` and `1` correspond to the different calls made to the `t8n` tool executed during the test:
@@ -120,24 +121,25 @@ will additionally run the `evm blocktest` command on every JSON fixture file and
 
 ```text
 📂 /tmp/evm-dump
-└── 📂 berlin__eip2930_access_list__test_acl__test_access_list
-    ├── 📄 fixtures.json
-    ├── 📂 fork_Berlin
-    │   ├── 📂 0
-    │   │   ├── 📄 args.py
-    │   │   ├── 📂 input
-    │   │   │   ├── 📄 alloc.json
-    │   │   │   ├── 📄 env.json
-    │   │   │   └── 📄 txs.json
-    │   │   ├── 📂 output
-    │   │   │   ├── 📄 alloc.json
-    │   ... ... ...
-    │
-    ├── 📄 verify_fixtures_args.py
-    ├── 📄 verify_fixtures_returncode.txt
-    ├── 📄 verify_fixtures.sh
-    ├── 📄 verify_fixtures_stderr.txt
-    └── 📄 verify_fixtures_stdout.txt
+└── 📂 blockchain_tests
+    └── 📂 berlin__eip2930_access_list__test_acl__test_access_list
+        ├── 📄 fixtures.json
+        ├── 📂 fork_Berlin
+        │   ├── 📂 0
+        │   │   ├── 📄 args.py
+        │   │   ├── 📂 input
+        │   │   │   ├── 📄 alloc.json
+        │   │   │   ├── 📄 env.json
+        │   │   │   └── 📄 txs.json
+        │   │   ├── 📂 output
+        │   │   │   ├── 📄 alloc.json
+        │   ... ... ...
+        │
+        ├── 📄 verify_fixtures_args.py
+        ├── 📄 verify_fixtures_returncode.txt
+        ├── 📄 verify_fixtures.sh
+        ├── 📄 verify_fixtures_stderr.txt
+        └── 📄 verify_fixtures_stdout.txt
 ```
 
 where the `verify_fixtures.sh` script can be used to reproduce the `evm blocktest` command.

docs/getting_started/executing_tests_command_line.md

@@ -152,7 +152,6 @@ Arguments defining filler location and output:
                         Don't group fixtures in JSON files by test function;
                         write each fixture to its own file. This can be used to
                         increase the granularity of --verify-fixtures.
-  --enable-hive         Output test fixtures with the hive-specific properties.
 
 Arguments defining debug behavior:
   --evm-dump-dir EVM_DUMP_DIR, --t8n-dump-dir EVM_DUMP_DIR

docs/getting_started/quick_start.md

@@ -84,7 +84,7 @@ The following requires a Python 3.10, 3.11 or 3.12 installation.
         2. The corresponding fixture file has been generated:
 
             ```console
-            head fixtures/berlin/eip2930_access_list/acl/access_list.json
+            head fixtures/blockchain_tests/berlin/eip2930_access_list/acl/access_list.json
             ```
 
 ## Next Steps

docs/getting_started/repository_overview.md

@@ -10,8 +10,9 @@ The most relevant folders and files in the repo are:
 │   ├── 📁 vm/
 │   └── 📁 ...
 ├─╴📁 fixtures/                  # default fixture output dir
-│   ├── 📁 eips/
-│   ├── 📁 vm/
+│   ├── 📁 blockchain_tests/
+│   ├── 📁 blockchain_tests_hive/
+│   ├── 📁 state_tests/
 │   └── 📁 ...
 ├─╴📁 src/                       # library & framework packages
 │   ├── 📁 ethereum_test_fork/

docs/getting_started/using_fixtures.md

@@ -37,10 +37,8 @@ The @ethereum/execution-spec-tests repository provides [releases](https://github
 | ------------------------------ | -------- | ------------------ |
 | `fixtures.tar.gz`              | Clients  | All tests until the last stable fork | "Must pass" |
 | `fixtures_develop.tar.gz`      | Clients  | All tests until the last development fork |
-| `fixtures_hive.tar.gaz`        | Hive     | All tests until the last stable fork in hive format |
-| `fixtures_develop_hive.tar.gz` | Hive     | All tests until the last development fork in hive format |
 
-The Hive format uses Engine API directives instead of the usual BlockchainTest format.
+The Hive format tests are included in subdirectory `blockchain_tests_hive` and these use Engine API directives instead of the usual BlockchainTest format.
 
 ## Obtaining the Most Recent Release Artifacts
 

docs/tutorials/state_transition.md

@@ -1,6 +1,6 @@
 # State Transition Tests
 
-This tutorial teaches you to create a state transition execution specification test. These tests verify that a blockchain, starting from a defined pre-state, will reach a specified post-state after executing a set of specific transactions.
+This tutorial teaches you to create a state transition execution specification test. These tests verify that a starting pre-state will reach a specified post-state after executing a single transaction.
 
 ## Pre-requisites
 
@@ -13,7 +13,7 @@ Before proceeding with this tutorial, it is assumed that you have prior knowledg
 
 ## Example Tests
 
-The most effective method of learning how to write tests is to study a couple of straightforward examples. In this tutorial we will go over the [Yul](https://github.com/ethereum/execution-spec-tests/blob/main/tests/example/test_yul_example.py#L17) state test.
+The most effective method of learning how to write tests is to study a couple of straightforward examples. In this tutorial we will go over the [Yul](https://github.com/ethereum/execution-spec-tests/blob/main/tests/homestead/yul/test_yul_example.py#L19) state test.
 
 ### Yul Test
 
@@ -83,7 +83,7 @@ The function definition ends when there is a line that is no longer indented. As
     env = Environment()
 ```
 
-This line specifies that `env` is an [`Environment`](https://github.com/ethereum/execution-spec-tests/blob/main/src/ethereum_test_tools/common/types.py#L445) object, and that we just use the default parameters.
+This line specifies that `env` is an [`Environment`](https://github.com/ethereum/execution-spec-tests/blob/main/src/ethereum_test_tools/common/types.py#L878) object, and that we just use the default parameters.
 If necessary we can modify the environment to have different block gas limits, block numbers, etc.
 In most tests the defaults are good enough.
 
@@ -102,7 +102,7 @@ It is a [dictionary](https://docs.python.org/3/tutorial/datastructures.html#dict
         "0x1000000000000000000000000000000000000000": Account(
 ```
 
-The keys of the dictionary are addresses (as strings), and the values are [`Account`](https://github.com/ethereum/execution-spec-tests/blob/main/src/ethereum_test_tools/common/types.py#L264) objects.
+The keys of the dictionary are addresses (as strings), and the values are [`Account`](https://github.com/ethereum/execution-spec-tests/blob/main/src/ethereum_test_tools/common/types.py#L517) objects.
 You can read more about address fields [in the static test documentation](https://ethereum-tests.readthedocs.io/en/latest/test_filler/state_filler.html#address-fields).
 
 ```python
@@ -145,7 +145,7 @@ Generally for execution spec tests the `sstore` instruction acts as a high-level
     }
 ```
 
-[`TestAddress`](https://github.com/ethereum/execution-spec-tests/blob/main/src/ethereum_test_tools/common/constants.py#L8) is an address for which the test filler has the private key.
+[`TestAddress`](https://github.com/ethereum/execution-spec-tests/blob/main/src/ethereum_test_tools/common/constants.py#L7) is an address for which the test filler has the private key.
 This means that the test runner can issue a transaction as that contract.
 Of course, this address also needs a balance to be able to issue transactions.
 
@@ -163,7 +163,7 @@ Of course, this address also needs a balance to be able to issue transactions.
     )
 ```
 
-With the pre-state specified, we can add a description for the [`Transaction`](https://github.com/ethereum/execution-spec-tests/blob/main/src/ethereum_test_tools/common/types.py#L516).
+With the pre-state specified, we can add a description for the [`Transaction`](https://github.com/ethereum/execution-spec-tests/blob/main/src/ethereum_test_tools/common/types.py#L1155).
 For more information, [see the static test documentation](https://ethereum-tests.readthedocs.io/en/latest/test_filler/state_filler.html#transaction)
 
 #### Post State
@@ -185,10 +185,10 @@ In this case, we look at the storage of the contract we called and add to it wha
 #### State Test
 
 ```python
-    state_test(env=env, pre=pre, post=post, txs=[tx])
+    state_test(env=env, pre=pre, post=post, tx=tx)
 ```
 
-This line calls the wrapper to the `StateTest` object that provides all the objects required (for example, the fork parameter) in order to fill the test, generate the test fixtures and write them to file (by default, `./fixtures/example/yul_example/test_yul.json`).
+This line calls the wrapper to the `StateTest` object that provides all the objects required (for example, the fork parameter) in order to fill the test, generate the test fixtures and write them to file (by default, `./fixtures/<blockchain,state>_tests/example/yul_example/test_yul.json`).
 
 ## Conclusion
 

docs/tutorials/state_transition_bad_opcode.md

@@ -267,7 +267,7 @@ Over the entire for loop, it yields 255 different tests.
     yield StateTest(
         env=env, 
         pre=pre, 
-        txs=[tx],
+        tx=tx,
         post=(post_valid if opc_valid(opc) else post_invalid),
     )
 ```

evm-config.yaml

@@ -4,5 +4,5 @@ main:
   ref: master
 develop:
   impl: geth
-  repo: lightclient/go-ethereum
-  ref: devnet-10
+  repo: ethereum/go-ethereum
+  ref: master

src/ethereum_test_forks/base_fork.py

@@ -143,6 +143,14 @@ def header_beacon_root_required(cls, block_number: int, timestamp: int) -> bool:
         """
         pass
 
+    @classmethod
+    @abstractmethod
+    def blob_gas_per_blob(cls, block_number: int, timestamp: int) -> int:
+        """
+        Returns the amount of blob gas used per blob for a given fork.
+        """
+        pass
+
     @classmethod
     @abstractmethod
     def get_reward(cls, block_number: int = 0, timestamp: int = 0) -> int: