feat(fw|ci): add fixture diff utility within CI and local CLI

[spencer-tb]

🗒️ Description

I thought I would give this a try. So far I've added an entry point diff_fixtures.py (like @danceratopz) suggested. This file will be used for both CI workflows and additionally for the CLI interface for local fixture diff'ing.

CI Workflows

diff_fixtures.py will have functions that the CI/CD workflows can use. My understanding is that we only need to be able to tell what files have changed. So far I am planning on adding additional json files that are created using these functions within the workflows, containing:

1. An ordered map of the fixture file name, to its content hashed (minus the _info section). Allowing us to check what files have changed specifically.
2. A cumulative hash of each fixture hash. Allowing us to simply check anything fixtures have changed.

Here is an example structure:

{
    "cumulative_hash": "ae79b35b75b95cea2fca707cb0750d08a65950160a63b50e5d9d4bba360e3a7e",
    "hash_map": {
        "cancun/eip4844_blobs/point_evaluation_precompile/point_evaluation_precompile_before_fork.json": [
            "6cb959f9baa5c76d60c5042c2434981bbc5994d065f2122b30f4998c9e075fa6"
        ],
        "cancun/eip4844_blobs/point_evaluation_precompile/point_evaluation_precompile_calls.json": [
            "e4e1affc2687a817d5cc971b3a6a73583b001c16091fefef0500d4f67514dbb5"
        ],
        ...

An additional step will be added to the build and release workflow, so that every time a PR is merged into main a new json file is created with the hashes. Similarly, I will add an equivalent file that is generated for each commit within a PR. This will be a separate workflow that will be added as an extenstion to the existing tox verifacations.

At this point we have 2 files that a comparison can be drawn from, using the cumulative hash first, and then the file by file hash afterwards. If we want to be more granular with diffing we can in the future. We can also only diff the files that we expect to change, to speed up the workflow.

Maybe a trigger within the PR will be nice to determine when the fixture diff should occur.

CLI Interface

The CLI command, I'm calling it dfx for now will be able to generate the same file locally and pull in the the json file generated from main, such that a diff can be created.

This will be more granular with more information, where as the workflows will ideally signal if a diff exists.

If we want we can add this to tox, but my gut feeling is that it should be a seperate utility.

🔗 Related Issues

#388

✅ Checklist

• All: Set appropriate labels for the changes.
• All: Considered squashing commits to improve commit history.
• All: Added an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

[spencer-tb]

I updated the fixture json comparison file to more granularly determine where changes occur.

We now have the following fixture json tree:

{
    "fixtures": {
        "cumulative_hash": "b71b7a8ae55c2e2490954f252b40ebd3b98da3e659130fd8f9f23a3bce26b6cc",
        "contents": {
            "state_tests": [
                {
                    "path": "state_tests",
                    "hash": "107dcae43f136fb5a6fb22020777edc54b2f990929be5e940b8da9442ae45c8a",
                    "contents": {
                        ...,
                        "shanghai": [
                            {
                                "path": "state_tests/shanghai",
                                "hash": "d8233b51482b811084565307274807cb47ade64f1faef8d7eb98ac1636045283",
                                "contents": {
                                    "eip3860_initcode": [
                                        {
                                            "path": "state_tests/shanghai/eip3860_initcode",
                                            "hash": "ec5777b2fc3b1c47810771e303fe7d0e75d72b3feb0717ce8f94066c4d15ad7b",
                                            "contents": {
                                                "initcode": [
                                                    {
                                                        "path": "state_tests/shanghai/eip3860_initcode/initcode",
                                                        "hash": "ec5777b2fc3b1c47810771e303fe7d0e75d72b3feb0717ce8f94066c4d15ad7b",
                                                        "contents": {
                                                            "create_opcode_initcode.json": [
                                                                {
                                                                    "path": "state_tests/shanghai/eip3860_initcode/initcode/create_opcode_initcode.json",
                                                                    "hash": "eb1e8aa17900b92f1d41db8ede48f40a43cafa886242c6a14f80cee79358fcad"
                                                                }
                                                            ],
                                                            "gas_usage.json": [
                                                                {
                                                                    "path": "state_tests/shanghai/eip3860_initcode/initcode/gas_usage.json",
                                                                    "hash": "9e8e463cbecc4d89d46e84ddf6c0f90a74f6d89891c235a511556798b70bb64a"
                                                                }
                                                            ],
...

This allows us to find out if changes occured at the following levels:

• test type, fork, category (eip4844_blobs), test module and fixture file

[spencer-tb]

When a PR is merged, during the build and release process on main we now also create the fixtures tree hash json artifact files. These can now be downloaded locally if the user has set thier github access token within thier .env file.

[spencer-tb]

Requires: #454 and #443

[spencer-tb]

Closing. This has gave me an idea though. We should only run the coverage script if the fixtures change. Not if the files change.