Ensure state tests only run against the intended fork #1470
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The statetest fixtures have different definitions for each fork and we must ensure to only run test against against the intended fork (e.g run Constantinople state tests against Constantinople VM). We can not rely on `pytest -k` matching for that as that matches against an identification string that includes the path and name of the test file which in some cases also contains fork names. A test file may be named "ConstantinopleSomething.json" but still contains individual definitions per fork.
cburgdorf
force-pushed
the
christoph/tests/do-not-rely-on-k-matching
branch
from
ede82e7 to
ba309fd
Compare
cburgdorf
changed the title
pipermerriam
approved these changes
Nov 15, 2018
|
It's pretty nice that this resulted in a non-trivial drop in total number of tests we run. 👍 |
8 tasks
veox
added a commit
to veox/trinity
that referenced
this pull request
May 16, 2019
Mostly a copy-paste from `py-evm` codebase; relevant PRs that introduced the functionality there: ethereum/py-evm#1470 ethereum/py-evm#1577
veox
added a commit
to veox/trinity
that referenced
this pull request
May 20, 2019
Mostly a copy-paste from `py-evm` codebase; relevant PRs that introduced the functionality there: ethereum/py-evm#1470 ethereum/py-evm#1577
veox
added a commit
to veox/trinity
that referenced
this pull request
May 21, 2019
Mostly a copy-paste from `py-evm` codebase; relevant PRs that introduced the functionality there: ethereum/py-evm#1470 ethereum/py-evm#1577
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What was wrong?
The statetest fixtures have different definitions
for each fork and we must ensure to only run
test against against the intended fork (e.g run
Constantinople state tests against Constantinople VM).
We can not rely on
pytest -kmatching for thatas that matches against an identification string that
includes the path and name of the test file which in
some cases also contains fork names. A test file may
be named "ConstantinopleSomething.json" but still
contains individual definitions per fork.
How was it fixed?
Add a mechanism so that the intended fork can be passed as
--forkand then internally filter the generated fixtures against that.Notice that this does only apply to state tests as these are the only test files expands into multiple test cases per fork.
The following table compares the number of executed tests per fork based on this new scheme.
Cute Animal Picture