Reconsider certain uses of property testing

[davesque]

What was wrong?

Though there is clear value in using property testing for certain kinds of tests (such as end-to-end encoding tests), I've recently had some doubts about some other ways we're using property testing.

For example:
https://github.com/ethereum/eth-abi/blob/master/tests/encoding/test_encoder_properties.py#L327-L395

To a large extent, that property test and others like it seem to be testing second implementations of functionality against the original implementations. This doesn't provide much value since the original implementation could have a flaw which wouldn't be revealed by testing it against a second implementation that uses the same logic.

How can it be fixed?

It seems like we should replace these kinds of property tests with traditional tests that check for correct behavior given specific inputs.

[carver]

No objection here. I find it very difficult to parse those tests for correctness, as is.

[davesque]

Thoughts on this @pipermerriam ?

[pipermerriam]

Agreed. Writing all of those tests was largely an experiment and I'd say it's failed as 1) they are very hard to read, and 2) they are effectively a second implementation of the logic (like you said).

[pipermerriam]

Simplifying them to something like the following might make a lot of sense:

• if it fails decoding: no assertions (unless we can think of good ones)
• assertions that numeric values are strictly positive for unsigned types.
• maybe assertions that decoded values are negative if we can reliably look at the right bit.
• assertions that numeric values are in the appropriate range for the given type.
• assertions that the value is an integer for types which should only yield integers.
• assertions that the resulting type is of the appropriate type for the decoder.
• assertions that on encoding the resulting value is the right length for any fixed length encoders.