Simple framework for gas benchmarking

[hashedone]

Gas usage benchmarking is a recurring topic, it would be nice to have simple utility for writing such benchmarks. It should be very simple to use, and with very much single responsibility - checking gas cost of performed calls. Under the hood real vm shall be used, with some help of its test utils, but benches on they own should not perform any verification (the behavior of contract is to be tested in UTs, and MTs, possibly some integration tests).

As a bonus it would be nice if the framework would report results in some common format, like json - it would make it probably much easier to integrate it with tools like grafana in future.

[ethanfrey]

This is done in several integration tests, but mainly in cosmwasm.

@maurolacy can show you a recent example he did which can be used as a template. (And keeping the vm dependency under a feature flag, so I can compile on my M1)

[maurolacy]

Sure, this has been done recently for benchmarking the PoE mixing functions. See https://github.com/confio/tgrade-contracts/blob/main/contracts/tg4-mixer/benches/main.rs.

Take into account this is not a full-fledged benchmark, though. I deliberately pass the Fn as an (optional) parameter, to avoid having to instantiate the contract first.

[hashedone]

@ethanfrey I went through this, but they are too limited - as they are using vm, they are not enough to test any kind of interaction between contracts (neither querying nor executing submessages). To do it this way, it would require to build another multitest-like wrapper to communicate those guys between them.

Even for testing the cw1-whitelist you want to test execute, and this is not possible with this approach.

[ethanfrey]

To do it this way, it would require to build another multitest-like wrapper to communicate those guys between them.

Let's build a new WasmModule implementation that uses the VM and wasm blobs rather than native calls.

This would also require adding some concept of gas metering to the multitest, and charging for storage, returned events, etc as well as the actual execution time.

But this seems like a large undertaking. I would push this off for some time. The ng vs current benchmarks are not that important. Once the framework is complete, we can look how to write such benchmarks. Or just use super simple ones based on isolated contract calls.

[maurolacy]

Benchmarking is in general a specific / punctual task. I mean, you usually want to measure the performance of some specific, isolated functionality.

For measuring the overall cost of execute, instantiate, etc. as we want to do in #505, perhaps a good option is to do it in a conventional way. That is, a CPU benchmark, in the context of a unit test (or multi-test). Instead of a gas usage benchmark, in the context of an integration test, and just for comparison purposes.

[uint]

@hashedone Is this really an ng contracts/Sylvia issue? This doesn't sound right

[JakeHartnell]

@de-husk implemented a framework, have a look: https://github.com/de-husk/cosm-orc

[uint]

Thanks, @JakeHartnell! I'm closing these issues since they're very old. If someone asks about gas profiling, I'll be sure to point them to cosm-orc.