draft: block stm backports test #24488

mmsqe · 2025-04-11T15:48:19Z

Description

This PR includes block store feature #24159, block stm features and cachekv improvement backports, and used for test under crypto-org-chain/ethermint#564

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

included the correct type prefix in the PR title, you can find examples of the prefixes below:
confirmed ! in the type prefix if API or client breaking change
targeted the correct branch (see PR Targeting)
provided a link to the relevant issue or specification
reviewed "Files changed" and left comments if necessary
included the necessary unit and integration tests
added a changelog entry to CHANGELOG.md
updated the relevant documentation or specification, including comments for documenting Go code
confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

Please see Pull Request Reviewer section in the contributing guide for more information on how to review a pull request.

I have...

confirmed the correct type prefix in the PR title
confirmed all author checklist items have been addressed
reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

generic interface generic btree generic cachekv generic transient store support ObjStore changelog Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> object store key Apply review suggestions fix merge conflict fix snapshot revert dependers prefix store support object store (cosmos#236) Problem: snapshot for object store is not skipped (cosmos#585) resolve

….53.x_obj_store

…tore

fix and add test

…tore

* Problem: MultiStore interface is bloated Solution: - Split out specialied methods from it, keeping the MultiStore generic * Update store/CHANGELOG.md Signed-off-by: yihuang <[email protected]> --------- Signed-off-by: yihuang <[email protected]>

* Problem: nested cache store not efficient Solution: - introduce copy-on-write btree based cache store temp * changelog * rename * Update store/cachekv/store.go Signed-off-by: yihuang <[email protected]> --------- Signed-off-by: yihuang <[email protected]>

Solution: - init cachestore on cachestore lazily. - cleanup some unused stuff. Update store/CHANGELOG.md Signed-off-by: yihuang <[email protected]>

add basic support in sdk: - add a TxExecutor baseapp option - add TxIndex/TxCount/MsgIndex in context Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> fix misspell fix lint run gci fix lint gci seems not compatible with gofumpt

…s#237) * Problem: no efficient way to collect fee Solution: - support an idea of virtual account in bank module, where the incoming coins are accumulated in a per-tx object store first, then accumulate and credit to the real account at end blocker. it's nesserary to support parallel tx execution, where we try not to access shared states. more efficient sum support SendCoinsFromModuleToAccountVirtual fix test fix test * fix lint * fix test * fix test * fix test * fix test * fix test * fix mock keeper * try fix lint * try fix lint * reuse code * try fix linter * Update x/bank/keeper/send.go Signed-off-by: yihuang <[email protected]> * algin panic call * fix error handling * try fix lint * nolintlint generate falst postiive --------- Signed-off-by: yihuang <[email protected]>

* Support RunAtomic API * add unit test

fix and add test

Solution: - fix and add test to support the usage pattern in ethermint add Discard method to CacheWrap better testing

* feat: save some memory allocations on un-used cache store * Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> * Update store/internal/btree/btree.go Signed-off-by: yihuang <[email protected]> --------- Signed-off-by: yihuang <[email protected]>

Solution: - fix the way context is updated

Solution: - add checks

…osmos#258) Solution: - add API NewFromParent to cache multistore. Update store/CHANGELOG.md Signed-off-by: yihuang <[email protected]> fix test fix lint

* mutex don't cover GetSigners don't clone header in context * Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> * update comment --------- Signed-off-by: yihuang <[email protected]>

…osmos#507) * Problem: mempool don't respect gas wanted returned by ante handler Solution: - support custom gas wanted in mempool * Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> * cleanup * cleanup * fix priorityIndex * fix process proposal * fix lint * fix lint --------- Signed-off-by: yihuang <[email protected]>

* Problem: redundant mutex for InsertWithGasWanted cfg of PriorityNonceMempool remains unchanged once assigned, so no lock is required * make mocks * cleanup * keep order of check MaxTx

… of same tx (cosmos#565) * Problem: signature verification result not cache between incarnations of same tx Closes: cosmos#564 Solution: - introduce incarnation cache that's shared between incarnations of the same tx * Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> * Update types/context.go Signed-off-by: yihuang <[email protected]> * Update types/context.go Signed-off-by: yihuang <[email protected]> * fix nil convert --------- Signed-off-by: yihuang <[email protected]> Problem: gas consumed differs after enabled cache (cosmos#570) * Problem: gas consumed differs after enabled cache * fix test * Revert "Problem: gas consumed differs after enabled cache" This reverts commit f33944e. * Revert "Problem: signature verification result not cache between incarnations of same tx (cosmos#565)" This reverts commit 5a1594f. * keep interface

* Problem: tx executor can't do dependency analysis Solution: - change the api to allow static analysis on tx body * fix * changelog * cleanup * Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> --------- Signed-off-by: yihuang <[email protected]>

ironbird-prod · 2025-04-11T15:48:21Z

Ironbird - launch a network

To use Ironbird, you can use the following commands:

/ironbird start - Launch a testnet with the specified chain and load test configuration.
/ironbird chains - List of chain images that ironbird can use to spin-up testnet
/ironbird loadtests - List of load test modes that ironbird can run against testnet

baseapp/abci_utils.go

+					txGasLimit = gasTx.GetGas()
+				}
+
+				stop := h.txSelector.SelectTxForProposal(ctx, uint64(req.MaxTxBytes), maxBlockGas, tx, txBz, txGasLimit)


baseapp/abci_utils.go

 			} else {
-				stop := h.txSelector.SelectTxForProposal(ctx, uint64(req.MaxTxBytes), maxBlockGas, memTx, txBz)
+				stop := h.txSelector.SelectTxForProposal(ctx, uint64(req.MaxTxBytes), maxBlockGas, memTx.Tx, txBz, memTx.GasWanted)


store/gaskv/store.go

 	gs.gasMeter.ConsumeGas(gs.gasConfig.ReadCostFlat, types.GasReadCostFlatDesc)
 	value = gs.parent.Get(key)

 	// TODO overflow-safe math?
 	gs.gasMeter.ConsumeGas(gs.gasConfig.ReadCostPerByte*types.Gas(len(key)), types.GasReadPerByteDesc)
-	gs.gasMeter.ConsumeGas(gs.gasConfig.ReadCostPerByte*types.Gas(len(value)), types.GasReadPerByteDesc)
+	gs.gasMeter.ConsumeGas(gs.gasConfig.ReadCostPerByte*types.Gas(gs.valueLen(value)), types.GasReadPerByteDesc)


store/gaskv/store.go

 	gs.gasMeter.ConsumeGas(gs.gasConfig.WriteCostFlat, types.GasWriteCostFlatDesc)
 	// TODO overflow-safe math?
 	gs.gasMeter.ConsumeGas(gs.gasConfig.WriteCostPerByte*types.Gas(len(key)), types.GasWritePerByteDesc)
-	gs.gasMeter.ConsumeGas(gs.gasConfig.WriteCostPerByte*types.Gas(len(value)), types.GasWritePerByteDesc)
+	gs.gasMeter.ConsumeGas(gs.gasConfig.WriteCostPerByte*types.Gas(gs.valueLen(value)), types.GasWritePerByteDesc)


store/gaskv/store.go

 	if gi.Valid() {
 		key := gi.Key()
 		value := gi.Value()

 		gi.gasMeter.ConsumeGas(gi.gasConfig.ReadCostPerByte*types.Gas(len(key)), types.GasValuePerByteDesc)
-		gi.gasMeter.ConsumeGas(gi.gasConfig.ReadCostPerByte*types.Gas(len(value)), types.GasValuePerByteDesc)
+		gi.gasMeter.ConsumeGas(gi.gasConfig.ReadCostPerByte*types.Gas(gi.valueLen(value)), types.GasValuePerByteDesc)


x/bank/keeper/virtual.go

+
+	key := make([]byte, len(addr)+8)
+	copy(key, addr)
+	binary.BigEndian.PutUint64(key[len(addr):], uint64(sdkCtx.TxIndex()))


x/bank/keeper/virtual.go

+
+	key := make([]byte, len(addr)+8)
+	copy(key, addr)
+	binary.BigEndian.PutUint64(key[len(addr):], uint64(sdkCtx.TxIndex()))


store/cachemulti/store.go

+	for _, store := range cms.stores {
+		store.Discard()
+	}


store/cachemulti/store.go

+		store = cms.initStore(key, cms.parentStore(key))
+	}
+	if key == nil || store == nil {
+		panic(fmt.Sprintf("kv store with key %v has not been registered in stores", key))


store/cachemulti/store.go

+func (cms Store) GetObjKVStore(key types.StoreKey) types.ObjKVStore {
+	store, ok := cms.getCacheWrapper(key).(types.ObjKVStore)
+	if !ok {
+		panic(fmt.Sprintf("store with key %v is not ObjKVStore", key))


store/cachemulti/store.go

+	for k, v := range cms.stores {
+		stores[k] = v.(types.BranchStore).Clone().(types.CacheWrap)
+	}


store/cachemulti/store.go

+	for k, v := range cms.stores {
+		if _, ok := other.stores[k]; !ok {
+			// clear the cache store if it's not in the other
+			v.Discard()
+		}
+	}


store/rootmulti/store.go

+		panic(fmt.Sprintf("store with key %v is not ObjKVStore", key))
+	}
+
+	return store


store/types/validity.go

+// AssertValidValueGeneric checks if the value is valid(value is not nil and within length limit)
+func AssertValidValueGeneric[V any](value V, isZero func(V) bool, valueLen func(V) int) {
+	if isZero(value) {
+		panic("value is nil")


store/types/validity.go

+		panic("value is nil")
+	}
+	AssertValidValueLength(valueLen(value))
+}


testutil/integration/router.go

+	for key := range okeys {
+		cms.MountStoreWithDB(okeys[key], storetypes.StoreTypeObject, nil)
+	}


x/bank/module.go

@@ -303,3 +306,7 @@

 	return nil
 }
+
+func (am AppModule) EndBlock(ctx context.Context) error {
+	return am.keeper.CreditVirtualAccounts(ctx)


This reverts commit dc5c16b.

yihuang and others added 30 commits March 27, 2025 16:15

cleanup

17043e4

Merge remote-tracking branch 'origin/release/v0.53.x' into release/v0…

78e3a48

….53.x_obj_store

fix test

f44f06b

lint

e1ee61f

Merge remote-tracking branch 'origin/main' into release/v0.53.x_obj_s…

b75bb98

…tore

resolve

e64ccc0

fix test

d2c1c3b

fix test

8892029

Problem: store key type assertion is incorrect (cosmos#244)

83f4859

fix and add test

Merge remote-tracking branch 'origin/main' into release/v0.53.x_obj_s…

d97d3d1

…tore

Problem: many memory allocs in cache store wrapping (cosmos#242)

ed5c416

Solution: - init cachestore on cachestore lazily. - cleanup some unused stuff. Update store/CHANGELOG.md Signed-off-by: yihuang <[email protected]>

Problem: no API to use the new CoW branch store (cosmos#243)

34d33dc

* Support RunAtomic API * add unit test

Problem: store key type assertion is incorrect (cosmos#244)

fb9fc8a

fix and add test

Problem: Restore don't work snapshot revert usage (cosmos#245)

f90fc95

Solution: - fix and add test to support the usage pattern in ethermint add Discard method to CacheWrap better testing

Problem: block gas used not set in context (cosmos#252)

13dca67

Solution: - fix the way context is updated

Problem: invalid tx returns negative gas wanted (cosmos#253)

d3485c7

Solution: - add checks

Problem: no api to create cachemulti store from external cache store (c…

ee6aa36

…osmos#258) Solution: - add API NewFromParent to cache multistore. Update store/CHANGELOG.md Signed-off-by: yihuang <[email protected]> fix test fix lint

small optimisations (cosmos#261)

a08c72e

* mutex don't cover GetSigners don't clone header in context * Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> * update comment --------- Signed-off-by: yihuang <[email protected]>

Problem: mempool.GasTx interface is not reused (cosmos#536)

524d2c1

* Problem: redundant mutex for InsertWithGasWanted cfg of PriorityNonceMempool remains unchanged once assigned, so no lock is required * make mocks * cleanup * keep order of check MaxTx

mod tidy

8f9c18b

cleanup

fb805e3

feat(x/tx): support bytes field as signer (cosmos#21850)

df7ea7b

github-project-automation bot added this to Interchain Public Works Apr 11, 2025

github-actions bot added C:CLI C:x/gov C:Store C:x/bank C:x/evidence C:x/upgrade C:x/feegrant C:x/group C:x/nft C:x/circuit labels Apr 11, 2025

github-advanced-security bot found potential problems Apr 11, 2025

View reviewed changes

github-actions bot added the C:x/tx label Apr 12, 2025

mmsqe added 3 commits April 12, 2025 19:12

Merge remote-tracking branch 'origin/main' into obj_main

9b09181

Revert "fix (sims): state written in sims loop (cosmos#20936)"

79e09b2

This reverts commit dc5c16b.

Merge remote-tracking branch 'origin/main' into obj_main

4187a9d

github-actions bot added the C:Simulations label Apr 15, 2025

mmsqe mentioned this pull request Apr 15, 2025

feat: implement object stores #24159

Open

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

draft: block stm backports test #24488

draft: block stm backports test #24488

Uh oh!

mmsqe commented Apr 11, 2025 •

edited

Loading

Uh oh!

ironbird-prod bot commented Apr 11, 2025

Uh oh!

Check failure

Check failure

Check failure

Check failure

Check failure

Check failure

Check failure

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Uh oh!

draft: block stm backports test #24488

Are you sure you want to change the base?

draft: block stm backports test #24488

Uh oh!

Conversation

mmsqe commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Author Checklist

Reviewers Checklist

Uh oh!

ironbird-prod bot commented Apr 11, 2025

Uh oh!

Check failure

Check failure

Check failure

Check failure

Check failure

Check failure

Check failure

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Uh oh!

mmsqe commented Apr 11, 2025 •

edited

Loading