fix(gno.land): make gno store cache respect rollbacks

[thehowl]

This pull request modifies the current defaultStore to support transactionality in the same way as our tm2 stores. A few new concepts are introduced:

• A TransactionStore interface is added, which extends the gnolang.Store interface to support a Write() method.
  • Together with corresponding implementations allowing for transactionality on its cache maps, this means that the Gno store retained by the VM keeper is only modified atomically after a transaction has completed.
• The tm2 BaseApp has the new "hooks" BeginTxHook and EndTxHook. The former is called right before starting a transaction, and the latter is called right after finishing it, together with the transaction result.
  • This allows us to plug in the Gno TransactionalStore in the sdk.Context through the BeginTxHook; and commit the result, if successful, in the EndTxHook.

Overview of changes

• gno.land
  • pkg/gnoland/app.go: the InitChainer is now additionally responsible of loading standard libraries. To separate the options related to app startup globally, and those to genesis, the InitChainer is now a method of its config struct, InitChainerConfig, embedded into the AppOptions.
  • pkg/gnoland/app.go: NewAppOptions is only used in NewApp, where most of its fields were modified, anyway. I replaced it by changing the validate() method to write default values.
  • pkg/gnoland/node_inmemory.go, pkg/integration/testing_integration.go: these changes were made necessary to support gnoland restart in our txtars, and supporting fast reloading of standard libraries (LoadStdlibCached).
  • pkg/sdk/vm/keeper.go: removed all the code to load standard libraries on Initialize, as it's now done in the InitChainer. The hack introduced in fix(sdk/vm): re-load iavl store from backup if empty #2568 is no longer necessary as well. Other changes show how the Gno Store is injected and retrieved from the sdk.Context.
• gnovm/pkg/gnolang/store.go
  • Fork and SwapStores have been removed, in favour of BeginTransaction. BeginTransaction creates a TransactionalStore; the "transaction-scoped" fields of the defaultStore are swapped with "transactional" versions (including the allocator, cacheObjects and cacheTypes/cacheNodes - the latter write to a txLogMap).
  • ClearObjectCache is still necessary for the case of a transaction with multiple messages.
  • The Map interface in txlog allows us to have a txLog data type stacked on top of another. This is useful for the cases where we use BeginTransaction in preprocess. (We previously used Fork.) See later in the "open questions" section.
  • I added an Iterator method on the txlog.Map interface - this will be compatible with RangeFunc, once we switch over to go1.23.
• tm2/pkg/sdk
  • As previously mentioned, two hooks were added on the BaseApp to support injecting application code right before starting a transaction and then right after ending it; allowing us to inject the code relating to the Gno.land store while maintaining the modules decoupled
• Other
  • gnovm/pkg/gnolang/machine.go has a one-line fix for a bug printing the machine, whereby it would panic if len(blocks) == 0.

Open questions / notes

• TransactionalStore should be a different implementation altogether; but decoupling the logic which is used within the stores without doing a massive copy-and-paste of the entire defaultStore implementation is non-trivial. See this comment, and this PR for a draft proposed store refactor, which would render the store more modular and testable.
• There is an alternative implementation, which in micro-benchmarks would be somewhat faster, in place of the txLog; it's still in gnolang/internal/txlog/txlog_test.go where it also has benchmarks. See 1347c5f for the solution which uses bufferedTxMap, without using the txlog.Map interface. The main problem with this approach is that it does not support stacking bufferedTxMaps, thus there is a different method to be used in preprocess - preprocessFork().

[codecov]

Codecov Report

Attention: Patch coverage is 77.40113% with 80 lines in your changes missing coverage. Please review.

Project coverage is 60.43%. Comparing base (65ee7a5) to head (c27c30c).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
gno.land/pkg/gnoland/app.go	78.72%	16 Missing and 4 partials ⚠️
gno.land/pkg/integration/testing_integration.go	40.74%	12 Missing and 4 partials ⚠️
gno.land/pkg/gnoland/node_inmemory.go	33.33%	12 Missing ⚠️
gnovm/pkg/gnolang/store.go	84.00%	10 Missing and 2 partials ⚠️
gno.land/pkg/sdk/vm/keeper.go	85.18%	7 Missing and 1 partial ⚠️
gnovm/pkg/gnolang/internal/txlog/txlog.go	92.00%	2 Missing and 2 partials ⚠️
tm2/pkg/sdk/baseapp.go	0.00%	2 Missing and 2 partials ⚠️
gno.land/pkg/integration/testing_node.go	88.23%	2 Missing ⚠️
gnovm/pkg/gnolang/machine.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2319      +/-   ##
==========================================
+ Coverage   60.24%   60.43%   +0.19%     
==========================================
  Files         562      563       +1     
  Lines       75031    75149     +118     
==========================================
+ Hits        45200    45419     +219     
+ Misses      26458    26342     -116     
- Partials     3373     3388      +15     

Flag	Coverage Δ
contribs/gnodev	60.65% <100.00%> (-0.76%)	⬇️
contribs/gnofaucet	15.31% <ø> (ø)
gno.land	67.21% <72.38%> (+2.64%)	⬆️
gnovm	64.46% <86.04%> (+0.12%)	⬆️
misc/genstd	80.54% <ø> (ø)
misc/logos	20.23% <ø> (+0.35%)	⬆️
tm2	62.06% <66.66%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thehowl]

A current update with the state of the code:

I'm trying to work "transaction" logic in the GnoVM store. The current store is not "transactional" (as in the database significance - ie. a set of operations that are performed together, by "committing" them or rolling them back) because if a transaction is rolled back, the "cache" fields of the GnoVM store are not.

The end goal is to make it have transaction logic, plugging it into this existing mechanism:

gno/tm2/pkg/sdk/baseapp.go

Lines 829 to 843 in 8aafb6c

	// Create a new context based off of the existing context with a cache wrapped
	// multi-store in case message processing fails.
	runMsgCtx, msCache := app.cacheTxContext(ctx, txBytes)
	result = app.runMsgs(runMsgCtx, msgs, mode)
	result.GasWanted = gasWanted
	// Safety check: don't write the cache state unless we're in DeliverTx.
	if mode != RunTxModeDeliver {
	return result
	}
	// only update state if all messages pass
	if result.IsOK() {
	msCache.MultiWrite()
	}

ie. have it as one of the Stores in the MultiStore, so that the BaseApp only calls Write to "commit" the changes to the database if all the messages in the transaction have passed.

So I've been trying to get the store out of the vm keeper, and into its own store which is plugged into other places.

At the time being, I've done the changes to the GnoVM store needed to accomodate committing and rolling back to a parent store.

Right now, trying to run a gnoland node quickly leads to a consensus failure. My hunch is that this comes from the fact that the current gnostore package creates its own base/iavl stores, while to be correct it should use the existing ones.

The GnoVM tests are succeeding

[jaekwon]

gno.TypeCheckMemPackage() is used by the keeper to check the type first w/ MsgAddPackage, but during type checking Go's type checker will import some mempackages, but the mempackage is first run by the import getter function.

The gno.Store's types are not cleared after a DeliverTx, nor are the nodes.
We cannot completely clear the type cache because that would require also the nodes to be re-preprocessed.
This is a major "known" limitation of the store, that upon restart all mempackages need to be re-preprocessed,
all nodes and types are cached.
If we simply call ClearCache() upon a type check failure in the screenshot below (in keeper MsgAddPkg), the next transaction will fail because the type is actually stored on disk, with reference to mem package nodes that got cleared.
So, currently we can't call ClearCache() without also re-preprocessing all the known mempackages.

I think the easiest way to solve this problem right now is to change the implementation of the usage of the type checker so that it doesn't call store.GetMemPackage, but only reads the mempackage through ReadMemPackage(). The type checker doesn't care about gno state, I think it only cares about the source code, so there's no need to run "GetMemPackage" which also runs the mempackage.

The hard way to solve this is to complete the implementation of the store and fix everything so that types and nodes can be cleared, and restored correctly lazily. Lazily loading a package where the nodes were cleared, would also involve preprocessing the loaded mempackage, but we wouldn't need to run it since it was already run before.

I think a hacky way to solve this is to insert a gno.Write() below:

	// Validate Gno syntax and type check.
	if err := gno.TypeCheckMemPackage(memPkg, gnostore); err != nil {
		gnostore.Write() // <--- hacky solution
		return ErrTypeCheck(err)
	}

Then the values constructed from store.GetMemPackage() will be stored, and a stdlib that was once initialized won't be re-initialized. Since the package value of "encoding/base64" wasn't saved the first time upon a type check (or something like that), store.GetMemPackage is called a second time in a later transaction for the same "encoding/base64", but it fails because the type is found from the store the second time around:

// in preprocess.go
				case *DeclaredType:
					// if store has this type, use that.
					tid := DeclaredTypeID(lastpn.PkgPath, n.Name)
					exists := false
					if dt := store.GetTypeSafe(tid); dt != nil {
						// XXX SECOND PASS HERE XXX
						dst = dt.(*DeclaredType)
						last.GetValueRef(store, n.Name, true).SetType(dst)
						exists = true
					}
					if !exists {
						// XXX FIRST PASS HERE XXX 
						// otherwise construct new *DeclaredType.
						// NOTE: this is where declared types are
						// actually instantiated, not in
						// machine.go:runDeclaration().
						dt2 := declareWith(lastpn.PkgPath, n.Name, tmp)
						// if !n.IsAlias { // not sure why this was here.
						dt2.Seal()
						// }
						*dst = *dt2
					}

Another solution might be to not use the store in the preprocessor for this purpose but I'm not sure what other problems will be encountered with this approach.

Either way for now I think the easiest solution is to not use GetMemPackage but ReadMemPackage for the type checker. (We shouldn't be running init() for the same package redundantly anwyays, so unless we're going to do the hacky gnostore.Write(), just not doing any initialization seems better.) The way to do that is, I think, instead of passing in a mere packageGetter callback to the gno.Store, we pass in an object that implements two methods separately; GetPackage() and GetMemPackage(). That way the type checker can call GetMemPackage, while during preprocessing of any code that imports a yet-unloaded stdlib, the store would end up calling GetPackage() which actually performs the initialization.

[jaekwon]

Maybe the solution being worked on here will work, but I'm not sure.
I also noticed that commit 8afb1a4 changed the impl of Fork() so that it map.Clones the types but keeps the cacheNode the same? That seems weird.

[thehowl]

I think the easiest way to solve this problem right now is to change the implementation of the usage of the type checker so that it doesn't call store.GetMemPackage, but only reads the mempackage through ReadMemPackage(). The type checker doesn't care about gno state, I think it only cares about the source code, so there's no need to run "GetMemPackage" which also runs the mempackage.

I read this this morning and got thinking to how this could be done -- ie., how I came up with #2504 :)

Since all stdlibs are eagerly loaded at startup, there is no needed for the package getter on-chain; so the scenario which is trying to be fixed here.

The issue is not fully solved, I'm sure, which is why I want to keep this PR and the underlying idea open. Ie., if we don't roll-back together with the transaction, a succeeding message can modify the GnoVM store (and its cache), to then be followed by a failing transaction; this sequence changes the GnoVM store, but not the underlying baseStore/iavlStore.

The hard way to solve this is to complete the implementation of the store and fix everything so that types and nodes can be cleared, and restored correctly lazily. Lazily loading a package where the nodes were cleared, would also involve preprocessing the loaded mempackage, but we wouldn't need to run it since it was already run before.

Yes, I tried going for that in a first approach (making the GnoVM store one which doesn't actually "store" anything, and make the word "cache" representative of what the maps are actually doing). But then I realised that it was kind of a more herculean task For Another Time, due to the amount of data structures involved in the serialization of BlockNodes.

I also noticed that commit 8afb1a4 changed the impl of Fork() so that it map.Clones the types but keeps the cacheNode the same? That seems weird.

Trust me, I would have loved to make it all maps.Clone if I could 😞

Unluckily, cloning the blocknodes is slow on a decently large store (enough to be very noticeable); so I decided against it. It was one of the things I wanted to "permanently" fix with this PR.

[jaekwon]

Agreed. Maybe we need a tag "RevisitLater" and close a bunch of PRs/issues for our peace of mind.

[thehowl]

I'm going to continue working on this in an effort to have a fix for #2605

[thehowl]

There is some cleanup to be done; tests are failing and need to be adjusted. But an empirical start of the chain and restart now runs successfully.

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

[thehowl]

Okay, the coverage is up to ~78%, and I'm well beyond the point where I'm adding tests on areas that are barely touched by this PR; so I'll stop here.

I addressed the issue that Milos brought up, which was a relatively simple change.

@zivkovicmilos, if you have a minute, take a second look. If it looks good to you I think we can likely push it for a final check with Jae.

@ltzmaxwell, if you have a minute, take a look at this: #2319 (comment)

[ajnavarro]

The code seems as good as it could be respecting the actual architecture. Waiting for @zivkovicmilos comments to be addressed to approve.

[zivkovicmilos]

Thank you for showing this bug that the core team's got hands too :)

[thehowl]

Merging for now, as this PR is important to move us forward. @jaekwon, feel free to review later. As I told you, I'd be happy if most of the work undertaken here could be simplified, eventually, by (a) making the caching work on a msg level rather than being permanent from chain start and (b) possibly have caching where it makes sense (ie. for BlockNodes and such) happen in a properly managed in-memory cache rather than being an append-only cache which would potentially end up using a lot of memory.

[piux2]

I'm not sure if this should be named 'commit.' Here, it only writes to the store, while the actual commit happens in Commit(), not in EndBlock().

[piux2]

I'm not sure if this should be named 'Commit.' It does not commit anything.

[thehowl]

You're welcome to make a PR to propose a rename to Write - or anything else

[thehowl]

(My reasoning here for the name comes from the terminology also used in SQL; which uses BEGIN, ROLLBACK and COMMIT)

[piux2]

"The tm2 BaseApp has the new "hooks" BeginTxHook and EndTxHook. The former is called right before starting a transaction, and the latter is called right after finishing it, together with the transaction result.
This allows us to plug in the Gno TransactionalStore in the sdk.Context through the BeginTxHook; and commit the result, if successful, in the EndTxHook."

Instead of creating an App.BeginTxHook and App.EndBlockHook, to keep ABCI clean, what do you think about implementing these at the module level as the VM keeper's BeginBlocker and EndBlocker, which are called within BaseApp.BeginBlock and BaseApp.EndBlocker?

[thehowl]

Instead of creating an App.BeginTxHook and App.EndBlockHook, to keep ABCI clean, what do you think about implementing these at the module level as the VM keeper's BeginBlocker and EndBlocker, which are called within BaseApp.BeginBlock and BaseApp.EndBlocker?

It's important we do rollbacks precisely as they are done for the underlying store.

The store rollbacks ALL changes from ALL messages in a transaction, but only in the failing transaction.

Scoping this at the block level would mean that the "entire block fails".

[piux2]

Thank you for the explanation, that makes sense. 


Here are two questions.

1. The logic of these injected hooks is specifically for vm gnoStore. All it does is mark preprocessed types and nodes in gnoStore.cachTypes and gnoStore.cachNode as successful in the txlog.source in memory when the AddPkg is executed successfully.

   It solves the original cached type and cached node issue, in which the preprocess ran out of gas. The underlying cache-wrapped MultiStore gnoStore.ivalStore and gnoStore.baseStore do not persist (equivalent to rollback), while the gnoStore.cachObject is cleared. Preprocessed gnoStore.cachType and gnoStore.cachType can not be cleared as a "limitation." ( equivalent to not roll back)

   As an alternative, what if we simply capture the panic in vm.AddPkg and remove preprocessed types and nodes from gnoStore.cachTypes and gnoStore.cachNodes for the message with the same txlog implementation?

   Will this achieve the same result with minimal structural changes to BaseApp without introducing a new transactional store and transaction concept on top of the existing cache-wrapped ms flow?

2. The core "limitation" that Jae mentioned above still exists—gnoStore.cachedType and cachNode cannot be cleared. Should they be cleared in general, even if the original issue is resolved in the current solution, where cacheType and cacheNode are still kept in memory as a separate cache on top of the cache-wrapped MultiStore?

[thehowl]

@piux2

1. As an alternative, what if we simply capture the panic in vm.AddPkg and remove preprocessed types and nodes from gnoStore.cachTypes and gnoStore.cachNodes for the message with the same txlog implementation?
   Will this achieve the same result with minimal structural changes to BaseApp without introducing a new transactional store and transaction concept on top of the existing cache-wrapped ms flow?

Remember, if any message in a transaction fails, then the entire transaction should fail. If we tackle this at the level of AddPkg (or other message handlers), then for a multi-message transaction we're performing only a partial rollback, because they will rollback the changes done to the gno Store within that same message, but not the changes done to the gno store by other messages. (While in the underlying tm2 Stores, all changes from all messages are rolled back, of course).

The core "limitation" that Jae mentioned above still exists—gnoStore.cachedType and cachNode cannot be cleared. Should they be cleared in general, even if the original issue is resolved in the current solution, where cacheType and cacheNode are still kept in memory as a separate cache on top of the cache-wrapped MultiStore?

In general, yes, but the underlying changes are a bit more complex, involve discussion and were out of scope here.

My current thinking is that:

• cacheTypes and cacheNodes should be scoped specifically to the message being run
• All PackageValues should keep a graph of all other packages they depend on (both directly - through imports - but also indirectly, through interface values)
• So that when we call a realm function or do a MsgRun with an import, we can do Preprocess on all of the imported packages on the fly, constructing cacheNodes and cacheTypes.

Of course the result of Preprocess itself can be cached, so in a real-life scenario we don't have to run it often, especially for standard libraries and very common packages. But it means also that we're not bounded by the nodes' RAM in how many packages we can have on-chain.