Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(gnovm)!: store refactor #2655

Draft
wants to merge 11 commits into
base: master
Choose a base branch
from
Draft

Conversation

thehowl
Copy link
Member

@thehowl thehowl commented Aug 2, 2024

This draft PR introduces a re-organized Store interface, with a few goals as compared to the current Store implementation:

  • Split the Store interface into smaller interfaces, and replace usages of the Store interface in the GnoVM codebase to only use the subset they need.
    • This has the main advantage of simplifying unit testing and mocking; allowing to create smaller implementations for what needs to be tested.
    • But furthermore, it makes explicit the extent to which a given function will attempt to access the Store.
  • Split the functionality of existing stores into wrappers with the philosophy of "do one thing and do it well", and allow composition of different stores depending on necessity.
    • As an example, an ObjectStore will eventually be 3 store implementations stacked together:
      1. A "reference manager" store, which when marshaling changes "compacts" many values to RefValues, and many types to RefTypes; and when unmarshaling expands RefTypes to their corresponding types. (This is done currently by functions like copyValueWithRefs / fillTypesOfValue).
      2. A "cache" store, which holds the result of unmarshaling values from the Store, and of new value
      3. The underlying ObjectStore, which retrieves and sets data in the tendermint 2 store.
    • Another type of store will be the "transaction log" store, which would exist for the current cacheTypes and cacheNodes: it keeps the values to be written, until Write() is called and the values are committed to their underlying "cache" maps.
  • Remove any methods which could be functions or options; or other specific implementations
    • For instance, now that PackageGetter is never used on-chain, the Store itself does not need to have a PackageGetter; the TestStore in gnovm/tests can make its own implementation which does the on-the-fly loading of packages and standard libraries.
    • In another instance, instead of setting NativeResolver and PackageInjector we can now set them at startup using StoreOptions - seeing as in our practical usages these are initialised once then never again
  • Ensure that the Store interfaces can be better adapted for implementations in non-blockchain contexts, rather than trying to fit defaultStore for all use-cases.

More development and notes will follow. For now, let's try to merge #2319 as a stepping stone first.

@github-actions github-actions bot added the 📦 🤖 gnovm Issues or PRs gnovm related label Aug 2, 2024
@thehowl thehowl changed the title begin store refactor feat(gnovm)!: store refactor Aug 2, 2024
Copy link

codecov bot commented Aug 2, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 54.88%. Comparing base (74df3ea) to head (d6b813d).

Additional details and impacted files
@@                    Coverage Diff                    @@
##           dev/morgan/issue-2283    #2655      +/-   ##
=========================================================
- Coverage                  54.89%   54.88%   -0.01%     
=========================================================
  Files                        593      593              
  Lines                      81579    81567      -12     
=========================================================
- Hits                       44782    44770      -12     
+ Misses                     33382    33380       -2     
- Partials                    3415     3417       +2     
Flag Coverage Δ
contribs/gnofaucet 14.46% <ø> (ø)
contribs/gnomd 0.00% <ø> (ø)
misc/autocounterd 0.00% <ø> (ø)
misc/genstd 80.54% <ø> (ø)
misc/goscan 0.00% <ø> (ø)
misc/logos 17.38% <ø> (-0.31%) ⬇️
misc/loop 0.00% <ø> (ø)
tm2 54.46% <ø> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ajnavarro ajnavarro self-requested a review August 27, 2024 09:51
thehowl added a commit that referenced this pull request Sep 5, 2024
This pull request modifies the current `defaultStore` to support
transactionality in the same way as our tm2 stores. A few new concepts
are introduced:

- A `TransactionStore` interface is added, which extends the
`gnolang.Store` interface to support a Write() method.
- Together with corresponding implementations allowing for
transactionality on its cache maps, this means that the Gno store
retained by the VM keeper is only modified atomically after a
transaction has completed.
- The tm2 `BaseApp` has the new "hooks" `BeginTxHook` and `EndTxHook`.
The former is called right before starting a transaction, and the latter
is called right after finishing it, together with the transaction
result.
- This allows us to plug in the Gno `TransactionalStore` in the
`sdk.Context` through the `BeginTxHook`; and commit the result, if
successful, in the `EndTxHook`.
## Overview of changes

- `gno.land`
- `pkg/gnoland/app.go`: the InitChainer is now additionally responsible
of loading standard libraries. To separate the options related to app
startup globally, and those to genesis, the InitChainer is now a method
of its config struct, `InitChainerConfig`, embedded into the
`AppOptions`.
- `pkg/gnoland/app.go`: `NewAppOptions` is only used in `NewApp`, where
most of its fields were modified, anyway. I replaced it by changing the
`validate()` method to write default values.
- `pkg/gnoland/node_inmemory.go`,
`pkg/integration/testing_integration.go`: these changes were made
necessary to support `gnoland restart` in our txtars, and supporting
fast reloading of standard libraries (`LoadStdlibCached`).
- `pkg/sdk/vm/keeper.go`: removed all the code to load standard
libraries on Initialize, as it's now done in the InitChainer. The hack
introduced in #2568 is no longer necessary as well. Other changes show
how the Gno Store is injected and retrieved from the `sdk.Context`.
- `gnovm/pkg/gnolang/store.go`
- Fork and SwapStores have been removed, in favour of BeginTransaction.
BeginTransaction creates a `TransactionalStore`; the
"transaction-scoped" fields of the defaultStore are swapped with
"transactional" versions (including the allocator, cacheObjects and
cacheTypes/cacheNodes - the latter write to a `txLogMap`).
- ClearObjectCache is still necessary for the case of a transaction with
multiple messages.
- The `Map` interface in `txlog` allows us to have a `txLog` data type
stacked on top of another. This is useful for the cases where we use
`BeginTransaction` in preprocess. (We previously used `Fork`.) See later
in the "open questions" section.
- I added an Iterator method on the `txlog.Map` interface - this will be
compatible with [RangeFunc](https://go.dev/wiki/RangefuncExperiment),
once we switch over to [go1.23](https://go.dev/doc/go1.23).
- `tm2/pkg/sdk`
- As previously mentioned, two hooks were added on the BaseApp to
support injecting application code right before starting a transaction
and then right after ending it; allowing us to inject the code relating
to the Gno.land store while maintaining the modules decoupled
- Other
- `gnovm/pkg/gnolang/machine.go` has a one-line fix for a bug printing
the machine, whereby it would panic if len(blocks) == 0.

## Open questions / notes

- TransactionalStore should be a different implementation altogether;
but decoupling the logic which is used within the stores without doing a
massive copy-and-paste of the entire defaultStore implementation is
non-trivial. See [this
comment](#2319 (comment)),
and [this PR](#2655) for a draft
proposed store refactor, which would render the store more modular and
testable.
- There is an alternative implementation, which in micro-benchmarks
would be somewhat faster, in place of the `txLog`; it's still in
`gnolang/internal/txlog/txlog_test.go` where it also has benchmarks. See
[1347c5f](1347c5f)
for the solution which uses `bufferedTxMap`, without using the
`txlog.Map` interface. The main problem with this approach is that it
does not support stacking bufferedTxMaps, thus there is a different
method to be used in preprocess - `preprocessFork()`.
Base automatically changed from dev/morgan/issue-2283 to master September 5, 2024 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📦 🤖 gnovm Issues or PRs gnovm related
Projects
Status: No status
Status: Triage
Development

Successfully merging this pull request may close these issues.

1 participant