Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

R4R: Store Refactor 1 #2985

Merged
merged 36 commits into from
Feb 2, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
a669428
in progress
mossid Dec 3, 2018
83dac5d
store/* now compiles
mossid Dec 3, 2018
fe1209a
make store/gas independent from types/, now pass ci
mossid Dec 3, 2018
c40055e
remove residue, rename files to match with convention
mossid Dec 3, 2018
ff0e9dc
fix lint
mossid Dec 3, 2018
b08044a
replace dependency types/ -> store/types/
mossid Dec 3, 2018
26e05ae
update pending
mossid Dec 3, 2018
324a849
fix gas.NewStore argument order
mossid Dec 3, 2018
9d806d6
rename {gas, cache, trace} -> {gas, cache, trace}kv
mossid Dec 5, 2018
cb8006b
Merge branch 'develop' into joon/store-refactor-1
cwgoes Dec 5, 2018
9876e30
adress comments in progress
mossid Dec 14, 2018
676e837
Merge branch 'joon/store-refactor-1' of github.com:cosmos/cosmos-sdk …
mossid Dec 14, 2018
ab8143e
Merge branch 'develop' into joon/store-refactor-1
mossid Dec 14, 2018
86b19d4
fix server
mossid Dec 14, 2018
2c316e3
address comments in progress
mossid Dec 14, 2018
43ab0c3
Merge branch 'develop' into joon/store-refactor-1
jackzampolin Dec 20, 2018
30a4a58
addressing comments
mossid Dec 29, 2018
4eda02b
address comments
mossid Jan 5, 2019
04631ad
fix test
mossid Jan 5, 2019
2429ef4
Merge branch 'develop' into joon/store-refactor-1
mossid Jan 21, 2019
cad00ca
address comments
mossid Jan 25, 2019
a1aaba3
Merge branch 'develop' into joon/store-refactor-1
mossid Jan 25, 2019
d2e8044
Merge branch 'develop' of github.com:cosmos/cosmos-sdk into joon/stor…
mossid Jan 25, 2019
fe063b3
mv Cp() back to store
mossid Jan 25, 2019
20e02f7
Update store/cachemulti/store.go
melekes Jan 29, 2019
67c5541
Update store/dbadapter/store.go
melekes Jan 29, 2019
11c202a
address comments in progress
mossid Jan 29, 2019
1e68319
Merge branch 'joon/store-refactor-1' of github.com:cosmos/cosmos-sdk …
mossid Jan 29, 2019
e1f3fec
address comments
mossid Jan 29, 2019
9b16f2f
Merge branch 'develop' into joon/store-refactor-1
Jan 30, 2019
13470c3
add README, fix lint, add AssertValidKey in prefix.Store
mossid Jan 30, 2019
997d88d
Merge branch 'joon/store-refactor-1' of github.com:cosmos/cosmos-sdk …
mossid Jan 30, 2019
af69aeb
modify README
mossid Jan 30, 2019
7cd7880
Update store/README.md
cwgoes Jan 31, 2019
ab82454
Update store/README.md
cwgoes Jan 31, 2019
ae9eeb9
address comments
mossid Jan 31, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions PENDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ IMPROVEMENTS
* [\#3420](https://github.com/cosmos/cosmos-sdk/issues/3420) Added maximum length to governance proposal descriptions and titles

* SDK
* [\#2986](https://github.com/cosmos/cosmos-sdk/pull/2986) Store Refactor
* \#3435 Test that store implementations do not allow nil values

* Tendermint
Expand Down
9 changes: 4 additions & 5 deletions baseapp/baseapp.go
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ func (app *BaseApp) Name() string {
// SetCommitMultiStoreTracer sets the store tracer on the BaseApp's underlying
// CommitMultiStore.
func (app *BaseApp) SetCommitMultiStoreTracer(w io.Writer) {
app.cms.WithTracer(w)
app.cms.SetTracer(w)
}

// Mount IAVL or DB stores to the provided keys in the BaseApp multistore
Expand Down Expand Up @@ -483,8 +483,7 @@ func handleQueryCustom(app *BaseApp, path []string, req abci.RequestQuery) (res
// BeginBlock implements the ABCI application interface.
func (app *BaseApp) BeginBlock(req abci.RequestBeginBlock) (res abci.ResponseBeginBlock) {
if app.cms.TracingEnabled() {
app.cms.ResetTraceContext()
app.cms.WithTracingContext(sdk.TraceContext(
app.cms.SetTracingContext(sdk.TraceContext(
map[string]interface{}{"blockHeight": req.Header.Height},
))
}
Expand Down Expand Up @@ -679,7 +678,7 @@ func (app *BaseApp) cacheTxContext(ctx sdk.Context, txBytes []byte) (
// TODO: https://github.com/cosmos/cosmos-sdk/issues/2824
msCache := ms.CacheMultiStore()
if msCache.TracingEnabled() {
msCache = msCache.WithTracingContext(
msCache = msCache.SetTracingContext(
sdk.TraceContext(
map[string]interface{}{
"txHash": fmt.Sprintf("%X", tmhash.Sum(txBytes)),
Expand Down Expand Up @@ -813,7 +812,7 @@ func (app *BaseApp) runTx(mode runTxMode, txBytes []byte, tx sdk.Tx) (result sdk
// EndBlock implements the ABCI application interface.
func (app *BaseApp) EndBlock(req abci.RequestEndBlock) (res abci.ResponseEndBlock) {
if app.deliverState.ms.TracingEnabled() {
app.deliverState.ms = app.deliverState.ms.ResetTraceContext().(sdk.CacheMultiStore)
app.deliverState.ms = app.deliverState.ms.SetTracingContext(nil).(sdk.CacheMultiStore)
}

if app.endBlocker != nil {
Expand Down
2 changes: 1 addition & 1 deletion baseapp/baseapp_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
"os"
"testing"

"github.com/cosmos/cosmos-sdk/store"
store "github.com/cosmos/cosmos-sdk/store/types"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand Down
7 changes: 3 additions & 4 deletions client/context/query.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,14 @@ import (

"strings"

"github.com/cosmos/cosmos-sdk/store/rootmulti"
abci "github.com/tendermint/tendermint/abci/types"
"github.com/tendermint/tendermint/crypto/merkle"
cmn "github.com/tendermint/tendermint/libs/common"
tmliteErr "github.com/tendermint/tendermint/lite/errors"
tmliteProxy "github.com/tendermint/tendermint/lite/proxy"
rpcclient "github.com/tendermint/tendermint/rpc/client"
tmtypes "github.com/tendermint/tendermint/types"

"github.com/cosmos/cosmos-sdk/store"
)

// GetNode returns an RPC client. If the context's client is not defined, an
Expand Down Expand Up @@ -207,7 +206,7 @@ func (ctx CLIContext) verifyProof(queryPath string, resp abci.ResponseQuery) err
}

// TODO: Instead of reconstructing, stash on CLIContext field?
prt := store.DefaultProofRuntime()
prt := rootmulti.DefaultProofRuntime()

// TODO: Better convention for path?
storeName, err := parseQueryStorePath(queryPath)
Expand Down Expand Up @@ -247,7 +246,7 @@ func isQueryStoreWithProof(path string) bool {
return false
case paths[0] != "store":
return false
case store.RequireProof("/" + paths[2]):
case rootmulti.RequireProof("/" + paths[2]):
return true
}

Expand Down
2 changes: 1 addition & 1 deletion cmd/gaia/cmd/gaiad/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ func main() {
func newApp(logger log.Logger, db dbm.DB, traceStore io.Writer) abci.Application {
return app.NewGaiaApp(
logger, db, traceStore, true,
baseapp.SetPruning(store.NewPruningOptions(viper.GetString("pruning"))),
baseapp.SetPruning(store.NewPruningOptionsFromString(viper.GetString("pruning"))),
baseapp.SetMinGasPrices(viper.GetString(server.FlagMinGasPrices)),
)
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/gaia/cmd/gaiadebug/hack.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ func runHackCmd(cmd *cobra.Command, args []string) error {
fmt.Println(err)
os.Exit(1)
}
app := NewGaiaApp(logger, db, baseapp.SetPruning(store.NewPruningOptions(viper.GetString("pruning"))))
app := NewGaiaApp(logger, db, baseapp.SetPruning(store.NewPruningOptionsFromString(viper.GetString("pruning"))))

// print some info
id := app.LastCommitID()
Expand Down
8 changes: 2 additions & 6 deletions server/mock/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,19 +26,15 @@ func (ms multiStore) CacheWrapWithTrace(_ io.Writer, _ sdk.TraceContext) sdk.Cac
panic("not implemented")
}

func (ms multiStore) ResetTraceContext() sdk.MultiStore {
panic("not implemented")
}

func (ms multiStore) TracingEnabled() bool {
panic("not implemented")
}

func (ms multiStore) WithTracingContext(tc sdk.TraceContext) sdk.MultiStore {
func (ms multiStore) SetTracingContext(tc sdk.TraceContext) sdk.MultiStore {
panic("not implemented")
}

func (ms multiStore) WithTracer(w io.Writer) sdk.MultiStore {
func (ms multiStore) SetTracer(w io.Writer) sdk.MultiStore {
panic("not implemented")
}

Expand Down
130 changes: 130 additions & 0 deletions store/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# Store

## CacheKV

`cachekv.Store` is a wrapper `KVStore` which provides buffered writing / cached reading functionalities over the underlying `KVStore`.

```go
type Store struct {
cache map[string]cValue
parent types.KVStore
}
```

### Get

`Store.Get()` checks `Store.cache` first in order to find if there is any cached value associated with the key. If the value exists, the function returns it. If not, the function calls `Store.parent.Get()`, sets the key-value pair to the `Store.cache`, and returns it.

### Set

`Store.Set()` sets the key-value pair to the `Store.cache`. `cValue` has the field `dirty bool` which indicates whether the cached value is different from the underlying value. When `Store.Set()` cache new pair, the `cValue.dirty` is set true so when `Store.Write()` is called it can be written to the underlying store.

### Iterator

`Store.Iterator()` have to traverse on both caches items and the original items. In `Store.iterator()`, two iterators are generated for each of them, and merged. `memIterator` is essentially a slice of the `KVPair`s, used for cached items. `mergeIterator` is a combination of two iterators, where traverse happens ordered on both iterators.

## CacheMulti

`cachemulti.Store` is a wrapper `MultiStore` which provides buffered writing / cached reading functionalities over the underlying `MutliStore`

```go
type Store struct {
db types.CacheKVStore
stores map[types.StoreKey] types.CacheWrap
}
```

`cachemulti.Store` cache wraps all substores in its constructor and hold them in `Store.stores`. `Store.GetKVStore()` returns the store from `Store.stores`, and `Store.Write()` recursively calls `CacheWrap.Write()` on the substores.

## DBAdapter

`dbadapter.Store` is a adapter for `dbm.DB` making it fulfilling the `KVStore` interface.

```go
type Store struct {
dbm.DB
}
```

`dbadapter.Store` embeds `dbm.DB`, so most of the `KVStore` interface functions are implemented. The other functions(mostly miscellaneous) are manually implemented.

## IAVL

`iavl.Store` is a base-layer self-balancing merkle tree. It is guaranteed that

1. Get & set operations are `O(log n)`, where `n` is the number of elements in the tree
2. Iteration efficiently returns the sorted elements within the range
3. Each tree version is immutable and can be retrieved even after a commit(depending on the pruning settings)

Specification and implementation of IAVL tree can be found in [https://github.com/tendermint/iavl].

## GasKV

`gaskv.Store` is a wrapper `KVStore` which provides gas consuming functionalities over the underlying `KVStore`.

```go
type Store struct {
gasMeter types.GasMeter
gasConfig types.GasConfig
parent types.KVStore
}
```

When each `KVStore` methods are called, `gaskv.Store` automatically consumes appropriate amount of gas depending on the `Store.gasConfig`.


## Prefix

`prefix.Store` is a wrapper `KVStore` which provides automatic key-prefixing functionalities over the underlying `KVStore`.

```go
type Store struct {
parent types.KVStore
prefix []byte
}
```

When `Store.{Get, Set}()` is called, the store forwards the call to its parent, with the key prefixed with the `Store.prefix`.

When `Store.Iterator()` is called, it does not simply prefix the `Store.prefix`, since it does not work as intended. In that case, some of the elements are traversed even they are not starting with the prefix.

## RootMulti

`rootmulti.Store` is a base-layer `MultiStore` where multiple `KVStore` can be mounted on it and retrieved via object-capability keys. The keys are memory addresses, so it is impossible to forge the key unless an object is a valid owner(or a receiver) of the key, according to the object capability principles.

## TraceKV

`tracekv.Store` is a wrapper `KVStore` which provides operation tracing functionalities over the underlying `KVStore`.

```go
type Store struct {
parent types.KVStore
writer io.Writer
context types.TraceContext
}
```

When each `KVStore` methods are called, `tracekv.Store` automatically logs `traceOperation` to the `Store.writer`.

```go
type traceOperation struct {
Operation operation
Key string
Value string
Metadata map[string]interface{}
}
```

`traceOperation.Metadata` is filled with `Store.context` when it is not nil. `TraceContext` is a `map[string]interface{}`.

## Transient

`transient.Store` is a base-layer `KVStore` which is automatically discarded at the end of the block.

```go
type Store struct {
dbadapter.Store
}
```

`Store.Store` is a `dbadapter.Store` with a `dbm.NewMemDB()`. All `KVStore` methods are reused. When `Store.Commit()` is called, new `dbadapter.Store` is assigned, discarding previous reference and making it garbage collected.
2 changes: 1 addition & 1 deletion store/memiterator.go → store/cachekv/memiterator.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package store
package cachekv

import (
"bytes"
Expand Down
12 changes: 7 additions & 5 deletions store/cachemergeiterator.go → store/cachekv/mergeiterator.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
package store
package cachekv

import (
"bytes"

"github.com/cosmos/cosmos-sdk/store/types"
)

// cacheMergeIterator merges a parent Iterator and a cache Iterator.
Expand All @@ -12,14 +14,14 @@ import (
//
// TODO: Optimize by memoizing.
type cacheMergeIterator struct {
parent Iterator
cache Iterator
parent types.Iterator
cache types.Iterator
ascending bool
}

var _ Iterator = (*cacheMergeIterator)(nil)
var _ types.Iterator = (*cacheMergeIterator)(nil)

func newCacheMergeIterator(parent, cache Iterator, ascending bool) *cacheMergeIterator {
func newCacheMergeIterator(parent, cache types.Iterator, ascending bool) *cacheMergeIterator {
iter := &cacheMergeIterator{
parent: parent,
cache: cache,
Expand Down
Loading