Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: ADR-065 Store V2 #15028

Merged
merged 22 commits into from
Mar 16, 2023
Merged

docs: ADR-065 Store V2 #15028

merged 22 commits into from
Mar 16, 2023

Conversation

alexanderbez
Copy link
Contributor

@alexanderbez alexanderbez commented Feb 14, 2023

Description

This ADR is currently a collaborative living document to outline the proposed changes the design and implementation of the storage primitives in the Cosmos SDK.

RENDERED


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • included the correct type prefix in the PR title
  • added ! to the type prefix if API or client breaking change
  • targeted the correct branch (see PR Targeting)
  • provided a link to the relevant issue or specification
  • followed the guidelines for building modules
  • included the necessary unit and integration tests
  • added a changelog entry to CHANGELOG.md
  • included comments for documenting Go code
  • updated the relevant documentation or specification
  • reviewed "Files changed" and left comments if necessary
  • confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed ! in the type prefix if API or client breaking change
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic
  • reviewed API design and naming
  • reviewed documentation is accurate
  • reviewed tests and test coverage
  • manually tested (if applicable)

@alexanderbez alexanderbez added T:Docs Changes and features related to documentation. T: ADR An issue or PR relating to an architectural decision record labels Feb 14, 2023
@alexanderbez alexanderbez mentioned this pull request Feb 21, 2023
19 tasks
to change the backing commitment store in the future should evidence arise to
warrant a better alternative. However there is promising work being done to IAVL
that should result in significant performance improvement <sup>[1,2]</sup>.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one thing to note with this refactor is we should think about removing the fast node system in iavl because it was a poor mans SS & SC separation. With this redesign we can remove it

Comment on lines 115 to 117
However, in order for the SC layer to remain lightweight and not duplicate a
majority of the data held in the SS layer, we need to have an aggressive pruning
strategy. A pruning strategy could be enforced programmatically or through
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you define what duplicate means here? sc isnt meant to store the values in ss. this would be duplicating data unnecessarily

Comment on lines 120 to 122
a pruning strategy might be too limiting. So we propose to have a configurable
pruning strategy, height retention and interval, with a default height retention
of 2,592,000 (1s block time over a month).
Copy link
Member

@tac0turtle tac0turtle Feb 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this is left to the node operator then they can get proofs is split between both ss and sc. With adr 40 they were talking about reconstructing the tree on the fly but since insertion order matters with iavl we cant make that assumption.

while being a bit more flexible with the underlying implementations and overall
less intrusive. Specifically, we propose to:

* Separate the concerns of state commitment (**SC**), needed for consensus, and
Copy link
Collaborator

@yihuang yihuang Mar 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been using a different separation of concerns to guide my design, which is based on the requirement of different roles of clients, it goes like this:

  1. consensus state machine (validator node)
    • it requires read/write on the latest version of the state
    • generating the root hash for consensus
    • export/restore snapshots for state-sync
  2. historical queries without merkle proofs (grpc query server)
    • it requires read-only access to historical states without generating merkle proofs
  3. historical queries with merkle proofs (/abci_query?proof=true)

The SS (or the versiondb) is to serve the requirement of 2, I'm experimenting with a new IAVL implementation which focuses on the requirement of 1 first, the requirement of 3 is considered as lower priority, it can be at least supported by the traditional design.
It's slightly different from simply separating SC and SS.

Copy link
Collaborator

@yihuang yihuang Mar 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One tricky part is, a separated SC will need to duplicate the keys no matter what, which is a significant duplication already, for example in our evm module, the keys are larger than values (32bytes for value and 20+32 bytes for keys).

Copy link
Member

@tac0turtle tac0turtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments based of slack discussions.

docs/architecture/adr-065-store-v2.md Show resolved Hide resolved
docs/architecture/adr-065-store-v2.md Show resolved Hide resolved
docs/architecture/adr-065-store-v2.md Outdated Show resolved Hide resolved
docs/architecture/adr-065-store-v2.md Show resolved Hide resolved
docs/architecture/adr-065-store-v2.md Outdated Show resolved Hide resolved
@yihuang
Copy link
Collaborator

yihuang commented Mar 5, 2023

One thing maybe good to include in the storage refactoring is, currently SDK stores some secondary indexes into the IAVL tree, which is only for convenience of query, I think we can have a special API for these cases, to only store them in SS, but not in SC?

@tac0turtle
Copy link
Member

One thing maybe good to include in the storage refactoring is, currently SDK stores some secondary indexes into the IAVL tree, which is only for convenience of query, I think we can have a special API for these cases, to only store them in SS, but not in SC?

agree, we dont need proofs on reverse indexes for queries. i wonder if there is an api where we can say this state should be included in a commitment and this state should not. It would help in reducing the duplicate data between ss and sc

@yihuang
Copy link
Collaborator

yihuang commented Mar 7, 2023

Will there be a separate ADR for cache store refactoring?

@tac0turtle
Copy link
Member

tac0turtle commented Mar 7, 2023

Will there be a separate ADR for cache store refactoring?

there will be a new cache store in store v2 from my understanding, it might make sense to include this here. imo it shouldnt be a refactor but a rewrite. same with other kv stores. I think this adr should mention the interfaces that will be adopted or a proposal on how they will look.

Copy link
Member

@tac0turtle tac0turtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few more comments, i think there is still more information missing in terms of trying to define some interfaces. How will the gas kv store look, how will the cache kv store look? Still some items that are fundamental to storage v2 rewrite that are missing here.

docs/architecture/adr-065-store-v2.md Show resolved Hide resolved
docs/architecture/adr-065-store-v2.md Show resolved Hide resolved
docs/architecture/adr-065-store-v2.md Show resolved Hide resolved
Copy link
Member

@tac0turtle tac0turtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how will state sync snapshots work?

@alexanderbez alexanderbez merged commit 8928ea1 into main Mar 16, 2023
@alexanderbez alexanderbez deleted the bez/adr-065-store-v2 branch March 16, 2023 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T: ADR An issue or PR relating to an architectural decision record T:Docs Changes and features related to documentation. Type: ADR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants