docs: ADR-049 state sync hooks

docs/architecture/adr-049-state-sync-hooks.md

@@ -0,0 +1,170 @@
+# ADR 049: State Sync Hooks
+
+## Changelog
+
+- Jan 19, 2022: Initial Draft
+
+## Status
+
+Draft, Under Implementation
+
+## Abstract
+
+This ADR provides hooks for app modules to publish snapshot of additional state(outside of IAVL tree) for state-sync.
+
+## Context
+
+New clients use state-sync to download snapshots of module state from peer nodes. Currently, the snapshot consists of a
+stream of `SnapshotStoreItem` and `SnapshotIAVLItem`, which means for app modules that maintain their states outside of
+the IAVL tree, they can not add their states to the snapshot stream for state-sync.
+
+## Decision
+
+A simple proposal based on our existing implementation is that, we can add two new message types: `SnapshotExtensionMeta` 
+and `SnapshotExtensionPayload`, and they are appended to the existing multi-store stream with `SnapshotExtensionMeta` 
+acting as a delimiter between extensions.
+
+```proto
+// SnapshotItem is an item contained in a rootmulti.Store snapshot.
+message SnapshotItem {
+  // item is the specific type of snapshot item.
+  oneof item {
+    SnapshotStoreItem        store             = 1;
+    SnapshotIAVLItem         iavl              = 2 [(gogoproto.customname) = "IAVL"];
+    SnapshotExtensionMeta    extension         = 3;
+    SnapshotExtensionPayload extension_payload = 4;
+  }
+}
+
+// SnapshotStoreItem contains metadata about a snapshotted store.
+message SnapshotStoreItem {
+  string name = 1;
+}
+
+// SnapshotIAVLItem is an exported IAVL node.
+message SnapshotIAVLItem {
+  bytes key     = 1;
+  bytes value   = 2;
+  int64 version = 3;
+  int32 height  = 4;
+}
+
+// SnapshotExtensionMeta contains metadata about an external snapshotter.
+message SnapshotExtensionMeta {
+  string name   = 1;
+  uint32 format = 2;
+}
+
+// SnapshotExtensionPayload contains payloads of an external snapshotter.
+message SnapshotExtensionPayload {
+  bytes payload = 1;
+}
+```
+
+The snapshot stream would look like this:
+
+```go
+// multi-store snapshot
+{SnapshotStoreItem | SnapshotIAVLItem, ...}
+// extension1 snapshot
+SnapshotExtensionMeta
+{SnapshotExtensionPayload, ...}
+// extension2 snapshot
+SnapshotExtensionMeta
+{SnapshotExtensionPayload, ...}
+```
+
+Add `extensions` field to snapshot `Manager` for extension snapshotters. Multistore snapshotter is a special one and it doesn't need a name because it is always placed at the beginning of the binary stream.
+
+```go
+type Manager struct {
+	store      *Store
+	multistore types.Snapshotter
+	extensions map[string]types.NamedSnapshotter
+    mtx                sync.Mutex
+	operation          operation
+	chRestore          chan<- io.ReadCloser
+	chRestoreDone      <-chan restoreDone
+	restoreChunkHashes [][]byte
+	restoreChunkIndex  uint32
+}
+```
+
+For extension snapshotters that have implemented the `NamedSnapshotter` interface, their names should be registered to the snapshotter `Manager` by calling 
+`RegisterExtensions` when setting up the application.
+
+```go
+// RegisterExtensions register extension snapshotters to manager
+func (m *Manager) RegisterExtensions(extensions ...types.NamedSnapshotter) error 
+```
+
+As with `Snapshotter` interface for `multistore` snapshotter, two more function signatures: `SnapshotFormat()` and `SupportedFormats()` are added.
+<ul>
+<li> removal of format parameter: `Snapshotter` chooses a format autonomously, not pass in from the caller. </li>
+<li> `Snapshotter` announces the format it uses to snapshot with method `SnapshotFormat()`. </li>
+<ul>
+
+```go
+type Snapshotter interface {
+	// Snapshot writes snapshot items into the protobuf writer.
+	Snapshot(height uint64, protoWriter protoio.WriteCloser) error
+
+	// Restore restores a state snapshot from the protobuf items read from the reader.
+	// If the ready channel is non-nil, it returns a ready signal (by being closed) once the
+	// restorer is ready to accept chunks.
+	Restore(height uint64, format uint32, protoReader protoio.ReadCloser) (SnapshotItem, error)
+
+	// SnapshotFormat returns the default format used to take a snapshot.
+	SnapshotFormat() uint32
+
+	// SupportedFormats returns a list of formats it can restore from.
+	SupportedFormats() []uint32
+}
+```
+
+Add `NamedSnapshotter` interface for extension snapshotters.
+
+```go
+type NamedSnapshotter interface {
+	Snapshotter
+
+	// SnapshotterName returns the name of snapshotter, it should be unique in the manager.
+	SnapshotterName() string
+}
+```
+
+## Consequences
+
+As a result of this implementation, we are able to create snapshots of binary chunk stream for the state that we maintain outside of the IAVL Tree, CosmWasm blobs for example. And new clients are able to fetch sanpshots of state for all modules that have implemented the corresponding interface from peer nodes. 
+
+
+### Backwards Compatibility
+
+This ADR introduces new proto message types, add field for extension snapshotters to snapshot `Manager`, add new function signatures `SnapshotFormat()`, `SupportedFormats()` to `Snapshotter` interface and add new `NamedSnapshotter` interface, so this is not backwards compatible.
+
+### Positive
+
+State maintained outside of IAVL tree like CosmWasm blobs can create snapshots by implementing extension snapshotters, and being fetched by new clients via state-sync.
+
+### Negative
+
+// Todo
+
+### Neutral
+
+All modules that maintain state outside of IAVL tree need to implement `NamedSnapshotter` and the snapshot manager need to call `RegisterExtensions` when setting up the application.
+
+## Further Discussions
+
+While an ADR is in the DRAFT or PROPOSED stage, this section should contain a summary of issues to be solved in future iterations (usually referencing comments from a pull-request discussion).
+Later, this section can optionally list ideas or improvements the author or reviewers found during the analysis of this ADR.
+
+## Test Cases [optional]
+
+Test cases for an implementation are mandatory for ADRs that are affecting consensus changes. Other ADRs can choose to include links to test cases if applicable.
+
+## References
+
+- https://github.com/cosmos/cosmos-sdk/pull/10961
+- https://github.com/cosmos/cosmos-sdk/issues/7340
+- https://hackmd.io/gJoyev6DSmqqkO667WQlGw