Store Decoding

[aaronc]

This discussion replaces #7097. In ADR 038 a mechanism for observing state changes as well as support for streaming that to disk was added. The intention with these state listeners is to eventually index state in a conventional data store with additional indexes for high performance. The current database being targeted is PostgreSQL.

In order to index state data in a database we need some way to decode key-value pairs. In #7097, static schema descriptors (like a YAML file to describe the schema) were proposed as one approach to this. However, some of the existing store key layouts are rather complex and a developing a suitable static descriptor might be pretty hard.

I propose an alternate mechanism here which basically involves specifying an interface which modules could implement to decode key-value pairs into more conventional relational DB "table" updates. In this model, we define a "table" as something consisting of a protobuf message type to define the table columns, a name, and a primary key composed of one or more fields in the protobuf message.

Here's a rough sketch of the propose interface:

type TableSchema interface {
  Tables() []TableInfo
}

type TableInfo interface {
  Name() string

  // Type is the protobuf type of this table
  Type() proto.Message

  // PrimaryKeyFields are the fields in the protobuf type which compose the primary key
  PrimaryKeyFields []string
}

type TableDecoder interface {
  Schema() TableSchema
  
  // Decode decodes a key-value pair into a TableUpdate. If value is set to nil this indicates
  // that the key-value pair was deleted from storage.
  Decode(key, value []byte) (TableUpdate, error)
}

type TableUpdate struct {
  Table string

  // true for update mode indicating that this is a non-overwriting patch update,
  // false for replace mode indicating that this update replaces a whole row
  UpdateOrReplace bool

  // in update mode Updated contains only changed fields and ClearedFields
  // indicates fields to be cleared, in replace mode the value of Updated replaces the whole table row
  Updated proto.Message
  
  // ClearedFields contains a list of fields to clear and should only be set in replace mode
  ClearedFields []string
}

Additional metadata to support a relational data mapping could be added to TableInfo as needed.

To make it easy for new modules to support table decoding, the table-store/ORM package proposed in #9156 would support the TableDecoder interface out of the box.

/cc @blushi @i-norden @robert-zaremba