ADR 0018: Inter-Runtime Communication Protocol (IRCP)

0018-ircp.md

@@ -0,0 +1,234 @@
+# ADR 0018: Inter-Runtime Communication Protocol (IRCP)
+
+## Component
+
+Oasis Core, Oasis SDK
+
+## Changelog
+
+- 2022-10-13: Initial draft
+
+## Status
+
+Proposed
+
+## Context
+
+Currently each runtime is its own relatively isolated environment. It can
+interact with the consensus layer via [incoming] and [outgoing] messages, but
+a standardized way to communicate with other parallel runtimes sharing the same
+consensus layer is missing.
+
+This ADR proposes the Inter-Runtime Communication Protocol (IRCP) which enables
+any two runtimes sharing the same consensus layer to securely communicate by
+exchanging standardized packets. It also proposes modifications to the runtime
+P2P network to support seamless relaying of messages without the need for any
+external processes.
+
+[incoming]: 0011-incoming-runtime-messages.md
+[outgoing]: 0003-consensus-runtime-token-transfer.md
+
+## Decision
+
+The following new concepts are introduced by this proposal:
+
+* Each runtime can signal that a **connection** should be established _to_
+  another runtime sharing the same consensus layer. For each unique destination
+  runtime only one connection may be established.
+
+* In the context of an established connection the runtime may establish one or
+  more **channels**. Each channel can be used to send and receive ordered
+  packets.
+
+The transport of packets does not require any new interactions with the
+consensus layer in order to make sure the protocol is scalable. Runtimes only
+rely on the common consensus layer for packet authentication by using it as a
+source of trusted state roots for all runtimes.
+
+### Channels
+
+TODO: Channel metadata structure.
+
+### Packets
+
+An IRCP packet has the following structure.
+
+```golang
+// Packet is an IRCP packet used for transporting data between runtimes.
+type Packet struct {
+	cbor.Versioned
+
+	// SeqNo is the packet sequence number.
+	SeqNo uint64 `json:"seqno"`
+	// AckNo is the acknowledgement number.
+	AckNo uint64 `json:"ackno,omitempty"`
+	// Src is the source runtime ID.
+	Src common.Namespace `json:"src"`
+	// SrcChannel is the source channel ID.
+	SrcChannel uint64 `json:"src_channel"`
+	// Dst is the destination runtime ID.
+	Dst common.Namespace `json:"dst"`
+	// DstChannel is the destination channel ID.
+	DstChannel uint64 `json:"dst_channel"`
+	// Height is the consensus layer height of when the packet was sent.
+	Height uint64 `json:"height"`
+
+	// Data is the runtime-dependent data that should be sent over the channel.
+	Data []byte `json:"data"`
+}
+```
+
+### Storage Keys
+
+IRCP requires the runtimes to use pre-defined keys for storage in order to be
+able to generate proofs of packets being sent.
+
+The following storage keys must be used:
+
+* **Connections.**
+	```
+	"ircp" 0x01 dst-runtime-id
+	```
+
+	Where `dst-runtime-id` is the serialized destination runtime ID.
+
+	The value is empty.
+
+* **Channels.**
+  ```
+  "ircp" 0x02 dst-runtime-id src-channel-id
+  ```
+
+  Where `dst-runtime-id` is the serialized destination runtime ID and
+  `src-channel-id` is the big-endian uint64 encoding of the channel ID.
+
+  The value is the serialized `Channel`.
+
+* **Outgoing packet queue.**
+  ```
+  "ircp" 0x03 packet-seqno
+  ```
+
+  Where `packet-seqno` is the big-endian uint64 encoding of the packet sequence
+  number.
+
+  The value is the serialized `Packet`.
+
+### Authentication
+
+Packets need to be authenticated before they can be processed. Authentication is
+based on proving that certain storage keys (as specified above) have been set in
+the source runtime.
+
+The runtimes should leverage the existing consensus layer light client interface
+as an oracle for getting trusted consensus block headers and from those block
+headers the trusted state roots for particular runtimes.
+
+Proofs are described by the following structures which facilitate batching of
+multiple packets under one proof as long as there are no gaps in sequence
+numbers.
+
+```golang
+// Proof is an IRCP proof.
+type Proof struct {
+	cbor.Versioned
+
+	// Height is the consensus layer height of the proof.
+	//
+	// Note that the height of the proof can be higher than or equal to the height of when the
+	// packet was sent since proofs can be refreshed.
+	Height uint64 `json:"height"`
+	// Entries are the entries of a proof.
+	Entries [][]byte `json:"entries"`
+}
+
+// VerifiableBatch is a batch of verifiable IRCP packets used for transporting data between
+// runtimes. Multiple packets may be combined and covered by a single proof.
+type VerifiableBatch struct {
+	// Packets are ordered IRCP packets. There may be no gaps in sequence numbers.
+	Packets []Packet `json:"packets"`
+	// Proof is the proof for the given packets.
+	Proof Proof `json:"proof"`
+}
+```
+
+### Runtime Queries
+
+Each runtime supporting IRCP needs to provide the following query methods for
+any given round.
+
+* `ircp.Connections() -> Vec<Connection>` returns the set of connections that
+	the runtime should establish with other runtimes.
+
+* `ircp.OutgoingPackets() -> Vec<Packet>` returns the set of packets that need
+	to be delivered to destination runtimes.
+
+Runtime support for IRCP is indicated by having a module named `ircp` reported
+in the `core.RuntimeInfo` query response. In case the module is not present,
+IRCP support for that runtime should be disabled by the host.
+
+### Fee Payments
+
+Instead of relayers paying for the transport of messages, the channel should do
+so. Each runtime implementing IRCP must make it possible for channels to have
+a balance used for paying fees of processing incoming messages. Exact payment
+semantics are up to the runtime.
+
+### Transport
+
+In order to avoid the need for additional external processes that would perform
+relaying of packets between networks, existing Oasis Core compute committee
+nodes are in charge of performing IRCP transport functions.
+
+A new P2P pubsub topic kind named `ircp` is added (in addition to existing
+`committee` and `tx` topics). Each node hosting a runtime will join the topic
+for:
+
+* the locally hosted runtime (for incoming and outgoing packets) and;
+* any _destination_ runtimes returned by the `ircp.Connections` query (for
+  outgoing packets only).
+
+The `ircp` topic accepts serialized `VerifiableBatch` messages, each
+representing a batch of packets that should be delivered to the runtime the
+topic is for.
+
+#### Outgoing Packet Processing
+
+TODO
+
+#### Transaction Pool
+
+For locally hosted runtimes the `VerifiableBatch` messages are delivered to the
+node's runtime transaction pool and are checked and placed into a special IRCP
+queue.
+
+A new IRCP queue is added to the transaction pool. The queue is aware of IRCP
+semantics and performs verification (via the usual runtime transaction checks)
+and ordering of packets.
+
+### SDK Module
+
+A new Oasis SDK module named `ircp` would be introduced to handle the runtime
+aspects of IRCP.
+
+#### Connections
+
+#### Channels
+
+#### Flows
+
+## Consequences
+
+> This section describes the consequences, after applying the decision. All
+> consequences should be summarized here, not just the "positive" ones.
+
+### Positive
+
+### Negative
+
+### Neutral
+
+## References
+
+> Are there any relevant PR comments, issues that led up to this, or articles
+> referenced for why we made the given design choice? If so, link them here.

README.md

@@ -112,4 +112,6 @@ The following records currently exist:
 * [ADR 0014](0014-runtime-signing-tx-with-hardware-wallet.md) - Signing Runtime Transactions with Hardware Wallet
 * [ADR 0015](0015-vrf-per-block-entropy.md) - Randomized Paratime Proposer Selection
 * [ADR 0016](0016-consensus-parameters-change-proposal.md) - Consensus Parameters Change Proposal
+* [ADR 0017](0017-app-standards.md) - ParaTime Application Standard Proposal Process
+* [ADR 0018](0018-ircp.md) - Inter-Runtime Communication Protocol
 <!-- markdownlint-enable line-length -->