Skip to content

Commit

Permalink
Schemas Pallet Documentation (#1968)
Browse files Browse the repository at this point in the history 
# Goal
The goal of this PR is to improve the documentation of the Pallets and
make that documentation be able to be used on docs.frequency.xyz.

Part of frequency-chain/docs#59

# Discussion
- Schemas Pallet

---------

Co-authored-by: Shannon Wells <[email protected]>
  • Loading branch information
@wilwade @shannonwells
wilwade and shannonwells authored May 15, 2024
1 parent 5e4d7a7 commit f91a6e3
Show file tree
Hide file tree
Showing 3 changed files with 105 additions and 32 deletions.
4 changes: 2 additions & 2 deletions common/primitives/src/schema.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,10 @@ pub enum PayloadLocation {
#[derive(Copy, Clone, RuntimeDebug, PartialEq, Eq, Encode, Decode, MaxEncodedLen, TypeInfo)]
pub enum SchemaSetting {
/// Schema setting to enforce append-only behavior on payload.
/// Applied to schemas of type `PayloadLocation::Itemized` or `PayloadLocation::Paginated`.
/// Applied to schemas of type `PayloadLocation::Itemized`.
AppendOnly,
/// Schema may enforce signature requirement on payload.
/// Applied to schemas of type `PayloadLocation::Paginated`.
/// Applied to schemas of type `PayloadLocation::Itemized` or `PayloadLocation::Paginated`.
SignatureRequired,
}

Expand Down
100 changes: 100 additions & 0 deletions pallets/schemas/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Schemas Pallet

The Schemas Pallet provides universal schema registration for data flowing through Frequency.

## Summary

This pallet provides an on chain repository for schemas, thereby allowing participants of the network to flexibly interact and exchange messages with each other with the correct human intent and data structure.
All Messages and Stateful Storage content must be attached to a Schema Identifier so that the content can be correctly parsed and interpreted.

### Data Structure vs Human Intent

Schemas provide for both consistent data structures, but also human intent of the message.
Some schemas may be structurally the same, but have a different interpretation of the contents.
For example, two schemas might both only have a hash for contents, but one is a recording of the hash for time validation purposes, while the other is to mark an off-chain vote.

### Schema Parameters

- Model: The actual JSON representing the data structure.
- Model Type: The type of serialization used. (Parquet, Avro, etc...)
- Settings: Various options for the Schema like signature requirements.
- Payload Location: The location the data for this Schema is stored.

#### Model Types

- [`Parquet`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.Parquet): Designed for lists and when a Provider is collecting items from many different MSAs and publishing them together.
- [`AvroBinary`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.ModelType.html#variant.AvroBinary): Useful for most generic data structures.

#### Settings

- [`AppendOnly`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.SchemaSetting.html#variant.AppendOnly)
- Prior data is immutable and all new data is appended to existing data.
- For Payload Locations: `Itemized`
- [`SignatureRequired`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.SchemaSetting.html#variant.SignatureRequired)
- An MSA control key signature is required instead of a delegation.
- For Payload Locations: `Itemized` or `Paginated`

#### Payload Locations

- [`OnChain`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.OnChain): Data is stored directly in the Messages pallet data storage, usually as `AvroBinary`.
- [`IPFS`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.IPFS): Data is stored in IPFS and Messages pallet stores the CID.
- [`Itemized`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Itemized): Data is stored in the Stateful Storage pallet as an array of individual items.
- [`Paginated`](https://frequency-chain.github.io/frequency/common_primitives/schema/enum.PayloadLocation.html#variant.Paginated): Data is stored in the Stateful Storage pallet as a list of paged blobs.

### Mainnet vs Testnet Schema Creation

Mainnet schemas must be approved by the Frequency Council.
This is to prevent malicious schemas and increase the documentation around the schemas available.

On Testnets, schemas can be created by anyone, so there are _no_ guarantees around schema correctness or quality.
If you want to use a particular schema on a testnet, it is suggested that you use specific Schema Ids or publish the needed schema yourself.

### Actions

The Schemas pallet provides for:

- Registering or proposing new Schemas.
- Retrieving schemas by their Id or name.
- Validating a Schema model.
- Retrieving last registered Schema Id.

## Interactions

### Extrinsics

| Name/Description | Caller | Payment | Key Events | Runtime Added |
| ----------------------------------------------------------------------------------------------------- | ----------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `set_max_schema_model_bytes`<br />Governance action to alter the maximum byte length of Schema models | Governance | Tokens | [`SchemaMaxSizeChanged`](https://frequency-chain.github.io/frequency/pallet_schemas/pallet/enum.Event.html#variant.SchemaMaxSizeChanged) | 1 |
| `propose_to_create_schema_v2`<br />Creates a proposal to the Frequency Council for a new schema | Token Account | Tokens | [`Proposed`](https://paritytech.github.io/polkadot-sdk/master/pallet_collective/pallet/enum.Event.html#variant.Proposed) | 66 |
| `create_schema_via_governance_v2`<br />Governance action version of `create_schema_v3` | Frequency Council | Tokens | [`SchemaCreated`](https://frequency-chain.github.io/frequency/pallet_schemas/pallet/enum.Event.html#variant.SchemaCreated) | 66 |
| `create_schema_v3`<br />Creates a new Schema. | Mainnet: Governance<br />Testnet: Token Account | Tokens | [`SchemaCreated`](https://frequency-chain.github.io/frequency/pallet_schemas/pallet/enum.Event.html#variant.SchemaCreated) | 1 |
| `propose_to_create_schema_name`<br />Creates a Council proposal to set the name of a Schema | Token Account | Tokens | [`Proposed`](https://paritytech.github.io/polkadot-sdk/master/pallet_collective/pallet/enum.Event.html#variant.Proposed) | 1 |
| `create_schema_name_via_governance`<br />Governance action to set the name of a Schema | Frequency Council | Tokens | [`SchemaNameCreated`](https://frequency-chain.github.io/frequency/pallet_schemas/pallet/enum.Event.html#variant.SchemaNameCreated) | 66 |

See [Rust Docs](https://frequency-chain.github.io/frequency/pallet_schemas/pallet/struct.Pallet.html) for more details.

### State Queries

| Name | Description | Query | Runtime Added |
| --------------------------------- | ------------------------------------------------------------------- | -------------------------------- | ------------- |
| Get Max Current Schema Identifier | Fetch current Schema Identifier maximum | `currentSchemaIdentifierMaximum` | 1 |
| Get Schema Model Max Bytes | Fetch maximum number of bytes per Schema Model as set by Governance | `governanceSchemaModelMaxBytes` | 1 |
| Get a Schema Info | Fetch the metadata and settings for a schema | `schemaInfos` | 62 |
| Get Schema Ids by Name | Fetch matching Schemas Ids by namespace and name | `schemaNameToIds` | 62 |
| Get Schema Payload/Model | Fetch the payload/model JSON for the specified Schema | `schemaPayloads` | 62 |

See the [Rust Docs](https://frequency-chain.github.io/frequency/pallet_schemas/pallet/storage_types/index.html) for additional state queries and details.

### RPCs

Note: May be restricted based on node settings and configuration.

| Name | Description | Call | Node Version |
| --------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| Get Schema by Id | Retrieves the schema for the given Schema Id | [`getBySchemaId`](https://frequency-chain.github.io/frequency/pallet_schemas_rpc/trait.SchemasApiServer.html#tymethod.get_by_schema_id) | v1.0.0+ |
| Check Schema Validity | Validates a schema model and returns “true” if the model is correct | [`checkSchemaValidity`](https://frequency-chain.github.io/frequency/pallet_schemas_rpc/trait.SchemasApiServer.html#tymethod.check_schema_validity) | v1.0.0+ |
| Get Schema Versions | Returns an array of schema versions | [`getVersions`](https://frequency-chain.github.io/frequency/pallet_schemas_rpc/trait.SchemasApiServer.html#tymethod.get_versions) | v1.10.0+ |

\* Must be enabled with off-chain indexing

See [Rust Docs](https://frequency-chain.github.io/frequency/pallet_schemas_rpc/trait.SchemasApiServer.html) for more details.
33 changes: 3 additions & 30 deletions pallets/schemas/src/lib.rs
View file
Original file line number Diff line number Diff line change
@@ -1,45 +1,18 @@
//! # Schemas Pallet
//! The Schemas pallet provides functionality for handling schemas.
//! Schema Management for Message and Stateful Storage
//!
//! ## Quick Links
//! - [Configuration: `Config`](Config)
//! - [Extrinsics: `Call`](Call)
//! - [Runtime API: `SchemasRuntimeApi`](../pallet_schemas_runtime_api/trait.SchemasRuntimeApi.html)
//! - [Custom RPC API: `SchemasApiServer`](../pallet_schemas_rpc/trait.SchemasApiServer.html)
//! - [Event Enum: `Event`](Event)
//! - [Error Enum: `Error`](Error)
//!
//! ## Overview
//!
//! This pallet provides an on chain repository for schemas, thereby allowing participants of the
//! network to flexibly interact and exchange messages with each other without facing the challenge
//! of sharing, managing and validating messages as well as schemas between them.
//!
//! <b>NOTE</b>: In this pallet we define the <b>payload</b> structure that is used in <a href="../pallet_messages/index.html">Messages Pallet</a>.
//!
//! The Schema pallet provides functions for:
//!
//! - Registering a new schema.
//! - Setting maximum schema model size by governance.
//! - Retrieving latest registered schema id.
//! - Retrieving schemas by their id.
//!
//!
//! ## Terminology
//!
//! - **Schema:** The structure that defines how a Message is stored and structured.
//! - **Schema Model:** Serialization/Deserialization details of the schema
//! - **Schema Model Type:** The type of the following Serialization/Deserialization. It can be
//! Avro, Parquet or ...
#![doc = include_str!("../README.md")]
//!
//! ## Implementations
//!
//! - [`SchemaValidator`](../common_primitives/schema/trait.SchemaValidator.html): Functions for accessing and validating Schemas. This implementation is what is used in the runtime.
//! - [`SchemaProvider`](../common_primitives/schema/trait.SchemaProvider.html): Allows another pallet to resolve schema information.
//!
//! ## Genesis config
//!
//! The Schemas pallet depends on the [`GenesisConfig`].
//!
// Substrate macros are tripping the clippy::expect_used lint.
#![allow(clippy::expect_used)]
#![cfg_attr(not(feature = "std"), no_std)]
Expand Down

0 comments on commit f91a6e3

Please sign in to comment.