Skip to content

Commit

Permalink
Introduce automatic ABI maintenance mechanism (1/2; prepare) (solana-…
Browse files Browse the repository at this point in the history
…labs#10335)

* Introduce automatic ABI maintenance mechanism

* Compile fix...

* Docs fix...

* Programs compilation fix...

* Simplify source credit

Co-authored-by: Michael Vines <[email protected]>

* Cargo.lock...

Co-authored-by: Michael Vines <[email protected]>
  • Loading branch information
ryoqun and mvines authored Jun 3, 2020
1 parent 03bedb2 commit b0522ba
Show file tree
Hide file tree
Showing 2 changed files with 74 additions and 5 deletions.
2 changes: 1 addition & 1 deletion src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@
* [Snapshot Verification](implemented-proposals/snapshot-verification.md)
* [Cross-Program Invocation](implemented-proposals/cross-program-invocation.md)
* [Program Derived Addresses](implemented-proposals/program-derived-addresses.md)
* [ABI Management](implemented-proposals/abi-management.md)
* [Accepted Design Proposals](proposals/README.md)
* [Ledger Replication](proposals/ledger-replication-to-implement.md)
* [Optimistic Confirmation and Slashing](proposals/optimistic-confirmation-and-slashing.md)
Expand All @@ -114,6 +115,5 @@
* [Slashing](proposals/slashing.md)
* [Tick Verification](proposals/tick-verification.md)
* [Block Confirmation](proposals/block-confirmation.md)
* [ABI Management](proposals/abi-management.md)
* [Rust Clients](proposals/rust-clients.md)
* [Optimistic Confirmation](proposals/optimistic_confirmation.md)
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ fields.
# Example

```patch
+#[frozen_abi(digest="1c6a53e9")]
+#[frozen_abi(digest="eXSMM7b89VY72V...")]
#[derive(Serialize, Default, Deserialize, Debug, PartialEq, Eq, Clone)]
pub struct Vote {
/// A stack of votes starting with the oldest vote
Expand All @@ -73,16 +73,85 @@ digest from the assertion test error message.

In general, once we add `frozen_abi` and its change is published in the stable
release channel, its digest should never change. If such a change is needed, we
should opt for defining a new struct like `FooV1`. And special release flow like
hard forks should be approached.
should opt for defining a new `struct` like `FooV1`. And special release flow
like hard forks should be approached.

# Implementation remarks

We use some degree of macro machinery to automatically generate unit tests
and calculate a digest from ABI items. This is doable by clever use of
`serde::Serialize` (`[1]`) and `any::typename` (`[2]`). For a precedent for similar
`serde::Serialize` (`[1]`) and `any::type_name` (`[2]`). For a precedent for similar
implementation, `ink` from the Parity Technologies `[3]` could be informational.

# Implementation details

The implementation's goal is to detect unintended ABI changes automatically as
much as possible. To that end, the digest of structural ABI information is
calculated with best-effort accuracy and stability.

When the ABI digest check is run, it dynamically computes an ABI digest by
recursively digesting the ABI of fields of the ABI item, by re-using the
`serde`'s serialization functionality, proc macro and generic specialization.
And then, the check `assert!`s that its finalized digest value is identical as
what is specified in the `frozen_abi` attribute.

To realize that, it creates an example instance of the type and a custom
`Serializer` instance for `serde` to recursively traverse its fields as if
serializing the example for real. This traversing must be done via `serde` to
really capture what kinds of data actually would be serialized by `serde`, even
considering custom non-`derive`d `Serialize` trait implementations.

# The ABI digesting process

This part is a bit complex. There is three inter-depending parts: `AbiExample`,
`AbiDigester` and `AbiEnumVisitor`.

First, the generated test creates an example instance of the digested type with
a trait called `AbiExample`, which should be implemented for all of digested
types like the `Serialize` and return `Self` like the `Default` trait. Usually,
it's provided via generic trait specialization for most of common types. Also
it is possible to `derive` for `struct` and `enum` and can be hand-written if
needed.

The custom `Serializer` is called `AbiDigester`. And when it's called by `serde`
to serialize some data, it recursively collects ABI information as much as
possible. `AbiDigester`'s internal state for the ABI digest is updated
differentially depending on the type of data. This logic is specifically
redirected via with a trait called `AbiEnumVisitor` for each `enum` type. As the
name suggests, there is no need to implement `AbiEnumVisitor` for other types.

To summarize this interplay, `serde` handles the recursive serialization control
flow in tandem with `AbiDigester`. The initial entry point in tests and child
`AbiDigester`s use `AbiExample` recursively to create an example object
hierarchal graph. And `AbiDigester` uses `AbiEnumVisitor` to inquiry the actual
ABI information using the constructed sample.

`Default` isn't enough for `AbiExample`. Various collection's `::default()` is
empty, yet, we want to digest them with actual items. And, ABI digesting can't
be realized only with `AbiEnumVisitor`. `AbiExample` is required because an
actual instance of type is needed to actually traverse the data via `serde`.

On the other hand, ABI digesting can't be done only with `AbiExample`, either.
`AbiEnumVisitor` is required because all variants of an `enum` cannot be
traversed just with a single variant of it as a ABI example.

Digestable information:

- rust's type name
- `serde`'s data type name
- all fields in `struct`
- all variants in `enum`
- `struct`: normal(`struct {...}`) and tuple-style (`struct(...)`)
- `enum`: normal variants and `struct`- and `tuple`- styles.
- attributes: `serde(serialize_with=...)` and `serde(skip)`

Not digestable information:

- Any custom serialize code path not touched by the sample provided by
`AbiExample`. (technically not possible)
- generics (must be a concrete type; use `frozen_abi` on concrete type
aliases)

# References

1. [(De)Serialization with type info · Issue #1095 · serde-rs/serde](https://github.com/serde-rs/serde/issues/1095#issuecomment-345483479)
Expand Down

0 comments on commit b0522ba

Please sign in to comment.