From b0522ba84560d7cf359377560ef0c8059eb238ea Mon Sep 17 00:00:00 2001 From: Ryo Onodera Date: Wed, 3 Jun 2020 20:51:56 +0900 Subject: [PATCH] Introduce automatic ABI maintenance mechanism (1/2; prepare) (#10335) * Introduce automatic ABI maintenance mechanism * Compile fix... * Docs fix... * Programs compilation fix... * Simplify source credit Co-authored-by: Michael Vines * Cargo.lock... Co-authored-by: Michael Vines --- src/SUMMARY.md | 2 +- .../abi-management.md | 77 ++++++++++++++++++- 2 files changed, 74 insertions(+), 5 deletions(-) rename src/{proposals => implemented-proposals}/abi-management.md (50%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index b54983ba8cf1a9..05aa7a3f8745e1 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -101,6 +101,7 @@ * [Snapshot Verification](implemented-proposals/snapshot-verification.md) * [Cross-Program Invocation](implemented-proposals/cross-program-invocation.md) * [Program Derived Addresses](implemented-proposals/program-derived-addresses.md) + * [ABI Management](implemented-proposals/abi-management.md) * [Accepted Design Proposals](proposals/README.md) * [Ledger Replication](proposals/ledger-replication-to-implement.md) * [Optimistic Confirmation and Slashing](proposals/optimistic-confirmation-and-slashing.md) @@ -114,6 +115,5 @@ * [Slashing](proposals/slashing.md) * [Tick Verification](proposals/tick-verification.md) * [Block Confirmation](proposals/block-confirmation.md) - * [ABI Management](proposals/abi-management.md) * [Rust Clients](proposals/rust-clients.md) * [Optimistic Confirmation](proposals/optimistic_confirmation.md) diff --git a/src/proposals/abi-management.md b/src/implemented-proposals/abi-management.md similarity index 50% rename from src/proposals/abi-management.md rename to src/implemented-proposals/abi-management.md index 61f04934b1949f..6cbd1fd1b9cd71 100644 --- a/src/proposals/abi-management.md +++ b/src/implemented-proposals/abi-management.md @@ -55,7 +55,7 @@ fields. # Example ```patch -+#[frozen_abi(digest="1c6a53e9")] ++#[frozen_abi(digest="eXSMM7b89VY72V...")] #[derive(Serialize, Default, Deserialize, Debug, PartialEq, Eq, Clone)] pub struct Vote { /// A stack of votes starting with the oldest vote @@ -73,16 +73,85 @@ digest from the assertion test error message. In general, once we add `frozen_abi` and its change is published in the stable release channel, its digest should never change. If such a change is needed, we -should opt for defining a new struct like `FooV1`. And special release flow like -hard forks should be approached. +should opt for defining a new `struct` like `FooV1`. And special release flow +like hard forks should be approached. # Implementation remarks We use some degree of macro machinery to automatically generate unit tests and calculate a digest from ABI items. This is doable by clever use of -`serde::Serialize` (`[1]`) and `any::typename` (`[2]`). For a precedent for similar +`serde::Serialize` (`[1]`) and `any::type_name` (`[2]`). For a precedent for similar implementation, `ink` from the Parity Technologies `[3]` could be informational. +# Implementation details + +The implementation's goal is to detect unintended ABI changes automatically as +much as possible. To that end, the digest of structural ABI information is +calculated with best-effort accuracy and stability. + +When the ABI digest check is run, it dynamically computes an ABI digest by +recursively digesting the ABI of fields of the ABI item, by re-using the +`serde`'s serialization functionality, proc macro and generic specialization. +And then, the check `assert!`s that its finalized digest value is identical as +what is specified in the `frozen_abi` attribute. + +To realize that, it creates an example instance of the type and a custom +`Serializer` instance for `serde` to recursively traverse its fields as if +serializing the example for real. This traversing must be done via `serde` to +really capture what kinds of data actually would be serialized by `serde`, even +considering custom non-`derive`d `Serialize` trait implementations. + +# The ABI digesting process + +This part is a bit complex. There is three inter-depending parts: `AbiExample`, +`AbiDigester` and `AbiEnumVisitor`. + +First, the generated test creates an example instance of the digested type with +a trait called `AbiExample`, which should be implemented for all of digested +types like the `Serialize` and return `Self` like the `Default` trait. Usually, +it's provided via generic trait specialization for most of common types. Also +it is possible to `derive` for `struct` and `enum` and can be hand-written if +needed. + +The custom `Serializer` is called `AbiDigester`. And when it's called by `serde` +to serialize some data, it recursively collects ABI information as much as +possible. `AbiDigester`'s internal state for the ABI digest is updated +differentially depending on the type of data. This logic is specifically +redirected via with a trait called `AbiEnumVisitor` for each `enum` type. As the +name suggests, there is no need to implement `AbiEnumVisitor` for other types. + +To summarize this interplay, `serde` handles the recursive serialization control +flow in tandem with `AbiDigester`. The initial entry point in tests and child +`AbiDigester`s use `AbiExample` recursively to create an example object +hierarchal graph. And `AbiDigester` uses `AbiEnumVisitor` to inquiry the actual +ABI information using the constructed sample. + +`Default` isn't enough for `AbiExample`. Various collection's `::default()` is +empty, yet, we want to digest them with actual items. And, ABI digesting can't +be realized only with `AbiEnumVisitor`. `AbiExample` is required because an +actual instance of type is needed to actually traverse the data via `serde`. + +On the other hand, ABI digesting can't be done only with `AbiExample`, either. +`AbiEnumVisitor` is required because all variants of an `enum` cannot be +traversed just with a single variant of it as a ABI example. + +Digestable information: + +- rust's type name +- `serde`'s data type name +- all fields in `struct` +- all variants in `enum` +- `struct`: normal(`struct {...}`) and tuple-style (`struct(...)`) +- `enum`: normal variants and `struct`- and `tuple`- styles. +- attributes: `serde(serialize_with=...)` and `serde(skip)` + +Not digestable information: + +- Any custom serialize code path not touched by the sample provided by + `AbiExample`. (technically not possible) +- generics (must be a concrete type; use `frozen_abi` on concrete type + aliases) + # References 1. [(De)Serialization with type info · Issue #1095 · serde-rs/serde](https://github.com/serde-rs/serde/issues/1095#issuecomment-345483479)