Introduce automatic ABI maintenance mechanism (2/2; rollout) #8012

ryoqun · 2020-01-29T09:35:31Z

Problem

ABI can change without noticing.

Summary of Changes

This pr is currently only containing the actual rollout code to enable frozen abi. most are quite simple. But some are advanced. :) I'll hope this pr will become an example repository pr for future abi maintenance by our team mates. :)

This landed on #10335

This PR currently completely changed since this original implementation:

Introduce macro-based lexical ABI digest calculation and check mechanism deriving from the rust type system.

As noted by Propose Solana ABI management #7524, the detection mechanism isn't perfect; both false positives and negatives are possible. For example, pure type rename is a false positive. The ABI-affecting struct field's change is a false negative. Both are due to being lexical comparison nature of this implementation.

Now, this is the correct:

This PR approached the problem with serde's type system via custom Serializer for runtime inspection for more precise structural comparison of types.

todo

~~- [ ] backport to v0.23~~
~~- [ ] Once merged, add more frozen_abi to other top most types for serialization.~~

Was split to #10335

Fixes #7738

programs/vote/src/vote_instruction.rs

sdk/macro/src/lib.rs

codecov · 2020-01-29T10:27:12Z

Codecov Report

Merging #8012 into master will increase coverage by 0.1%.
The diff coverage is 97.5%.

@@           Coverage Diff            @@
##           master   #8012     +/-   ##
========================================
+ Coverage    81.9%   82.1%   +0.1%     
========================================
  Files         310     316      +6     
  Lines       71844   71960    +116     
========================================
+ Hits        58903   59095    +192     
+ Misses      12941   12865     -76

sdk/src/lib.rs

mvines

direction looks great so far

runtime/src/hard_forks.rs

ci/nits.sh

runtime/src/hard_forks.rs

sdk/Cargo.toml

sdk/build.rs

sdk/macro-frozen-abi-dummy/Cargo.toml

sdk/macro-frozen-abi-dummy/src/lib.rs

sdk/macro-frozen-abi/Cargo.toml

sdk/src/abi_digester.rs

sdk/src/lib.rs

sdk/macro-frozen-abi/src/lib.rs

sdk/src/abi_digester.rs

runtime/Cargo.toml

sdk/src/lib.rs

ryoqun · 2020-02-04T16:32:30Z

@mvines Status update!

Almost feature complete...
This is what frozen_abi can capture about the ABI for the Bank struct. Pretty impressive, imho:
https://gist.github.com/ryoqun/d0dea9e40b7898cbbe71eecef263c3c5 I think it's even ready for its own crate. ;)

Still, the main files are WAAAYY ugly. Don't even look at it.
Also, this PR alone got a bit too large. So, as a precaution, I'm keeping a patch set not a single commit for possible split into smaller PRs. On the other hand, I'm also thinking that this PR is ok as it is because this is technically test-code-only PR and its volume is dominated by the 2 completely new added files. Is there any preferred way for your easiest review life?

Let's start with easy documentations.

Currently I'm intentionally making CI red to show the ABI test is actually failing at the coverage job only, other jobs are completely not affected.

[note to be source-commented]:

The cfg matrix is rather complicated. There are now 3 independent conditions due to separate and unavoidable reasons...:

rustc version: stable vs nightly (the very impl of frozen_abi requires feature(specialization))
target: cfg(test) vs cfg(not(test)) (really constrain any frozen_abi impl under test to avert risks)
feature: feature = "program" vs not(feature = "program") (frozen_abi isn't needed by smart contracts)

And all of combinations (= 8) should be working now.

I've constructed rather complicated plumbing for my teammates to avoid cfg_attr(................, fronzen_abi(digest="...")) and simply write frozen_abi(digest = "...") and derive(AbiDigestSample).

Is there any thoughts in the cfg & usage aspects?

ryoqun · 2020-02-07T05:14:19Z

TODO: include bincode's serialized_size into the ABI digesting. Also might be useful for lightweight glance of memory foot prints like #8156

SKIP

stale · 2020-06-25T23:02:22Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale · 2020-07-02T23:19:14Z

This stale pull request has been automatically closed. Thank you for your contributions.

$ diff -u /tmp/abi8/*7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj* /tmp/abi8/*9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w* --- /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj 2020-06-18 18:01:22.831228087 +0900 +++ /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w 2020-07-03 15:59:58.430695244 +0900 @@ -140,7 +140,7 @@ field u8 primitive u8 field solana_sdk::instruction::InstructionError - enum InstructionError (variants = 34) + enum InstructionError (variants = 35) variant(0) GenericError (unit) variant(1) InvalidArgument (unit) variant(2) InvalidInstructionData (unit) @@ -176,6 +176,7 @@ variant(31) CallDepth (unit) variant(32) MissingAccount (unit) variant(33) ReentrancyNotAllowed (unit) + variant(34) MaxSeedLengthExceeded (unit) variant(9) CallChainTooDeep (unit) variant(10) MissingSignatureForFee (unit) variant(11) InvalidAccountIndex (unit)

ryoqun · 2020-07-03T09:46:36Z

sdk/macro-frozen-abi/src/lib.rs

                        #p!("sed -i -e 's/{}/{}/g' $(git grep --files-with-matches frozen_abi)", #expected_digest, hash);
                    }
                    ::log::warn!("Not testing the abi digest under SOLANA_ABI_BULK_UPDATE!");
                } else {
-                  assert_eq!(#expected_digest, format!("{}", hash), "Possibly ABI changed? Confirm the diff by rerunning before and after this test failed with SOLANA_ABI_DUMP_DIR");
+                    if let Ok(dir) = ::std::env::var("SOLANA_ABI_DUMP_DIR") {
+                        assert_eq!(#expected_digest, actual_digest, "Possibly ABI changed? Examine the diff in SOLANA_ABI_DUMP_DIR!: $ diff -u {}/*{}* {}/*{}*", dir, #expected_digest, dir, actual_digest);


@mvines I've made error message explicit how to proceed when encountering this. And, maintenance should be like this: aed8c47

ryoqun · 2020-07-03T09:50:35Z

runtime/src/bank.rs

@@ -74,6 +74,7 @@ pub const SECONDS_PER_YEAR: f64 = 365.25 * 24.0 * 60.0 * 60.0;
 pub const MAX_LEADER_SCHEDULE_STAKES: Epoch = 5;

 type BankStatusCache = StatusCache<Result<()>>;
+#[frozen_abi(digest = "9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w")]


I believe this digest is recently updated by #10748 (FYI: @mvines @jackcmay ) aed8c47

ryoqun · 2020-07-03T09:57:30Z

@mvines I think this pr is ready for review and merge! I've rebased and there was little abi changes so far. So, it looks like there will be little churn for engineers to endure even after merging this. :)

mvines · 2020-07-06T05:19:34Z

runtime/src/serde_snapshot/legacy.rs

@@ -8,6 +10,9 @@ pub(super) struct SerializableAccountStorageEntry {
    count_and_status: (usize, AccountStorageStatus),
 }

+#[cfg(all(test, RUSTC_WITH_SPECIALIZATION))]
+impl IgnoreAsHelper for SerializableAccountStorageEntry {}


It would be nice to have some brief documentation on when IgnoreAsHelper should be used, perhaps in
docs/src/implemented-proposals/abi-management.md

I see! I'll do this as a follow-up pr.

ryoqun · 2023-05-26T06:35:10Z

ci/test-checks.sh

@@ -52,6 +52,7 @@ _ ci/order-crates-for-publishing.py
 _ cargo +"$rust_stable" fmt --all -- --check

 # -Z... is needed because of clippy bug: https://github.com/rust-lang/rust-clippy/issues/4612
+# run nightly clippy for `sdk/` as there's a moderate amount of nightly-only code there


here: #31833

ryoqun added the work in progress This isn't quite right yet label Jan 29, 2020

ryoqun requested a review from mvines January 29, 2020 09:35

ryoqun commented Jan 29, 2020

View reviewed changes

programs/vote/src/vote_instruction.rs Outdated Show resolved Hide resolved

ryoqun commented Jan 29, 2020

View reviewed changes

sdk/macro/src/lib.rs Outdated Show resolved Hide resolved

ryoqun commented Jan 29, 2020

View reviewed changes

sdk/macro/src/lib.rs Outdated Show resolved Hide resolved

mvines reviewed Jan 29, 2020

View reviewed changes

sdk/src/lib.rs Outdated Show resolved Hide resolved

mvines reviewed Jan 29, 2020

View reviewed changes

runtime/src/hard_forks.rs Outdated Show resolved Hide resolved