Introduce automatic ABI maintenance mechanism (1/2; prepare)

[ryoqun]

Problem

ABI is hard to track.

Summary of Changes

This is the preparation pr for the next rollup of frozen_abi

Part of #7738

[codecov]

Codecov Report

Merging #10335 into master will decrease coverage by 0.1%.
The diff coverage is 77.6%.

@@            Coverage Diff            @@
##           master   #10335     +/-   ##
=========================================
- Coverage    81.4%    81.3%   -0.2%     
=========================================
  Files         290      294      +4     
  Lines       67610    68228    +618     
=========================================
+ Hits        55081    55487    +406     
- Misses      12529    12741    +212     

[ryoqun]

@mvines The CI is green! This is implementation-only split of #8012. This is still rather big pr, but could you review this?

[mvines]

@garious / @aeyakovenko - I'm almost 100% sure that ABI management is going to be a thing that both of you will rage about tripping over after it's rolled out (maybe more @aeyakovenko ;) ), so I suggest you both participate in building the shackles. Care to review?

[ryoqun]

This is the starting point!

[ryoqun]

@garious / @aeyakovenko - I'm almost 100% sure that ABI management is going to be a thing that both of you will rage about tripping over after it's rolled out (maybe more @aeyakovenko ;) ), so I suggest you both participate in building the shackles. Care to review?

For devx matter, I've taken extra effort to easy the burden, btw. The implementation is really advanced/complex (at least for half-year rust programmer), well worth for its own crate but it's really encapsulated, I believe.

For devx, managing abi sanely just boils down to this test assertion: https://github.com/solana-labs/solana/pull/10335/files#r433615921

Basically, this is where you'll start to being tripped over. When you encounter this failure on CI, just run the abi test twice and diff the dumped abi digest text file and justify nothing odd is added/removed there, then commit the new digest. I made that digest file parsable for humans.

Or just page @ryoqun or remove frozen_abi if it's really urgent.

As the last resort, if you dare, take the look of the internal this PR is about to add. :)

Btw, this pr stranded for several months unfortunately for more important snapshot thing and it evolved for each rebase, I'm pretty sure this mechanism can detect most of abi changes nicely and updating is one-line change for engineers.

[ryoqun]

Wow, cool. I'll merge this finally! :)

Pull request has been modified.

[ryoqun]

@garious / @aeyakovenko - I'm almost 100% sure that ABI management is going to be a thing that both of you will rage about tripping over after it's rolled out (maybe more @aeyakovenko ;) ), so I suggest you both participate in building the shackles. Care to review?

For devx matter, I've taken extra effort to easy the burden, btw. The implementation is really advanced/complex (at least for half-year rust programmer), well worth for its own crate but it's really encapsulated, I believe.

For devx, managing abi sanely just boils down to this test assertion: https://github.com/solana-labs/solana/pull/10335/files#r433615921

Basically, this is where you'll start to being tripped over. When you encounter this failure on CI, just run the abi test twice and diff the dumped abi digest text file and justify nothing odd is added/removed there, then commit the new digest. I made that digest file parsable for humans.

Or just page @ryoqun or remove frozen_abi if it's really urgent.

As the last resort, if you dare, take the look of the internal this PR is about to add. :)

Btw, this pr stranded for several months unfortunately for more important snapshot thing and it evolved for each rebase, I'm pretty sure this mechanism can detect most of abi changes nicely and updating is one-line change for engineers.

@garious @aeyakovenko Because I got +1, I've merged this preparation pr and master is green at the moment. I guess it's now the last call for actually enabling abi maintenance (or burden). For more concrete example for what's needed by engineers to maintain the abi, please take a look at #8012, which I've rebased already.