Lack of consistency checks for stakes cache

[jstarry]

Problem

After the feature gate for #24278 is activated, validators will calculate stake rewards using vote and stake account state cached in the StakesCache. However, there are currently no consistency checks that ensure that the stakes cache accurately reflects the state of all active vote and stake accounts in accounts db.

One potential pitfall is that the stakes cache implementation assumes that stake and vote accounts cannot have their owner changed during transaction processing. This is safe now but the stake and vote programs could introduce a future change that resets the owner of a closed account and break that assumption.

Context: #24200 (comment)

Proposed Solution

TBD

[ryoqun]

cc: @HaoranYi might worth to do considering stake rewards v2

[HaoranYi]

@ryoqun Great observation. Yeah. This is another blocker for #24200. I will keep it in my mind for the rewards v2 work.

[jeffwashington]

@behzadnouri , @t-nelson , @HaoranYi can one of you please help clarify? Is this an issue for activation? An issue that still needs work done on it?

[t-nelson]

this was addressed in #24200

[jeffwashington]

Sorry to be pedantic.
@behzadnouri @t-nelson @HaoranYi just being clear.
This issue has do-not-close from @behzadnouri last month.
@t-nelson says it is addressed in #24200, which looks like it went in over a year ago.
@behzadnouri can you please confirm? Thank you.

[behzadnouri]

The source of inconsistency was patched in #26479 and the sanity checks here
https://github.com/solana-labs/solana/blob/936d2cf3c/runtime/src/stakes.rs#L224-L263
will cause panic at start-up if stakes-cache is inconsistent with accounts-db:

#24200 only added these metrics:
https://github.com/solana-labs/solana/blob/936d2cf3c/runtime/src/bank.rs#L1092-L1094
to track if there are inconsistencies between cache and accounts-db before feature activation; but the feature activation will bypass those metrics.