SnapshotPackagerService purges old bank snapshots

[brooksprumo]

Problem

We retain more bank snapshots than we need, longer than we need.

Bank snapshots are currently used for one thing: creating a snapshot archive. And currently in development, we'll also use bank snapshots to speed up restart by booting from a bank snapshot instead of a snapshot archive. In any case, an observation is that once a snapshot has been archived, all bank snapshots older are no longer useful.

(We want to keep around the bank snapshot that was just archived for the fast boot case mentioned above.)

Also, bank snapshots contain hard-links to account storage files (for fast boot). We should free those up as fast as possible, since they can artificially take up space if the accounts are stored in a RAM disk.

Note that we still have an outstanding problem to solve: What about bank snapshots that are not archived? This happens for accounts hash verifier requests, and if a node has disabled snapshots (which still must produce bank snapshots for AHV, but not snapshot archives). I'll be working on that separately from this PR.

Summary of Changes

SnapshotPackagerService will purge bank snapshots older than the one that was just archived.

[codecov]

Codecov Report

Merging #31511 (e8db8ac) into master (39a092c) will increase coverage by 81.4%.
The diff coverage is 88.8%.

@@             Coverage Diff             @@
##           master   #31511       +/-   ##
===========================================
+ Coverage        0    81.4%    +81.4%     
===========================================
  Files           0      731      +731     
  Lines           0   205114   +205114     
===========================================
+ Hits            0   167019   +167019     
- Misses          0    38095    +38095     

[apfitzge]

LGTM - logic seems sound that we don't want to keep the snapshots (hardlinks) around. They are no longer necessary after we archive (keeping current).

Taking it a step further, what reason is there for having more than the most reason snapshot?

[brooksprumo]

LGTM - logic seems sound that we don't want to keep the snapshots (hardlinks) around. They are no longer necessary after we archive (keeping current).

Taking it a step further, what reason is there for having more than the most reason snapshot?

We will want to keep around the most recent bank snapshot so that we can boot from it (eventually). Beyond that, there's no need.

When the bank snapshots were put into temp directories, they were cleaned up as part of a normal running validator in AccountsBackgroundService when taking a new snapshot (bank, not archive). That's not really safe now, since the bank snapshots for archival are not in temp dirs... We're kinda hoping that AHV and SPS run fast enough so that ABS doesn't rug the bank snapshots... That's a future PR to fix up.

I'm sure there will be more to learn/fix here as well.