Check for newer incremental snapshot before downloading #28143
Conversation
There was a problem hiding this comment.
I was just thinking about this! Thanks for taking a look. I was originally think about being more invasive and changing bootstrap to run the discovery of incremental snapshots after downloading the full snapshot. Your impl is a more straight-forward way to accomplish this task. If we discover issues later, we can also go full-bore with bootstrap.rs.
There was a problem hiding this comment.
In general I would prefer get last snapshot hash from gossip as in calling code, or send the bootstrap to second round for finding snapshots again (it'll skip downloading full snapshot ... maybe 😀)
Second, the snapshot hash of this newer incremental snapshot must be checked with known validators, if at least one of them is defined (--known-validator option).
| // Check and see if we've already got the incremental snapshot; if not, download it | ||
| if let Some(incremental_snapshot_hash) = incremental_snapshot_hash { | ||
| // If the full snapshot download took more than a few minutes there is probably a newer incremental snapshot | ||
| let newest_known_incremental_snapshot = match check_for_newer_incremental_snapshot( |
There was a problem hiding this comment.
I would use shadowing here, we don't need original variable below.
let incremental_snapshot_hash = match check_for_newer_incremental_snapshot(
...
And it'll reduce PR too.
PS. I don't have write permission for suggesting.
|
@brooksprumo Maybe good simple solution is just check the difference between last snapshot slot and current slot after bootstrapping. If it greater then |
Yeah, that sounds good to me! This would also hopefully fix the problem where the gossip tables are not fully populated and cause bootstrap to not see any incremental snapshots to download. With the extra time of downloading a full snapshot, that should be plenty of time to get the gossip tables with better information. |
|
Related issue: #26180 |
github-actions
bot
added
the
stale
Problem
When a validator downloads snapshots on startup it follows these steps:
Step 2 can take a few minutes (eg on my dev machine it takes 10-20 minutes to download the 34GB mainnet-beta snapshot). With default configs (incremental snapshots every 100 slots, retain 4 snapshots) each incremental snapshot persists for about 8 minutes. By the time we get to step 3 the incremental snapshot has been deleted and we get a 404.
Without an incremental snapshot the node attempts to start from the full snapshot which may be very outdated.
Summary of Changes
After downloading the full snapshot get the slot and hash of the latest incremental snapshot before attempting the incremental snapshot download.