Skip to content
This repository has been archived by the owner on Jan 22, 2025. It is now read-only.

Remove ledger purge batching #10830

Merged
merged 1 commit into from
Jun 29, 2020

Conversation

sakridge
Copy link
Contributor

@sakridge sakridge commented Jun 29, 2020

Problem

Batching purges seems to lower performance overall since each delete_range seems to be constant time. Nodes can get into a situation of deleting a huge range of slots and they will issue many delete_ranges which each cause writes to the db and slowing down insert.

Some example of the node in tds that died #10821:

[2020-06-26T06:58:10.443246600Z INFO  solana_core::ledger_cleanup_service] first_slot=0 total_slots=386755 total_shreds=50039179 max_ledger_shreds=50000000, iterate_time took 323ms
[2020-06-26T06:58:10.443759936Z INFO  solana_core::ledger_cleanup_service] purging data from slots 0 to 20606000

It had 50m shreds in 386k slots. It probably only had data back to slot: 20,606,000 - 386,755 = 20,219,245. But it walked from slot 0 all the way up.

Summary of Changes

Don't purge in batches, just do one delete_range.

Fixes #

@sakridge sakridge force-pushed the remove-purge-batching branch from d247bcb to b451775 Compare June 29, 2020 17:41
@sakridge sakridge added the v1.2 label Jun 29, 2020
@sakridge sakridge requested review from carllin and mvines June 29, 2020 17:41
@codecov
Copy link

codecov bot commented Jun 29, 2020

Codecov Report

Merging #10830 into master will decrease coverage by 0.0%.
The diff coverage is 92.3%.

@@            Coverage Diff            @@
##           master   #10830     +/-   ##
=========================================
- Coverage    81.8%    81.8%   -0.1%     
=========================================
  Files         303      303             
  Lines       71087    71067     -20     
=========================================
- Hits        58220    58192     -28     
- Misses      12867    12875      +8     

@sakridge sakridge merged commit 583cec9 into solana-labs:master Jun 29, 2020
@sakridge sakridge deleted the remove-purge-batching branch June 29, 2020 21:44
mergify bot pushed a commit that referenced this pull request Jun 29, 2020
(cherry picked from commit 583cec9)
mergify bot added a commit that referenced this pull request Jun 29, 2020
(cherry picked from commit 583cec9)

Co-authored-by: sakridge <[email protected]>
@ryoqun ryoqun mentioned this pull request Jun 30, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants