Skip to content

Commit

Permalink
Snapshot calculation must continue
Browse files Browse the repository at this point in the history
Do not block snapshot calculation even if there's an ongoing snapshot
stream. If this stream takes a long time and we don't calculate
snapshot, we run the risk of increasing the size of Raft logs so much
that it causes group wide issues.
  • Loading branch information
manishrjain committed Apr 30, 2019
1 parent b190127 commit ecb4547
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions worker/draft.go
Original file line number Diff line number Diff line change
Expand Up @@ -1000,10 +1000,12 @@ func (n *node) calculateSnapshot(discardN int) (*pb.Snapshot, error) {
_, span := otrace.StartSpan(n.ctx, "Calculate.Snapshot")
defer span.End()

if atomic.LoadInt32(&n.streaming) > 0 {
span.Annotate(nil, "Skipping calculateSnapshot due to streaming")
return nil, nil
}
// We do not need to block snapshot calculation because of a pending stream. Badger would have
// pending iterators which would ensure that the data above their read ts would not be
// discarded. Secondly, if a new snapshot does get calculated and applied, the follower can just
// ask for the new snapshot. Blocking snapshot calculation has caused us issues when a follower
// somehow kept streaming forever. Then, the leader didn't calculate snapshot, instead it
// kept appending to Raft logs forever causing group wide issues.

first, err := n.Store.FirstIndex()
if err != nil {
Expand Down

0 comments on commit ecb4547

Please sign in to comment.