Make snapshot process snapshot all unflushed blocks #1017
Conversation
richardartoul
commented
Oct 4, 2018
•
richardartoul
commented
- - Modify snapshot behavior to snapshot all unflushed blocks within the retention period
- - Update integration tests to capture new behavior
richardartoul
changed the title
Codecov Report
@@ Coverage Diff @@
## master #1017 +/- ##
========================================
- Coverage 71.1% 71.1% -0.1%
========================================
Files 732 732
Lines 61493 61520 +27
========================================
+ Hits 43749 43763 +14
- Misses 14922 14937 +15
+ Partials 2822 2820 -2
Continue to review full report at Codecov.
|
richardartoul
changed the title
src/dbnode/storage/util.go
Outdated
| @@ -49,7 +49,7 @@ func timesInRange(startInclusive, endInclusive time.Time, windowSize time.Durati | |||
| return nil | |||
| } | |||
| times := make([]time.Time, 0, ni) | |||
| for t := endInclusive; !t.Before(startInclusive); t = t.Add(-windowSize) { | |||
There was a problem hiding this comment.
Were there assumptions elsewhere in the code that relies on the time ranges always including the end time? Also fix the comment for this function.
|
@richardartoul talked offline: maybe add a stat for the number of blocks being snapshot |
richardartoul
force-pushed
the
ra/snapshot-everything
branch
from
c7b8972 to
ee4f425
Compare
| snapshotFiles, err := fs.SnapshotFiles( | ||
| filePathPrefix, namespace, shard) | ||
| if err != nil { | ||
| panic(err) |
There was a problem hiding this comment.
don't panic here, propagate error
There was a problem hiding this comment.
meh just saw its only for integration tests, w/e. feel free to leave as is.
| candidateTimes := timesInRange(earliest, latest, blockSize) | ||
| return filterTimes(candidateTimes, func(t time.Time) bool { | ||
| // Snapshot anything that is unflushed. | ||
| return ns.NeedsFlush(t, t) |
There was a problem hiding this comment.
do you need to check snapshot enabled here
