raft: Garbage collect WAL files #1327

aaronlehmann · 2016-08-05T22:50:02Z

We currently garbage collect snapshot files (keeping only
KeepOldSnapshot outdated snapshots, which defaults to 0). However, we
don't garbage collect the WAL files that the snapshots replace.

Delete any WALs which are so old that they only contain information that
predates the oldest of the snapshots we have retained. This means that
by default, we will remove old WALs once they are supplanted by a
snapshot. However, if KeepOldSnapshots is set above 0, we will keep
whichever WALs are necessary to catch up from the oldest of the retained
snapshots. This makes sure that the old snapshots we retain are actually
useful, and avoids adding an independent knob for WAL retention that
might end up with an inconsistent setting.

Also, fix serious brokenness in the the deletion of old snapshots (it
was deleting the most recent outdated snapshots, instead of the oldest).

Tested using the following patch, using a failure loop to create a steady stream of writes to raft, and restarting the daemon frequently:

diff --git a/manager/state/raft/raft.go b/manager/state/raft/raft.go
index e4d6360..0c96c51 100644
--- a/manager/state/raft/raft.go
+++ b/manager/state/raft/raft.go
@@ -301,8 +301,8 @@ func DefaultNodeConfig() *raft.Config {
 // DefaultRaftConfig returns a default api.RaftConfig.
 func DefaultRaftConfig() api.RaftConfig {
    return api.RaftConfig{
-       KeepOldSnapshots:           0,
-       SnapshotInterval:           10000,
+       KeepOldSnapshots:           2,
+       SnapshotInterval:           5,
        LogEntriesForSlowFollowers: 500,
        ElectionTick:               3,
        HeartbeatTick:              1,
diff --git a/vendor/github.com/coreos/etcd/wal/wal.go b/vendor/github.com/coreos/etcd/wal/wal.go
index 99f1a9c..3004f43 100644
--- a/vendor/github.com/coreos/etcd/wal/wal.go
+++ b/vendor/github.com/coreos/etcd/wal/wal.go
@@ -46,7 +46,7 @@ const (

    // the expected size of each wal segment file.
    // the actual size might be bigger than it.
-   segmentSizeBytes = 64 * 1000 * 1000 // 64MB
+   segmentSizeBytes = 64 * 1000 // 64k
 )

 var (

Also added a unit test that adds a lot of data to force a WAL rollover, checks that the old WAL is deleted iff it completely predates the snapshot, and restarts the cluster to make sure the remaining wal/snapshot combination contains all the data.

cc @abronan @runshenzhu @aluzzardi

aluzzardi · 2016-08-05T22:54:49Z

/cc @chanwit, this will fix the problem you had with wal files taking too much space.

I believe that if you want to try this out, you could stop dockerd, start swarmd with this patch and -d /var/lib/docker/swarm and it will automatically garbage collect the extra .wal files.

@aaronlehmann is that correct?

aaronlehmann · 2016-08-05T22:57:10Z

Not quite. The garbage collection happens every time a snapshot is triggered, so it takes on the order of 10,000 raft commits to make it happen.

aaronlehmann · 2016-08-05T23:25:54Z

I'm having some trouble with the unit test in CI. It seems to get stuck. It passes locally for me. I think it may just be too memory-intensive for the CI virtual machine. The WAL package has a hardcoded 64 MB file rotation threshold, so we need to fill the logs with that much data, and multiplied by 3 nodes in the cluster it must end up being too much for the VM to handle.

I've skipped the test when -test.short is specified, for now. Let me know if there are any better ideas.

codecov-io · 2016-08-05T23:32:14Z

Current coverage is 55.07% (diff: 43.10%)

Merging #1327 into master will decrease coverage by <.01%

@@             master      #1327   diff @@
==========================================
  Files            80         80          
  Lines         12582      12606    +24   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           6931       6943    +12   
- Misses         4693       4702     +9   
- Partials        958        961     +3

Powered by Codecov. Last update 866211a...c72cdf1

runshenzhu · 2016-08-05T23:46:01Z

manager/state/raft/storage.go

-			err := os.Remove(filepath.Join(n.snapDir(), snapFile))
-			if err != nil && removeErr == nil {
-				removeErr = err
+		if curSnapshotIdx >= 0 && i > curSnapshotIdx {


Can we construct remainingSnapshots within this loop? So that it can reduce disk I/O by avoiding the call of ioutil.ReadDir, (or it may not harm the performance because of cache? I don't know).

Also, it may reduce some redundant code.

@runshenzhu: Updated.

runshenzhu · 2016-08-06T00:10:29Z

manager/state/raft/storage.go

+	}
+
+	// Sort snapshot filenames in lexical order
+	sort.Sort(sort.StringSlice(remainingSnapshots))


remainingSnapshots is already sorted, (in reversed order), because snapshots is sorted.

We may even replace this array with var oldestSnap string and construct it from the loop above.

@runshenzhu: Further simplified.

We currently garbage collect snapshot files (keeping only KeepOldSnapshot outdated snapshots, which defaults to 0). However, we don't garbage collect the WAL files that the snapshots replace. Delete any WALs which are so old that they only contain information that predates the oldest of the snapshots we have retained. This means that by default, we will remove old WALs once they are supplanted by a snapshot. However, if KeepOldSnapshots is set above 0, we will keep whichever WALs are necessary to catch up from the oldest of the retained snapshots. This makes sure that the old snapshots we retain are actually useful, and avoids adding an independent knob for WAL retention that might end up with an inconsistent setting. Also, fix serious brokenness in the the deletion of old snapshots (it was deleting the most recent outdated snapshots, instead of the oldest). Signed-off-by: Aaron Lehmann <[email protected]>

runshenzhu · 2016-08-06T00:19:31Z

LGTM

chanwit · 2016-08-06T16:20:13Z

Thank you @aaronlehmann for implementing this and @aluzzardi for the ping.
This PR SGTM.

An addition requirement here. Is it possible to make KeepOldSnapshots configurable?
This would be really great:

for production, we set the value to 2 and disk space will never ran out.
for testing, we set the value to 0 and Swarm will keep all WALs for time-based troubleshooting (like we did during the analysis).

aaronlehmann · 2016-08-08T16:31:46Z

Yes, I agree that KeepOldSnapshots should be configurable. I opened an issue on Docker a few days ago: moby/moby#25451

aaronlehmann · 2016-08-08T16:32:42Z

But note that setting KeepOldSnapshots to 0 would mean that it only keeps the latest snapshot. If you want to keep all of them, it should be set to a large number.

chanwit · 2016-08-09T15:34:17Z

@aaronlehmann I see thank you for that :-)

abronan · 2016-08-10T23:12:15Z

LGTM

chanwit · 2016-08-11T00:55:09Z

🎉 🎉

aaronlehmann added area/raft process/cherry-pick labels Aug 5, 2016

aaronlehmann added this to the 1.12.1 milestone Aug 5, 2016

GordonTheTurtle added the status/0-triage label Aug 5, 2016

aaronlehmann force-pushed the gc-wals branch 2 times, most recently from ecd52e4 to 24ed09e Compare August 5, 2016 23:23

runshenzhu reviewed Aug 5, 2016
View reviewed changes

aaronlehmann force-pushed the gc-wals branch from 24ed09e to 27848a5 Compare August 5, 2016 23:55

runshenzhu reviewed Aug 6, 2016
View reviewed changes

aaronlehmann force-pushed the gc-wals branch from 27848a5 to c72cdf1 Compare August 6, 2016 00:13

aaronlehmann added status/2-code-review and removed status/0-triage labels Aug 6, 2016

abronan merged commit f0e46aa into moby:master Aug 10, 2016

aaronlehmann deleted the gc-wals branch August 10, 2016 23:30

aaronlehmann added process/cherry-picked and removed process/cherry-pick labels Aug 10, 2016

aaronlehmann mentioned this pull request Aug 11, 2016

Revendor etcd/raft #1353

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raft: Garbage collect WAL files #1327

raft: Garbage collect WAL files #1327

aaronlehmann commented Aug 5, 2016

aluzzardi commented Aug 5, 2016

aaronlehmann commented Aug 5, 2016

aaronlehmann commented Aug 5, 2016

codecov-io commented Aug 5, 2016 •

edited

Loading

runshenzhu Aug 5, 2016

aaronlehmann Aug 5, 2016

runshenzhu Aug 6, 2016

aaronlehmann Aug 6, 2016

runshenzhu commented Aug 6, 2016

chanwit commented Aug 6, 2016

aaronlehmann commented Aug 8, 2016

aaronlehmann commented Aug 8, 2016

chanwit commented Aug 9, 2016

abronan commented Aug 10, 2016

chanwit commented Aug 11, 2016

raft: Garbage collect WAL files #1327

raft: Garbage collect WAL files #1327

Conversation

aaronlehmann commented Aug 5, 2016

aluzzardi commented Aug 5, 2016

aaronlehmann commented Aug 5, 2016

aaronlehmann commented Aug 5, 2016

codecov-io commented Aug 5, 2016 • edited Loading

Current coverage is 55.07% (diff: 43.10%)

runshenzhu Aug 5, 2016

Choose a reason for hiding this comment

aaronlehmann Aug 5, 2016

Choose a reason for hiding this comment

runshenzhu Aug 6, 2016

Choose a reason for hiding this comment

aaronlehmann Aug 6, 2016

Choose a reason for hiding this comment

runshenzhu commented Aug 6, 2016

chanwit commented Aug 6, 2016

aaronlehmann commented Aug 8, 2016

aaronlehmann commented Aug 8, 2016

chanwit commented Aug 9, 2016

abronan commented Aug 10, 2016

chanwit commented Aug 11, 2016

codecov-io commented Aug 5, 2016 •

edited

Loading