-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compacting Incremental Snapshot Files #88
Comments
We haven't planned compacting the incremental backup files into one as of now. But if you want we create a sub command that will take the incremental snaps and compact them and push them at command execution. |
We just need this for our restore process which is manual at the moment. So it would be enough if there would be a sub command which works on a folder containing the full backup and the incremental updates (e.g. downloaded from the corresponding S3 bucket folder) and creates a compacted full backup at the end. We could run this locally. This one we could then easily feed in when we restore the cluster. |
There is one way to do this indirectly. If you delete the member directory and do a data directory initialization, the data directory will be restored from the latest full and incremental snapshot. Can you check if this is sufficient? |
You can use |
Thanks guys! I will have a look into it. |
Hey guys, So, please think about compacting the incremental snapshots so that we can significantly speed up the restoration process. |
Yes. This is needed. |
We had listed 4 topics for optimization for etcd-backup-restore.
I think that this requirement for compaction of delta snapshots comes under the database restoration optimisation and should be picked along with it. We can even pick it in parallel to the topic of database verification if we have the bandwidth. |
@shreyas-s-rao @swapnilgm Should this ticket be rather moved to https://github.com/gardener/etcd-druid/issues? While we are at it, maybe there are more issues in this repo that should go there? |
Makes sense. These issue were crated prior to introducing etcd-druid. |
I think there is no real benefit of doing incrementals at all, when looking at our environment, full backup is ~100MB and a increment ~30-40MB uncompressed. I really would propose to switch to full backups only but compress them with lz4, this will lead to smaller full backups than the actual incremental files. The decompression will add ~0.2 sec. per file before the actual restore can start. But overall the restoration time will decrease by factors. related to: gardener/etcd-backup-restore#263 |
@majst01 We require incremental snapshots in a general sense to avoid frequent large full snapshots which usually create a network costs. If you do want to avoid delta snapshots, you can simply configure Regarding backup compression, we have opened gardener/etcd-backup-restore#255 and it's on the project roadmap. |
Thanks @shreyas-s-rao for the hints. We tried ourselves to set the backup to do full backups only, but from our experience setting |
@majst01 I just tried the following and delta snapshots were disabled with only fullsnapshots enabled. backup:
...
fullSnapshotSchedule: "* * * * *"
...
deltaSnapshotPeriod: 0s
...
Did you mean the above? |
It really depends on the workload pattern in the cluster. In most clusters the delta snapshots are much smaller than 100M but in the seed and garden control-plane, it is quite high. As a consequence, the full snapshot size is also quite high (2-3G). We have an issue to compress/decompress snapshots (full as well as delta) gardener/etcd-backup-restore#255. We also have this issue to compact the delta snapshots in the background. In the special case, the full snapshots are small and comparable to the delta snapshot, it might still make sense to do only full snapshots as you mentioned. Would it make sense to make this configurable in gardener? |
@amshuman-kr in which resource did you set these setting ?
|
The |
The |
/assign |
This issue was partially addressed in gardener/etcd-backup-restore#301. The functionality will be complete once #191 is completed. |
This issue is now fully addressed with #197. Hence closing it. |
Question: How are you guys dealing with the incremental backup files when restoring a cluster? I am asking, because in Kubify we expect one full snapshot file to trigger a restore operation. What is the best way to compact all the incremental backup files into one? Do you have already something handy?
The text was updated successfully, but these errors were encountered: