Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Improve performance of restoration, especially for larger backups #604

Closed
7 tasks done
shreyas-s-rao opened this issue Mar 15, 2023 · 4 comments · Fixed by #609
Closed
7 tasks done
Assignees
Labels
kind/enhancement Enhancement, improvement, extension status/closed Issue is closed (either delivered or triaged)

Comments

@shreyas-s-rao
Copy link
Collaborator

shreyas-s-rao commented Mar 15, 2023

Enhancement (What you would like to be added):
Improve multiple aspects of restoration, to allow restoration of large ETCDs without running out of disk space or exceeding ETCD DB quota.

Motivation (Why is this needed?):
Large ETCDs lead to large backups, with more number of delta snapshots, larger delta snapshots or both. Trying to restore such large ETCDs might fail due to (a) running out of disk space due to storing too many temp delta snapshot files on disk until restoration is complete, (b) embedded etcd DB running out of space during restoration.

Approach/Hint to the implement solution (optional):

  • [BUG] Temp files from restoration are not cleaned up as expected #603
  • Instead of cleaning up the temp files after restoration succeeds, cleanup each temp delta snapshot file as and when the delta snapshot is successfully applied to the embedded etcd
  • Periodically run defragmentation on the embedded etcd during restoration #636
  • Decompress download delta snapshots just before applying them, thus saving disk space when persisting delta snapshots locally
  • Make the temp dir configurable via CLI, since it is currently hardcoded to /tmp, in order to decouple from the /tmp folder which might be re-purposed by the OS at any time in the future
  • Name the temp files after delta snapshot file names to make debugging issues with cleanup easier
  • Improve logging for restoration - ex: log total number of deltas before starting restoring
@shreyas-s-rao shreyas-s-rao added the kind/enhancement Enhancement, improvement, extension label Mar 15, 2023
@shreyas-s-rao shreyas-s-rao added this to the v0.22.0 milestone Mar 15, 2023
@shreyas-s-rao
Copy link
Collaborator Author

/assign

@ishan16696
Copy link
Member

backup-restore v0.22.0 already has been released, @shreyas-s-rao you have wrongly added v0.22.0 as milestone. so I'm updating the milestone to v0.23.0. Please feel free to update milestone further

@shreyas-s-rao
Copy link
Collaborator Author

Oops, my bad. Thanks for correcting it.

@shreyas-s-rao
Copy link
Collaborator Author

Periodically run (compaction and) defragmentation on the embedded etcd during restoration, either after every X deltas applied or after every Y GB of etcd DB size reached

This task will be handled in a separate issue (probably gardener/etcd-druid#232 once it moves to etcd-backup-restore repo).

@gardener-robot gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement Enhancement, improvement, extension status/closed Issue is closed (either delivered or triaged)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants