Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support for backup compaction #301

Merged
merged 8 commits into from
Jun 9, 2021
Merged

Added support for backup compaction #301

merged 8 commits into from
Jun 9, 2021

Conversation

abdasgupta
Copy link
Contributor

@abdasgupta abdasgupta commented Jan 5, 2021 •
edited by amshuman-kr
Loading

What this PR does / why we need it:
So far, compaction was done manually on the ETCD instance running in seed cluster. But with this PR, compaction is introduced as a command to the ETCDBR. At this point, ETCDBR will be able to restore-defragment-compact-snapshot without any manual intervention in between.

Which issue(s) this PR fixes:
Fixes gardener/etcd-druid#88

Release note:

Added compaction command support to ETCD Backup Restore. Users can now use compaction sub command with `etcdbrctl` tool to compact basesnapshot and all it's subsequent deltasnapshot to one single compacted snapshot.

Snapshots are not stored under separate prefix (Backup-XXX) in backup storage. Instead, all of the snapshots are stored under `v2` backup version prefix in a flat structure.

Support for `v1` backup version is still there but only for restoration and snapshot garbage collection. So backup storages that had `v1` backup version and where snapshots were stored  under separate prefix (Backup-XXX), can still be used with compaction sub command (as well as for restoration). However, the new snapshots will be stored under `v2` backup version prefix.

The support for the `v1` backup structure with separate prefix (Backup-XXX) will be dropped in a subsequent release.

@gardener-robot gardener-robot added needs/review Needs review size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) needs/second-opinion Needs second review by someone else labels Jan 5, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 5, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 added needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jan 5, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 5, 2021
@gardener-robot-ci-1 gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 5, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 12, 2021
@gardener-robot-ci-1 gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 12, 2021
amshuman-kr
amshuman-kr suggested changes Jan 21, 2021
View reviewed changes
Copy link
Collaborator

@amshuman-kr amshuman-kr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sorry about the delay in the review @abdasgupta! Some questions and suggestions are below.

cmd/compact.go Outdated Show resolved Hide resolved
cmd/compact.go Outdated Show resolved Hide resolved
cmd/compact.go Show resolved Hide resolved
cmd/compact.go Outdated Show resolved Hide resolved
cmd/compact.go Outdated Show resolved Hide resolved
pkg/compactor/compactor.go Outdated Show resolved Hide resolved
pkg/compactor/compactor.go Outdated Show resolved Hide resolved
pkg/compactor/compactor_suite_test.go Outdated Show resolved Hide resolved
pkg/compactor/compactor_test.go Show resolved Hide resolved
pkg/types/compactor.go Outdated Show resolved Hide resolved
@gardener-robot gardener-robot added the needs/changes Needs (more) changes label Jan 21, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 29, 2021
@gardener-robot-ci-1 gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 29, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 29, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 29, 2021
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 29, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 29, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Feb 2, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Feb 2, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Feb 3, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jun 3, 2021
@gardener-robot
Copy link

@abdasgupta You have pull request review with status CHANGES_REQUESTED, please check

1 similar comment
@gardener-robot
Copy link

@abdasgupta You have pull request review with status CHANGES_REQUESTED, please check

@gardener-robot-ci-1 gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jun 9, 2021
amshuman-kr
amshuman-kr approved these changes Jun 9, 2021
View reviewed changes
Copy link
Collaborator

@amshuman-kr amshuman-kr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@gardener-robot gardener-robot added reviewed/lgtm Has approval for merging and removed needs/changes Needs (more) changes needs/review Needs review needs/second-opinion Needs second review by someone else labels Jun 9, 2021
@gardener-robot gardener-robot added needs/second-opinion Needs second review by someone else and removed reviewed/lgtm Has approval for merging labels Jun 9, 2021
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 9, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 9, 2021
@amshuman-kr amshuman-kr changed the title Added support for compaction Added support for backup compaction Jun 9, 2021
@amshuman-kr amshuman-kr merged commit 017c6d6 into gardener:master Jun 9, 2021
@abdasgupta abdasgupta deleted the working branch June 9, 2021 08:20
@abdasgupta abdasgupta mentioned this pull request Jun 9, 2021
Closed
@ishan16696 ishan16696 mentioned this pull request Jun 13, 2021
Merged
This was referenced Jun 14, 2021
Closed
Closed
@abdasgupta abdasgupta mentioned this pull request Aug 11, 2021
Merged
This was referenced Sep 13, 2021
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ishan16696 ishan16696 ishan16696 requested changes

@amshuman-kr amshuman-kr amshuman-kr approved these changes

Assignees
No one assigned
Labels
needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/second-opinion Needs second review by someone else size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py)
Projects
None yet
Milestone
v0.13.0
Development

Successfully merging this pull request may close these issues.

Compacting Incremental Snapshot Files
8 participants
@abdasgupta @ishan16696 @gardener-robot @amshuman-kr @gardener-robot-ci-1 @gardener-robot-ci-2 @gardener-robot-ci-3 @aaronfern