Use storage version API #85

roycaihw · 2020-12-07T23:04:05Z

This PR changes the storage migrator trigger controller to use the new storage version API, which supports HA clusters. This PR also adds a test pull-kube-storage-version-migrator-ha-master, which rolling-updates an HA cluster to change the storage version of CRD from apiextensions.k8s.io/v1beta1 to apiextensions.k8s.io/v1.

The two existing tests pull-kube-storage-version-migrator-fully-automated-e2e and pull-kube-storage-version-migrator-disruptive are failing now because they exercise storage migration for custom resources. These tests will pass when custom resource storage version support is merged upstream: kubernetes/kubernetes#96403 (which I'm working on this week)

roycaihw · 2020-12-08T05:51:36Z

/test pull-kube-storage-version-migrator-ha-master

k8s-ci-robot · 2020-12-08T05:55:20Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: roycaihw
To complete the pull request process, please assign caesarxuchao after the PR has been reviewed.
You can assign the PR to them by writing /assign @caesarxuchao in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2020-12-14T20:24:42Z

@roycaihw: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kube-storage-version-migrator-fully-automated-e2e	`4e2ae36`	link	`/test pull-kube-storage-version-migrator-fully-automated-e2e`
pull-kube-storage-version-migrator-disruptive	`4e2ae36`	link	`/test pull-kube-storage-version-migrator-disruptive`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

caesarxuchao

I need more time to think about the logic in storageversion_handler.go, so you don't need to address my comments in that file.

We need to revisit the storageState API. Currently it's recording the storageVersionHash, we need to convert it to recording storage version.

The storageState.status.lastHeartbeatTime field probably can be removed, because we now uses watch instead of polling, we won't miss the storage version change.

caesarxuchao · 2020-12-15T08:22:09Z

pkg/trigger/controller.go

+		storageVersionInformer: apiserverinternalinformers.NewStorageVersionInformer(kubeClient,
+			0,
+			cache.Indexers{}),
+		queue:                  workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "migration_triggering_controller"),


Can we name this queue and respective enqueue/dequeue function as well? Maybe just migrationQueue.

caesarxuchao · 2020-12-15T08:23:55Z

pkg/trigger/controller.go

+func (mt *MigrationTrigger) updateStorageVersion(_ interface{}, obj interface{}) {
+	mt.addStorageVersion(obj)
+}
+


I think we should handle deleteStorageVersion as well. The controller can garbage collect respective storageVersionMigration and storageState objects.

caesarxuchao · 2020-12-15T08:29:02Z

pkg/trigger/discovery_handler.go

 	m := &migrationv1alpha1.StorageVersionMigration{
 		ObjectMeta: metav1.ObjectMeta{
-			GenerateName: storageStateName(resource) + "-",


We need to garbage collects the old storageVersionMigration. cleanMigrations won't GC the old ones. We can do this in a follow-up.

caesarxuchao · 2020-12-15T08:43:03Z

pkg/trigger/controller.go

@@ -47,25 +50,40 @@ const (
 )

 type MigrationTrigger struct {


I think we can split the Trigger to two controllers. The storageVersion control loop and the migration control loop is decoupled.

caesarxuchao · 2020-12-15T08:44:12Z

pkg/trigger/storageversion_handler.go

+)
+
+func (mt *MigrationTrigger) processStorageVersion(ctx context.Context, sv *apiserverinternalv1alpha1.StorageVersion) error {
+	klog.V(2).Infof("processing storage version %#v", sv)


In production we don't want to log the entire storageVersion object.

caesarxuchao · 2020-12-15T08:50:45Z

pkg/trigger/storageversion_handler.go

+		}
+	}
+	storageVersionChanged := found && (ss.Status.CurrentStorageVersionHash != *sv.Status.CommonEncodingVersion ||
+		lastTransitionTime != mt.lastSeenTransitionTime[sv.Name]) && foundCondition


Can we make a standalone variable for case storageVersionlastTransitionTime != mt.lastSeenTransitionTime[sv.Name], maybe naming it storageVersionPossiblyFlipped (or a better name)?

caesarxuchao · 2020-12-15T09:14:03Z

Also I don't remember if the migrator skips migrating the event resource currently, perhaps event resource doesn't have a storageVersionHash.

We should skip migrating events, it's transient anyway.

fejta-bot · 2021-03-15T09:49:03Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

roycaihw · 2021-03-16T18:13:14Z

/remove-lifecycle stale

fejta-bot · 2021-06-14T18:17:18Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

fejta-bot · 2021-07-14T18:44:54Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

k8s-triage-robot · 2021-08-13T19:12:22Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2021-08-13T19:12:31Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot assigned caesarxuchao Dec 7, 2020

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 7, 2020

k8s-ci-robot requested review from caesarxuchao and lavalamp December 7, 2020 23:04

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 7, 2020

roycaihw force-pushed the kinder-e2e branch from cb242d8 to 69a0921 Compare December 8, 2020 05:55

roycaihw force-pushed the kinder-e2e branch from 69a0921 to 1f7b751 Compare December 8, 2020 05:59

This was referenced Dec 8, 2020

[WIP] use storage version API #82

Closed

Add an HA test case #84

Closed

roycaihw force-pushed the kinder-e2e branch 10 times, most recently from 472da72 to 2b87611 Compare December 14, 2020 01:39

roycaihw added 3 commits December 14, 2020 11:56

use storage version API

966033f

generated

9efdb7c

add a HA rolling upgrade test

4e2ae36

roycaihw force-pushed the kinder-e2e branch from eb72d7b to 4e2ae36 Compare December 14, 2020 19:56

roycaihw mentioned this pull request Dec 14, 2020

add rbac rule for storage version garbage collector kubernetes/kubernetes#97293

Merged

caesarxuchao reviewed Dec 15, 2020

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 15, 2021

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 16, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 14, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 14, 2021

k8s-ci-robot closed this Aug 13, 2021

furkatgofurov7 mentioned this pull request Jun 21, 2022

Handle status.storedVersions migration during upgrades kubernetes-sigs/cluster-api#6691

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use storage version API #85

Use storage version API #85

roycaihw commented Dec 7, 2020 •

edited

Loading

roycaihw commented Dec 8, 2020

k8s-ci-robot commented Dec 8, 2020

k8s-ci-robot commented Dec 14, 2020

caesarxuchao left a comment

caesarxuchao Dec 15, 2020

caesarxuchao Dec 15, 2020

caesarxuchao Dec 15, 2020

caesarxuchao Dec 15, 2020

caesarxuchao Dec 15, 2020

caesarxuchao Dec 15, 2020

caesarxuchao commented Dec 15, 2020

fejta-bot commented Mar 15, 2021

roycaihw commented Mar 16, 2021

fejta-bot commented Jun 14, 2021

fejta-bot commented Jul 14, 2021

k8s-triage-robot commented Aug 13, 2021

k8s-ci-robot commented Aug 13, 2021

Use storage version API #85

Use storage version API #85

Conversation

roycaihw commented Dec 7, 2020 • edited Loading

roycaihw commented Dec 8, 2020

k8s-ci-robot commented Dec 8, 2020

k8s-ci-robot commented Dec 14, 2020

caesarxuchao left a comment

Choose a reason for hiding this comment

caesarxuchao Dec 15, 2020

Choose a reason for hiding this comment

caesarxuchao Dec 15, 2020

Choose a reason for hiding this comment

caesarxuchao Dec 15, 2020

Choose a reason for hiding this comment

caesarxuchao Dec 15, 2020

Choose a reason for hiding this comment

caesarxuchao Dec 15, 2020

Choose a reason for hiding this comment

caesarxuchao Dec 15, 2020

Choose a reason for hiding this comment

caesarxuchao commented Dec 15, 2020

fejta-bot commented Mar 15, 2021

roycaihw commented Mar 16, 2021

fejta-bot commented Jun 14, 2021

fejta-bot commented Jul 14, 2021

k8s-triage-robot commented Aug 13, 2021

k8s-ci-robot commented Aug 13, 2021

roycaihw commented Dec 7, 2020 •

edited

Loading