From e16c2d1b99fed379746af44defd7d3ccff789398 Mon Sep 17 00:00:00 2001 From: Huamin Chen Date: Tue, 13 Mar 2018 21:52:27 +0000 Subject: [PATCH 1/5] document pvc finalizer issue during 1.10 -> 1.9 downgrade Signed-off-by: Huamin Chen --- .../pvc-finalizer-downgrade-issue.md | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 docs/reference/pvc-finalizer-downgrade-issue.md diff --git a/docs/reference/pvc-finalizer-downgrade-issue.md b/docs/reference/pvc-finalizer-downgrade-issue.md new file mode 100644 index 0000000000000..7e82e63231fcf --- /dev/null +++ b/docs/reference/pvc-finalizer-downgrade-issue.md @@ -0,0 +1,32 @@ +* TOC +{:toc} + +--- +title: Kubernetes Downgrade issue from 1.10 to 1.9 due to PV/PVC Protection +--- + +## PV/PVC Protection in Kubernetes 1.10 + +When enabled, [PV/PVC Protection](docs/admin/admission-controllers.md#storage-object-in-use-protection-beta) prevents PV/PVC from being removed when the finalizers are removed. + +## Downgrading issue + +After downgrading from Kubernetes 1.10 to 1.9, PV/PVCs that are created in Kubernetes 1.10 with PVC Protection cannot be removed. + +## Workaround + +Currently PV/PVC finalizers have to be manually removed so PV/PVC can be removed after downgrading to Kubernetes 1.9. + +Before downgrading to Kubernetes 1.9, disable `StorageObjectInUseProtection` plugin and restart admission controller. + +Then patch PV/PVC (e.g. `pv1`) using the following command: + +```bash +kubectl patch pv pv1 --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' +``` + +then verify the finalizers are removed: +```bash +kubectl get pv pv1 -o yaml |grep finalizer +# (result should be empty) +``` From c94778b653a86ed8bda7b4015892850aad52ed7a Mon Sep 17 00:00:00 2001 From: Huamin Chen Date: Wed, 14 Mar 2018 19:31:30 +0000 Subject: [PATCH 2/5] feedback addressed Signed-off-by: Huamin Chen --- .../pvc-finalizer-downgrade-issue.md | 39 ++++++++++++------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/docs/reference/pvc-finalizer-downgrade-issue.md b/docs/reference/pvc-finalizer-downgrade-issue.md index 7e82e63231fcf..7606d9471e2f9 100644 --- a/docs/reference/pvc-finalizer-downgrade-issue.md +++ b/docs/reference/pvc-finalizer-downgrade-issue.md @@ -2,31 +2,42 @@ {:toc} --- -title: Kubernetes Downgrade issue from 1.10 to 1.9 due to PV/PVC Protection +title: Downgrading issue from Kubernetes 1.10 to 1.9 due to StorageObjectInUseProtection --- -## PV/PVC Protection in Kubernetes 1.10 +## Storage Object in Use Protection in Kubernetes 1.10 -When enabled, [PV/PVC Protection](docs/admin/admission-controllers.md#storage-object-in-use-protection-beta) prevents PV/PVC from being removed when the finalizers are removed. +When enabled, [Storage Object in Use Protection](/docs/admin/admission-controllers.md#storage-object-in-use-protection-beta) prevents PV/PVC from being removed when the finalizers are still present. ## Downgrading issue -After downgrading from Kubernetes 1.10 to 1.9, PV/PVCs that are created in Kubernetes 1.10 with PVC Protection cannot be removed. +After downgrading from Kubernetes 1.10 to 1.9, PV/PVCs that contain finalizers cannot be removed until their finalizers are removed. ## Workaround -Currently PV/PVC finalizers have to be manually removed so PV/PVC can be removed after downgrading to Kubernetes 1.9. +Currently the pv-protection and pvc-protection finalizers must be removed manually before you downgrade so that PVs and PVCs can be removed after you downgrade to version 1.9. Here's what to do: + + +Before downgrading to Kubernetes 1.9, disable `StorageObjectInUseProtection` plugin by using apiserver command line option `--disable-admission-plugins=StorageObjectInUseProtection` and restart apiserver: + +- if apiserver is managed by systemd, stop it using `systemd stop `. Modify apiserver service unit service file, reload the service, and start it using `systemd start` command +- if apiserver is started through command line, kill the process and run the command again with the above command line option -Before downgrading to Kubernetes 1.9, disable `StorageObjectInUseProtection` plugin and restart admission controller. Then patch PV/PVC (e.g. `pv1`) using the following command: -```bash -kubectl patch pv pv1 --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' -``` +- Patch the PV or PVC, as in the following command, where `pv1` is the name of the PV to patch: + + ```bash + kubectl patch pv pv1 --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' + ```` + +- Verify the finalizers are removed: + + ```bash + kubectl get pv pv1 -o yaml |grep finalizer + ``` + + The result should be empty. -then verify the finalizers are removed: -```bash -kubectl get pv pv1 -o yaml |grep finalizer -# (result should be empty) -``` +- You can now safely downgrade to version 1.9. From a1edd39aeb5f4399f281495983afc48df1155e52 Mon Sep 17 00:00:00 2001 From: Huamin Chen Date: Fri, 16 Mar 2018 18:45:18 +0000 Subject: [PATCH 3/5] change the flow: allow downgrade first but deal with finalizer after downgrade Signed-off-by: Huamin Chen --- docs/reference/pvc-finalizer-downgrade-issue.md | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/docs/reference/pvc-finalizer-downgrade-issue.md b/docs/reference/pvc-finalizer-downgrade-issue.md index 7606d9471e2f9..c2340272e6bd8 100644 --- a/docs/reference/pvc-finalizer-downgrade-issue.md +++ b/docs/reference/pvc-finalizer-downgrade-issue.md @@ -11,20 +11,15 @@ When enabled, [Storage Object in Use Protection](/docs/admin/admission-controlle ## Downgrading issue -After downgrading from Kubernetes 1.10 to 1.9, PV/PVCs that contain finalizers cannot be removed until their finalizers are removed. +In Kubernetes 1.9, `PVCProtection` feature is turned off by default. After downgrading from Kubernetes 1.10 to 1.9, PV/PVCs that contain finalizers cannot be removed until their finalizers are removed. ## Workaround -Currently the pv-protection and pvc-protection finalizers must be removed manually before you downgrade so that PVs and PVCs can be removed after you downgrade to version 1.9. Here's what to do: +Currently the pv-protection and pvc-protection finalizers must be removed manually after you downgrade so that PVs and PVCs can be removed. +If `PVCProtection` feature is turned off in Kubernetes 1.9, here is what do do: -Before downgrading to Kubernetes 1.9, disable `StorageObjectInUseProtection` plugin by using apiserver command line option `--disable-admission-plugins=StorageObjectInUseProtection` and restart apiserver: - -- if apiserver is managed by systemd, stop it using `systemd stop `. Modify apiserver service unit service file, reload the service, and start it using `systemd start` command -- if apiserver is started through command line, kill the process and run the command again with the above command line option - - -Then patch PV/PVC (e.g. `pv1`) using the following command: +Patch PV/PVC (e.g. `pv1`) using the following command: - Patch the PV or PVC, as in the following command, where `pv1` is the name of the PV to patch: @@ -40,4 +35,4 @@ Then patch PV/PVC (e.g. `pv1`) using the following command: The result should be empty. -- You can now safely downgrade to version 1.9. +- Then the PV and PVC can be removed. \ No newline at end of file From 47f31479bac28927b4423297b84cfb54db37ccc9 Mon Sep 17 00:00:00 2001 From: Huamin Chen Date: Mon, 19 Mar 2018 14:49:15 +0000 Subject: [PATCH 4/5] be more specific about finalizer element to remove Signed-off-by: Huamin Chen --- docs/reference/pvc-finalizer-downgrade-issue.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/reference/pvc-finalizer-downgrade-issue.md b/docs/reference/pvc-finalizer-downgrade-issue.md index c2340272e6bd8..06a8ca240951b 100644 --- a/docs/reference/pvc-finalizer-downgrade-issue.md +++ b/docs/reference/pvc-finalizer-downgrade-issue.md @@ -19,18 +19,20 @@ Currently the pv-protection and pvc-protection finalizers must be removed manual If `PVCProtection` feature is turned off in Kubernetes 1.9, here is what do do: -Patch PV/PVC (e.g. `pv1`) using the following command: +Patch PV/PVC (e.g. `pv1` and `pvc1`) using the following command: -- Patch the PV or PVC, as in the following command, where `pv1` is the name of the PV to patch: +- Patch the PV or PVC, as in the following command, where `pv1` is the name of the PV to patch, and `pvc1` is the name of the PVC to patch: ```bash - kubectl patch pv pv1 --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' + kubectl patch pv pv1 --type=json -p='[{"op": "remove", "path": "/metadata/finalizers", "value": "kubernetes.io/pv-protection"}]' + kubectl patch pvc pvc1 --type=json -p='[{"op": "remove", "path": "/metadata/finalizers", "value": "kubernetes.io/pvc-protection"}]' ```` - Verify the finalizers are removed: ```bash kubectl get pv pv1 -o yaml |grep finalizer + kubectl get pvc pvc1 -o yaml |grep finalizer ``` The result should be empty. From df9d49f544bde9067478881cf655ab4d1095e6e3 Mon Sep 17 00:00:00 2001 From: Jennifer Rondeau Date: Wed, 21 Mar 2018 14:46:07 -0700 Subject: [PATCH 5/5] Update pvc-finalizer-downgrade-issue.md Copyedits only to bump build again --- docs/reference/pvc-finalizer-downgrade-issue.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/pvc-finalizer-downgrade-issue.md b/docs/reference/pvc-finalizer-downgrade-issue.md index 06a8ca240951b..f61a1eafcc127 100644 --- a/docs/reference/pvc-finalizer-downgrade-issue.md +++ b/docs/reference/pvc-finalizer-downgrade-issue.md @@ -7,7 +7,7 @@ title: Downgrading issue from Kubernetes 1.10 to 1.9 due to StorageObjectInUsePr ## Storage Object in Use Protection in Kubernetes 1.10 -When enabled, [Storage Object in Use Protection](/docs/admin/admission-controllers.md#storage-object-in-use-protection-beta) prevents PV/PVC from being removed when the finalizers are still present. +When enabled, [Storage Object in Use Protection](/docs/admin/admission-controllers.md#storage-object-in-use-protection-beta) prevents PersistentVolume and PersistentVolumeClaim objects from being removed if the finalizers are still present. ## Downgrading issue @@ -17,7 +17,7 @@ In Kubernetes 1.9, `PVCProtection` feature is turned off by default. After downg Currently the pv-protection and pvc-protection finalizers must be removed manually after you downgrade so that PVs and PVCs can be removed. -If `PVCProtection` feature is turned off in Kubernetes 1.9, here is what do do: +If the `PVCProtection` feature is turned off in Kubernetes 1.9, here is what do do: Patch PV/PVC (e.g. `pv1` and `pvc1`) using the following command: @@ -37,4 +37,4 @@ Patch PV/PVC (e.g. `pv1` and `pvc1`) using the following command: The result should be empty. -- Then the PV and PVC can be removed. \ No newline at end of file +- Then the PV and PVC can be removed.