Helm chart v2.0.0-rc2 fails to install on fresh k8s 1.18.8 #88

rhummelmose · 2020-10-26T11:10:44Z

Attempting to install the chart fails with error

➜  duah-infrastructure git:(master) ✗ helm install keda kedacore/keda --version 2.0.0-rc2 --namespace keda       
Error: rendered manifests contain a resource that already exists. Unable to continue with install: CustomResourceDefinition "scaledjobs.keda.sh" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "keda"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "keda"

Expected Behavior

It is able to install on a fresh cluster

Actual Behavior

It does not install

Steps to Reproduce the Problem

Attempt to install

Specifications

KEDA Version: 2.0.0-rc2
Platform & Version: Azure, AKS, fresh as of 26/10/2020
Kubernetes Version: 1.18.8

tomkerkhove · 2020-10-26T12:07:24Z

I'll take a look at this and have created #87 to avoid this in the future

RemiGaudin · 2020-10-27T14:21:15Z

Interestingly I have the same issue with version 2.0.0-rc2, but it's working fine with version 2.0.0-rc

tomkerkhove · 2020-10-27T18:53:16Z

@ahmelsayed do you have bandwidth to help here?

jeffhollan · 2020-10-29T16:15:49Z

Discussed in standup - hoping to get rid of the problem. @tomkerkhove shipped 2.0.0-rc3 works now, but it has helm 2. Need to fix so 2.0.0 can use helm 3 and not support helm 2.

@TsuyoshiUshio / @ahmelsayed let me know if you can help look at this. I may be able to spend some cycles tonight as well

arschles · 2020-10-29T16:17:01Z

I can give a hand here as well, will check it out later today

tomkerkhove · 2020-10-29T16:41:21Z

Thanks @arschles!

rhummelmose · 2020-10-29T21:03:25Z

Awesome guys. @arschles let me know if you can't find the time and I'll try to fix it during the weekend.

jeffhollan · 2020-11-03T17:07:36Z

Can confirm that 2.0.0-rc3 works, but it isn't published to the index so had to clone. Let me clean it up a little and publish

kainwei · 2021-05-11T09:46:19Z

I have met this problem today as well. Difference is I was updating keda-2.2.0 to keda-2.2.2.
Error message:

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. 
Unable to continue with update: CustomResourceDefinition "scaledjobs.keda.sh" 
in namespace "" exists and cannot be imported into the current release: 
invalid ownership metadata; label validation error: 
missing key "app.kubernetes.io/managed-by": must be set to "Helm"; 
annotation validation error: missing key "meta.helm.sh/release-name": must be set to "keda"; 
annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "keda"

I had searched on Internet, I found that is because the newer version's chart template crd(CustomResourceDefinition) already exist(highly likely from your installed versions before) in your k8s api resources and they had a conflict.
Actually, I don't know why this happened, because my another cluster has no this issue. So I searched for this again, I found a way to delete the api resources, then reinstall the keda to solve this problem.

Below is my steps what I have done(I used helm 3):

show your api resources and conflict resources:

$  kubectl api-resources|grep scaled
scaledjobs                                sj               keda.sh                        true         ScaledJob
scaledobjects                             so               keda.sh                        true         ScaledObject

list the template for each version(From this we can know keda 2.2.2 use name: scaledobjects.keda.sh(this is the conflict resource) which 2.2.0 not, that is why I can update my lower version before to 2.2.0 but cannot update the keda to 2.2.1 and 2.2.2. we can also know the difference between 2 versions, like the CustomResourceDefinition ):

$ helm template keda kedacore/keda --version 2.2.2|grep -E 'scaled|CustomResourceDefinition'
kind: CustomResourceDefinition
# Source: keda/templates/03-crd-scaledjobs.keda.sh.yaml
kind: CustomResourceDefinition
 name: scaledjobs.keda.sh
   plural: scaledjobs
   singular: scaledjob
       description: ScaledJob is the Schema for the scaledjobs API
# Source: keda/templates/04-crd-scaledobjects.keda.sh.yaml
kind: CustomResourceDefinition
 name: scaledobjects.keda.sh
   plural: scaledobjects
   singular: scaledobject
kind: CustomResourceDefinition
 - scaledjobs
 - scaledjobs/status
 - scaledobjects
 - scaledobjects/finalizers
 - scaledobjects/status
$ helm template keda kedacore/keda --version 2.2.0|grep -E 'scaled|CustomResourceDefinition'
 - scaledjobs
 - scaledjobs/status
 - scaledobjects
 - scaledobjects/finalizers
 - scaledobjects/status
$

Now it is our solution for this problem, delete the api resource and reinstall the keda:
uninstall the current version

$ helm  uninstall keda -n keda 
release "keda" uninstalled
$

now, conflict resource still here:

$  kubectl api-resources|grep scaled
scaledjobs                                sj               keda.sh                        true         ScaledJob
scaledobjects                             so               keda.sh                        true         ScaledObject
$

Delete them by kubectl delete:

$ helm template keda kedacore/keda --version 2.2.1 | kubectl   delete -f -
customresourcedefinition.apiextensions.k8s.io "clustertriggerauthentications.keda.sh" deleted
customresourcedefinition.apiextensions.k8s.io "scaledjobs.keda.sh" deleted
customresourcedefinition.apiextensions.k8s.io "triggerauthentications.keda.sh" deleted
Error from server (NotFound): error when deleting "STDIN": serviceaccounts "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "scaledobjects.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "keda-operator-external-metrics-reader" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator:system:auth-delegator" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator-hpa-controller-external-metrics" not found
Error from server (NotFound): error when deleting "STDIN": rolebindings.rbac.authorization.k8s.io "keda-operator-auth-reader" not found
Error from server (NotFound): error when deleting "STDIN": services "keda-operator-metrics-apiserver" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "keda-operator-metrics-apiserver" not found
Error from server (NotFound): error when deleting "STDIN": apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" not found
$

Run it again you would find all of the template resource was deleted:

$ helm template keda kedacore/keda --version 2.2.1 | kubectl  delete -f -
Error from server (NotFound): error when deleting "STDIN": serviceaccounts "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "clustertriggerauthentications.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "scaledjobs.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "scaledobjects.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "triggerauthentications.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "keda-operator-external-metrics-reader" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator:system:auth-delegator" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator-hpa-controller-external-metrics" not found
Error from server (NotFound): error when deleting "STDIN": rolebindings.rbac.authorization.k8s.io "keda-operator-auth-reader" not found
Error from server (NotFound): error when deleting "STDIN": services "keda-operator-metrics-apiserver" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "keda-operator-metrics-apiserver" not found
Error from server (NotFound): error when deleting "STDIN": apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" not found
$

Try use kubectl api-resources to check, we found it's clear:

$ kubectl api-resources |grep scaled
$

4 reinstall and successful:

$ helm install keda kedacore/keda -n keda
NAME: keda
LAST DEPLOYED: Tue May 11 11:33:38 2021
NAMESPACE: keda
STATUS: deployed
REVISION: 1
TEST SUITE: None
$ helm list -n keda
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART     	APP VERSION
keda	keda     	1       	2021-05-11 11:33:38.133763248 +0000 UTC	deployed	keda-2.2.2	2.2.0      
$ kubectl get pods -n keda
NAME                                              READY   STATUS    RESTARTS   AGE
keda-operator-546fc6c546-wfh98                    1/1     Running   0          7m24s
keda-operator-metrics-apiserver-df57db588-sl7m9   1/1     Running   0          7m24s
$

Problem I have met in this process:

each time the conflict resource might be not same, you need to try different version's template to find where is it:
like:
...
helm template keda kedacore/keda --version 2.2.0 | kubectl delete -f -
helm template keda kedacore/keda --version 2.2.1 | kubectl delete -f -
helm template keda kedacore/keda --version 2.2.2 | kubectl delete -f -
...
sometimes 'kubectl delete' shows that the resource was deleted, but it still there! I had tried to install the version which I can install without error(for my case it was 2.2.0) and uninstall process by '--no-hooks' (command: helm3 uninstall keda -n keda --no-hooks), then it worked. I don't really now the reason, because in official website, there is only one explaination: "prevent hooks from running during uninstallation"

About how to fix the conflict resource in helm, I referred this article:
https://stackoverflow.com/questions/59443834/helm-3-install-for-resouces-that-exist

At last, hope my article can be helpful to whom met this problem!

tomkerkhove · 2021-05-13T12:38:59Z

Thanks for the write-up, if I get it correctly it's just due to Helm 3, right?

kainwei · 2021-05-14T00:09:33Z

I am not sure what will happened in Helm2, but in my case, yes, I used Helm 3.

tomkerkhove self-assigned this Oct 26, 2020

tomkerkhove transferred this issue from kedacore/keda Oct 26, 2020

This was referenced Oct 26, 2020

Deploy Helm chart on Kind cluster #89

Merged

KEDA might break existing deployment on cluster which already has another External Metrics Adapter installed kedacore/keda#470

Open

tomkerkhove mentioned this issue Oct 28, 2020

Re-introduce support for Helm v2.x #91

Merged

tomkerkhove mentioned this issue Nov 3, 2020

Error with helm install on 2.0.0-rc2 #93

Closed

jeffhollan self-assigned this Nov 3, 2020

jeffhollan mentioned this issue Nov 3, 2020

remove helm 2 and index 2.0.0-rc3 #94

Merged

tomkerkhove closed this as completed in #94 Nov 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Helm chart v2.0.0-rc2 fails to install on fresh k8s 1.18.8 #88

Helm chart v2.0.0-rc2 fails to install on fresh k8s 1.18.8 #88

rhummelmose commented Oct 26, 2020

tomkerkhove commented Oct 26, 2020

RemiGaudin commented Oct 27, 2020

tomkerkhove commented Oct 27, 2020

jeffhollan commented Oct 29, 2020

arschles commented Oct 29, 2020 •

edited

Loading

tomkerkhove commented Oct 29, 2020

rhummelmose commented Oct 29, 2020

jeffhollan commented Nov 3, 2020

kainwei commented May 11, 2021 •

edited

Loading

tomkerkhove commented May 13, 2021

kainwei commented May 14, 2021

Helm chart v2.0.0-rc2 fails to install on fresh k8s 1.18.8 #88

Helm chart v2.0.0-rc2 fails to install on fresh k8s 1.18.8 #88

Comments

rhummelmose commented Oct 26, 2020

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

tomkerkhove commented Oct 26, 2020

RemiGaudin commented Oct 27, 2020

tomkerkhove commented Oct 27, 2020

jeffhollan commented Oct 29, 2020

arschles commented Oct 29, 2020 • edited Loading

tomkerkhove commented Oct 29, 2020

rhummelmose commented Oct 29, 2020

jeffhollan commented Nov 3, 2020

kainwei commented May 11, 2021 • edited Loading

tomkerkhove commented May 13, 2021

kainwei commented May 14, 2021

arschles commented Oct 29, 2020 •

edited

Loading

kainwei commented May 11, 2021 •

edited

Loading