ks (0.11) complains that resource was modified when updating Kubeflow

[jlewi]

I have an existing Kubeflow deployment when I try updating it I get

ERROR handle object: can't update deployments kubeflow.tf-job-operator-v1alpha2: Operation cannot be fulfilled on deployments.extensions "tf-job-operator-v1alpha2": the object has been modified; please apply your changes to the latest version and try again 

This appears to have started with ks 0.11

My version

ks version
ksonnet version: 0.11.0
jsonnet version: v0.10.0
client-go version: 

ks 0.11 uses merge patching and I suspect that might be the issue?

[jlewi]

The following workaround worked

ks show jlewi-kubeflow -c kubeflow-core | kubectl apply -f 

[jlewi]

@bryanl is this an issue with ksonnet or with K8s?

[pdmack]

Same thing:

ERROR handle object: can't update deployments kubeflow.issue-summarization-ui: Operation cannot be fulfilled on deployments.apps "issue-summarization-ui": the object has been modified; please apply your changes to the latest version and try again

[bryanl]

I'm going to dig in and investigate. I will report back here.

[bajacondor]

I'm seeing the same with deployments.extensions when modifying the image property.
ERROR handle object: can't update deployments cicd.simple-app: Operation cannot be fulfilled on deployments.extensions "simple-app": the object has been modified; please apply your changes to the latest version and try again

[bryanl]

I've researched the cause of this error and we will need to add a retries to our apply action. While not optimal, it's safe to return ks apply again.

[jlewi]

Thanks @bryanl

[jlewi]

I hit a similar issue trying to update a service

ERROR handle object: merging object with existing state: Service "envoy" is invalid: [metadata.resourceVersion: Invalid value: "": must be specified for an update, spec.clusterIP: Invalid value: "": field is immutable] 

@bryanl Is this really an issue with optimistic locking? It seems like ks apply is trying to set the value to the empty string which doesn't work.

[jlewi]

I get the same error if I pipe the output of ks show to ks apply

ks show default -c iap-ingress | kubectl apply -f -
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
ingress.extensions "envoy-ingress" configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
certificate.certmanager.k8s.io "envoy-ingress-tls" configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
serviceaccount "envoy" configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
clusterrolebinding.rbac.authorization.k8s.io "envoy" configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
clusterrole.rbac.authorization.k8s.io "envoy" configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.extensions "envoy" configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.extensions "iap-enabler" configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap "envoy-config" configured
service "whoami-app" created
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.extensions "whoami-app" configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
cloudendpoint.ctl.isla.solutions "jlewi-kubeflow" configured
The Service "envoy" is invalid: 
* metadata.resourceVersion: Invalid value: "": must be specified for an update
* spec.clusterIP: Invalid value: "": field is immutable

[jlewi]

I noticed while updating the dev.kubeflow.org that problem does seem to be related to retries.

I kept running the command

ks apply default -c kubeflow-core

Each time I ran it it complained for a different object and I got an error like

ERROR handle object: can't update deployments kubeflow.tf-job-operator-v1alpha2: Operation cannot be fulfilled on deployments.extensions "tf-job-operator-v1alpha2": the object has been modified; please apply your changes to the latest version and try again 

But each time I reran the command it got a little bit further and eventually it succeeded.

[jlewi]

It looks like the error with updating the service

ERROR handle object: merging object with existing state: Service "envoy" is invalid: [metadata.resourceVersion: Invalid value: "": must be specified for an update, spec.clusterIP: Invalid value: "": field is immutable] 

Is a different error and now one that is fixed by retrying.

[jlewi]

Opened #1145 for the problems with node port services because that seems to be a different issue.

[jlewi]

/lifecycle stale