Credential was deleted due to "Failed to update online status"

[qibobo]

When there are multiple CRD consume the Binding CR and they will fetch the binding secret and save them in another secret. So when the following error occurred, the ICO will delete and regenerate IBM service credential, which make the consumers' secret stale.

{"level":"info","ts":1600754662.29132,"logger":"binding","msg":"Failed to update online status, will delete external resource ","code-engine-cos-codeengine-credential":"Operation cannot be fulfilled on bindings.ibmcloud.ibm.com \"code-engine-cos-codeengine-credential\": the object has been modified; please apply your changes to the latest version and try again"}

I believe the update error happens inside https://github.com/IBM/cloud-operators/blob/master/controllers/binding_controller.go#L340.

How about using client-go RetryOnConflict https://github.com/kubernetes/client-go/blob/master/util/retry/util.go#L68-L105 to avoid deleteCredential when it is a conflict error?

[cdlliuy]

@JohnStarich How about using client-go RetryOnConflict https://github.com/kubernetes/client-go/blob/master/util/retry/util.go#L68-L105 to avoid deleteCredential when it is a conflict error?

[JohnStarich]

Agreed, deleting the resource should be avoided where we could do an update instead. I think we should also use ctrl.CreateOrUpdate, which automatically creates the resource if it does not exist, or mutates and updates if it does.

If you would like to open a PR, we can review 👍

[JohnStarich]

Fixed in #226 and release on GitHub in v1.0.1