Unable to remove finalizer on custom resource

[jasmdk]

When implementing a controller that registers a finalizer, then it is supposed to do any necessary cleanup when it gets a MODIFIED event where the deletionTimestamp is set - see https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-deletion
When I remove my finalizer, I need to delete it from the list of finalizers and update the resource and only when all finalizers are removed, the resource is actually deleted.

If I make my change on the current resource and invoke 'createOrReplace' then I get an exception with 'resourceVersion should not be set on objects to be created' because the createOrReplace (io.fabric8.kubernetes.client.dsl.base.BaseOperation#createOrReplace) first tries to reload the object from the server, but as it is being deleted it gets nothing and therefore tries to create the object.

resource.getMetadata().getFinalizers().remove(FINALIZER_NAME);
customResourceClient.crClient.createOrReplace(resource);

2020-06-18 14:39:53,782 ERROR [com.sys.syn.ope.pos.PostgresDBReconsiler] (pool-4-thread-1) Unexpected exception: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.96.0.1/apis/synergy.systematic.com/v1/namespaces/default/postgresdbs. Message: resourceVersion should not be set on objects to be created. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=resourceVersion should not be set on objects to be created, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:251)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:848)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:368)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:429)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:426)
        at com.systematic.synergy.operator.postgresdb.PostgresDBReconsiler.handleFinalizer(PostgresDBReconsiler.java:97)
        at com.systematic.synergy.operator.postgresdb.PostgresDBReconsiler.handleEvent(PostgresDBReconsiler.java:69)
        at com.systematic.synergy.operator.postgresdb.PostgresDBResourceCache$1.lambda$eventReceived$0(PostgresDBResourceCache.java:70)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

For finalizers I need some way of updating (or patching) the resource withouth first reloading it from server. I also tried listening for the DELETE event in a mutating webhook, but on a delete I am not allowed to patch the resource from the webhook when it is being deleted.

[manusa]

Relates to #2292

[rohanKanojia]

I think this issue should get fixed by #2372 (since we not do create() first and then get()/replace() afterwards as a fallback).

I tried it out with a simple CronTab custom resource:

~/go/src/github.com/fabric8io/kubernetes-client : $ kubectl get crontabs
NAME               AGE
my-third-crontab   6d19h
~/go/src/github.com/fabric8io/kubernetes-client : $ kubectl get crontab -o yaml
apiVersion: v1
items:
- apiVersion: stable.example.com/v1
  kind: CronTab
  metadata:
    creationTimestamp: "2020-07-24T13:48:32Z"
    finalizers:
    - crontabs.stable.example.com
    - demo.fabric8.io
    generation: 1
    managedFields:
    - apiVersion: stable.example.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"crontabs.stable.example.com": {}
            v:"demo.fabric8.io": {}
        f:spec:
          .: {}
          f:cronSpec: {}
          f:image: {}
          f:replicas: {}
      manager: okhttp
      operation: Update
      time: "2020-07-31T09:19:40Z"
    name: my-third-crontab
    namespace: default
    resourceVersion: "222504"
    selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-third-crontab
    uid: bd75891f-767a-4f16-9977-82e1be2d73fb
  spec:
    cronSpec: '* * * * */6'
    image: my-third-cron-image
    replicas: 5
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I tried removing one finalized "demo.fabric8.io" using createOrReplace():

public class CustomResourceCreateOrReplace {
    public static void main(String[] args) {
        try (KubernetesClient client = new DefaultKubernetesClient()) {
            CustomResourceDefinitionContext context = new CustomResourceDefinitionContext.Builder()
                    .withGroup("stable.example.com")
                    .withVersion("v1")
                    .withPlural("crontabs")
                    .withScope("Namespaced")
                    .withName("crontabs.stable.example.com")
                    .build();

            MixedOperation<CronTab, CronTabList, DoneableCronTab, Resource<CronTab, DoneableCronTab>> cronTabClient =
                    client.customResources(context, CronTab.class, CronTabList.class, DoneableCronTab.class);

            CronTab ct = getCronTab();

            cronTabClient.inNamespace("default").createOrReplace(ct);
        }
    }

    private static CronTab getCronTab() {
        CronTab cronTab = new CronTab();
        cronTab.setMetadata(new ObjectMetaBuilder()
                .withFinalizers(Collections.singletonList("crontabs.stable.example.com"))
                .withName("my-third-crontab")
                .build());
        CronTabSpec cronTabSpec = new CronTabSpec();
        cronTabSpec.setCronSpec("* * * * */6");
        cronTabSpec.setImage("my-third-cron-image");
        cronTabSpec.setReplicas(5);
        cronTab.setSpec(cronTabSpec);

        return cronTab;
    }
}

When I ran this code, I was able to see finalized being removed:

~/go/src/github.com/fabric8io/kubernetes-client : $ kubectl get crontab -o yaml
apiVersion: v1
items:
- apiVersion: stable.example.com/v1
  kind: CronTab
  metadata:
    creationTimestamp: "2020-07-24T13:48:32Z"
    finalizers:
    - crontabs.stable.example.com
    generation: 1
    managedFields:
    - apiVersion: stable.example.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"crontabs.stable.example.com": {}
        f:spec:
          .: {}
          f:cronSpec: {}
          f:image: {}
          f:replicas: {}
      manager: okhttp
      operation: Update
      time: "2020-07-31T09:19:40Z"
    name: my-third-crontab
    namespace: default
    resourceVersion: "222517"
    selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-third-crontab
    uid: bd75891f-767a-4f16-9977-82e1be2d73fb
  spec:
    cronSpec: '* * * * */6'
    image: my-third-cron-image
    replicas: 5
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[rohanKanojia]

Closing this issue. But feel free to reopen in case you face any problems :-)