PostDelete Hook not work with Application generated from ApplicationSet

[dtrouillet]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When I create an ApplicationSet that create one simple Application (one configmap and one Job as hook with argocd.argoproj.io/hook: PostDelete annotation) the postDelete hook are never called. The finalizer on Application are not be created.

If I create the same Application manually, the finalizer exist.

To Reproduce

Create an ApplicationSet :

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test-damien
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            env: test
            kind: blue
  goTemplate: true
  template:
    metadata:
      name: >-
        test-damien
    spec:
      destination:
        namespace: test-damien
        server: '{{ .server }}'
      ignoreDifferences:
        - group: apps
          jsonPointers:
            - /spec/replicas
          kind: Deployment
      project: default
      revisionHistoryLimit: 10
      sources:
        - path: .
          repoURL: [email protected]:dtrouillet/argocd-postdelete-issue.git
          targetRevision: master
      syncPolicy:
        syncOptions:
          - CreateNamespace=true

The content of the repo test is here : https://github.com/dtrouillet/argocd-postdelete-issue

Expected behavior

I expect an Application with finalizer and PostDelete hook executed when I delete the Application

Screenshots

Version
argocd: v2.10.0+2175939
BuildDate: 2024-02-06T14:28:43Z
GitCommit: 2175939
GitTreeState: clean
GoVersion: go1.21.3
Compiler: gc
Platform: linux/amd64

Logs

Paste any relevant application logs here.

[dtrouillet]

Related to #16595

[sidineyc]

Also hitting this on 2.10.4. Have you found any workarounds @dtrouillet ?

[Jessehuze]

This is blocking our org from moving forward with some key initiatives - would love to see this get more traction.

[dtrouillet]

Also hitting this on 2.10.4. Have you found any workarounds @dtrouillet ?

No unfortunately.

[theoribeiro]

This is blocking our org from moving forward with some key initiatives - would love to see this get more traction.

Same for us. We tried a couple of workarounds but nothing seems to accomplish the same thing as a successful PostDelete hook would. In our use case we have ephemeral environments that are generated by an ApplicationSet based on a PR. We need to clean the environment after the PR is closed.
Can anyone think of a workaround for it?

[joebowbeer]

@alexmt Any leads as to why this is failing?

[hau21um]

Similar story here. We use in our AppSet

      finalizers:
        - resources-finalizer.argocd.argoproj.io

when Application is created manually with PostDelete hook it adds (at least in our case)

        - post-delete-finalizer.argocd.argoproj.io  
        - post-delete-finalizer.argocd.argoproj.io/cleanup  

So combining all 3 finalizers in AppSet template can work as workaround.
Unfortunately we use git generator with automated sync policy and expect that application will be removed once its git folder is removed, but we observe race situation, that

• Application starts refreshing once its git folder is deleted and this refresh ends with Error (as there is nothing to render by CMP plugin from non-existing folder. or CMP plugin is not discovered )
• ApplicationSet triggers deletion of the application.

At the end application is deleted, but the PostDelete hook is not triggered. If I tweak the AppSet to delete the App, but still keeping the app git folder in usable state, PostDelete hook is triggered (with the extra finalizers in AppSet template).

[joebowbeer]

@hau21um Do I understand correctly that you have tried adding finalizers to the ApplicationSet template as a workaround? Like the following?

  template:
    metadata:
      name: 'test-{{name}}'
      finalizers:
      - resources-finalizer.argocd.argoproj.io 
      - post-delete-finalizer.argocd.argoproj.io  
      - post-delete-finalizer.argocd.argoproj.io/cleanup  

Do you think your issue w/CMP is a separate issue?

[hau21um]

@joebowbeer yap adding finalizers as you described partially fixed it for me and if not trying to modify the application source repository together with deletion of application is working fine then. (argocd 2.10.7)

[joebowbeer]

@hau21um I tried the workaround and it doesn't seem to be working for me.

Sorry for lack of detail. It may be replacing one problem with another.

[joebowbeer]

@dtrouillet The fix was released in 2.11.0-rc3

I tested using an appset similar to yours, derived from the dynamic list-generator example in the docs with your hook(s) added as an additional source:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: elements-yaml
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://github.com/joebowbeer/argo-cd.git
              revision: joebowbeer-test-1
              files:
                - path: applicationset/examples/list-generator/list-elementsYaml-example.yaml
          - list:
              elementsYaml: "{{ .key.components | toJson }}"
  template:
    metadata:
      name: "{{.name}}"
    spec:
      project: default
      syncPolicy:
        automated:
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
      sources:
        - chart: "{{.chart}}"
          repoURL: "{{.repoUrl}}"
          targetRevision: "{{.version}}"
          helm:
            releaseName: "{{.releaseName}}"
        - path: .
          repoURL: https://github.com/joebowbeer/argocd-postdelete-issue
          targetRevision: HEAD
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{.namespace}}"

Adding/removing array elements in the list-generator example will add/remove applications.

I modified your hook(s) Jobs to give them names derived from the helm release, and to use a combined BeforeHookCreation,HookSucceeded deletion policy:

metadata:
  name: {{ .Release.Name }}-post-delete
  annotations:
    argocd.argoproj.io/hook: PostDelete
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

[dtrouillet]

Thanks a lot @joebowbeer.
I will try this fix soon.