argocd app wait --selector selects apps that are being deleted; will then hang or throw "not found" error once the app is gone

[michaelajr]

Describe the bug

When using a label selector with argocd app wait, it seem to select apps that are also in the process of being deleted.

argocd app wait -l some-label=some-value --timeout 1200

Depending on timing, one of two things will happen. If the wait starts waiting on an app that is that is being deleted, it will start waiting- the app gets fully removed - and it will timeout waiting. Or... the app gets deleted before it starts waiting on it, and it throws an error saying some-app not found.

To Reproduce

• Create an app of apps with child apps (we have 25 or so)
• Sync so they are all in the cluster and working
• Remove a child app from the bootstrap app
• Run sync. You will see the app start to get pruned. The sync will exit.
• As the app is being removed...run wait with a selector that selects all the child apps
• You should see the behavior mentioned here.

Expected behavior

I would expect argocd app wait -l some-label=some-value not to select apps that are being deleted.

Version

argocd: v1.8.3+0f9c684.dirty
  BuildDate: 2021-01-23T03:45:34Z
  GitCommit: 0f9c68427882bf4633d395cbfcd7c9271795fd9b
  GitTreeState: dirty
  GoVersion: go1.15.7
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.8.4+28aea3d
  BuildDate: 2021-02-05T17:56:06Z
  GitCommit: 28aea3dfdede00443b52cc584814d80e8f896200
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.8.1 2020-07-16T00:58:46Z
  Helm Version: v3.4.1+gc4e7485
  Kubectl Version: v1.17.8
  Jsonnet Version: v0.17.0