ArgoCD doesn't respect application.namespaces

[mrysavy]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I tried to use application namespaces (#9755) but there are problems in multi-argocd-instances environments. Problem is that neither application-controller nor server respects specified application namespaces.

There are two problems with argocd instance with application namespaces:

1. server displays Application resources from all namespaces (not only from the argocd home namespace and allowed namespaces using ARGOCD_APPLICATION_NAMESPACES env or --application-namespaces argument)
   note: argocd server UI displays unwanted Applications only when there is a change on them and refreshing UI these ones go away, but this is very confusing
2. argocd CLI/GUI can access any Application resource in any namespace (independent of "application namespaces" setting)
3. application-controller decorate foreign Application resources belonging to another argocd instance with UnknownError with message like application 'helm-guestbook' in namespace 'argocd1' is not permitted to use project 'default'

To Reproduce

1. deploy two argocd instances - argocd1 without "application namespaces" and argocd2 with default as "application namespace"

helm install argocd1 argo/argo-cd -n argocd1 --create-namespace --set "server.config.application\.instanceLabelKey=argocd.argoproj.io/instance-1" --set server.service.type=NodePort --set server.service.nodePortHttp=31080 --set server.service.nodePortHttps=31443
helm install argocd2 argo/argo-cd -n argocd2 --create-namespace --set "server.config.application\.instanceLabelKey=argocd.argoproj.io/instance-2" --set server.service.type=NodePort --set server.service.nodePortHttp=32080 --set server.service.nodePortHttps=32443 --set crds.install=false --set "configs.params.application\.namespaces=default"

2. allow default namespace for argocd2 AppProject default

kubectl patch -n argocd2 appproject/default --type merge -p '{"spec": {"sourceNamespaces": ["default"]}}'

3. create namespaces for applications

kubectl create namespace helm-guestbook-1
kubectl create namespace helm-guestbook-2
kubectl create namespace helm-guestbook-default

4. create Application resources in namespaces: argocd1 (for argocd1 instance), argocd2 (for argocd2 instance) and default (also for argocd2 instance as an application namespace)

cat <<EOF | kubectl create -f-
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: helm-guestbook
  namespace: argocd1
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: helm-guestbook-1
    server: https://kubernetes.default.svc
  project: default
  source:
    path: helm-guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: master
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: helm-guestbook
  namespace: argocd2
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: helm-guestbook-2
    server: https://kubernetes.default.svc
  project: default
  source:
    path: helm-guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: master
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: helm-guestbook
  namespace: default
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: helm-guestbook-default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: helm-guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: master
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

Now all three applications is accessible from both instances (tried with both argocd contexts):

# switch argocd CLI context to argocd1
argocd app get argocd1/helm-guestbook
argocd app get argocd2/helm-guestbook
argocd app get default/helm-guestbook
# switch argocd CLI context to argocd2
argocd app get argocd1/helm-guestbook
argocd app get argocd2/helm-guestbook
argocd app get default/helm-guestbook

Application argocd1/helm-guestbook (which belongs to argocd1 instance) has error from argocd2 application controller (which has nothing to do with it).

$ argocd app get argocd1/helm-guestbook | grep -A1 CONDITION

CONDITION     MESSAGE                                                                                        LAST TRANSITION
UnknownError  application 'helm-guestbook' in namespace 'argocd1' is not permitted to use project 'default'  2022-10-30 23:37:20 +0100 CET

Stopping argocd2 application-controller and refreshing the application this error disappears

Expected behavior

I would expect following:

• argocd CLI displays only "own" application, i.e. Application resources only from its "home" namespace or "application namespaces"
• argocd-server GUI displays only "own" application, i.e. Application resources only from its "home" namespace or "application namespaces"
• argocd-application-controller ignores foreign Application resources, i.e. application from namespaces other than "home" namespace or "application namespaces"

Version

argocd: v2.5.0+b895da4.dirty
  BuildDate: 2022-10-25T19:12:35Z
  GitCommit: b895da457791d56f01522796a8c3cd0f583d5d91
  GitTreeState: dirty
  GoVersion: go1.19.2
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.5.0+b895da4
  BuildDate: 2022-10-25T14:40:01Z
  GitCommit: b895da457791d56f01522796a8c3cd0f583d5d91
  GitTreeState: clean
  GoVersion: go1.18.7
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.1+g9f88ccb
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

[mrysavy]

No, it doesn't. Mentioned issue is about the namespace of resources from Application's source but my issue is about the namespace which the Application.argoproj.io resides in - the new feature comming from this pull request: #9755 (I edited the issue while ago). út 1. 11. 2022 v 15:26 odesílatel Ryan Umstead ***@***.***> napsal:

…

Duplicate of #8386 <#8386>? Relevant docs <https://github.com/argoproj/argo-cd/blob/9353328eb8ae7357538146269d41aa692d99c4cf/docs/operator-manual/application.yaml#L123> — Reply to this email directly, view it on GitHub <#11116 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEHL4DQG7M3UD3GFRZ4XIDWGESB5ANCNFSM6AAAAAARSSLXPQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[IxDay]

Got exactly the same bug. I also tried to scope the default appproject to the defined namespaces just in case. It did not work and I am in the same situation as @mrysavy

[oscarsix]

same problem

[willemm]

We are experiencing the same problem.

I'm not that well at home in the argocd source, but my best guess is that this bit of code is setting the status on the argocd.argoproj.io/applcation that it shouldn't touch:

argo-cd/controller/appcontroller.go

Line 1870 in 82f006c

if _, err := ctrl.getAppProj(app); err != nil {

I traced the error message I'm seeing in the status.conditions field from the source in the code, and this place seems to just set the condition field but not the health field, which another bit of code does do, so that's my guess.

Doesn't explain why it's also popping up in the UI. I'm guessing that's because the update process somehow triggers a push to the UI (stream/applications api?).