Resources go out of synch in GitOps deployments

[bsctl]

Bug description

Some resources continuously go out of synch when deploying with a GitOps tool, eg. ArgoCD:

• capsule-ca and capsule-tls secrets
• capsule-mutating-webhook-configuration MutatingWebhookConfiguration
• capsule-validating-webhook-configuration ValidatingWebhookConfiguration

How to reproduce

1. Deploy Capsule Helm chart with ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: capsule
  namespace: argocd
spec:
  destination:
    namespace: capsule-system
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: 'https://github.com/clastix/capsule'
    path: charts/capsule
    targetRevision: v0.0.5
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 3

2. Check the synch status on ArgoCD

Expected behavior

Once deployed resources should not go out of synch

Logs

$ kubectl -n capsule-system get secret capsule-ca -w
NAME         TYPE     DATA   AGE
capsule-ca   Opaque   2      2m16s
capsule-ca   Opaque   0      2m35s
capsule-ca   Opaque   2      4m21s
...

Additional context

• Capsule version: (capsule --version)
• Helm Chart version: (helm list -n capsule-system)
• Kubernetes version: (kubectl version)

[prometherion]

A possible solution for this is to let Capsule create or update the {Mutating,ValidatingWebhookConfiguration} resources.

But we got a problem with the Secret ones, because we're mounting them and, if they don't exist, the capsule-controller-manager Pod cannot be started.

We got two options here:

1️⃣ not mounting Secrets but let the Operator write the content to its roots
2️⃣ creating an additional Job that checks for the Secret resources and, if they're not there, creating it with the desired content; this would require an additional command in the Capsule code-base.

This is happening only in the scenario where a CA and related certificate is not provided, but I think it's common practice.

[MaxFedotov]

2️⃣ creating an additional Job that checks for the Secret resources and, if they're not there, creating it with the desired content; this would require an additional command in the Capsule code-base.

the same way, as we do in https://github.com/clastix/capsule-proxy/blob/master/charts/capsule-proxy/templates/post-install-job.yaml?

[prometherion]

the same way, as we do in https://github.com/clastix/capsule-proxy/blob/master/charts/capsule-proxy/templates/post-install-job.yaml?

I was thinking about this, yes, but in the end, I think we can easily address this with the opinionated ArgoCD options.

I tested the scenario in an ArgoCD environment: the issue is that ArgoCD doesn't play so well with the 3-way merge patch as Helm 3 does, in fact, it complains regarding the outdated Secret resources content and webhooks CA.

We can easily ask ArgoCD to ignore these fields, using the /spec/ignoreDifferences key, as follows.

spec:
  destination:
    namespace: capsule-system
    server: https://kubernetes.default.svc
  ignoreDifferences:
  - group: ""
    jsonPointers:
    - /data
    kind: Secret
    name: capsule-tls
  - group: ""
    jsonPointers:
    - /data
    kind: Secret
    name: capsule-ca
  - group: admissionregistration.k8s.io
    jsonPointers:
    - /webhooks/0/clientConfig/caBundle
    kind: MutatingWebhookConfiguration
    name: capsule-mutating-webhook-configuration
  - group: admissionregistration.k8s.io
    jsonPointers:
    - /webhooks/0/clientConfig/caBundle
    - /webhooks/1/clientConfig/caBundle
    - /webhooks/2/clientConfig/caBundle
    - /webhooks/3/clientConfig/caBundle
    - /webhooks/4/clientConfig/caBundle
    - /webhooks/5/clientConfig/caBundle
    - /webhooks/6/clientConfig/caBundle
    - /webhooks/7/clientConfig/caBundle
    - /webhooks/8/clientConfig/caBundle
    - /webhooks/9/clientConfig/caBundle
    kind: ValidatingWebhookConfiguration
    name: capsule-validating-webhook-configuration
  project: default
  source:
    path: charts/capsule
    repoURL: https://github.com/clastix/capsule
    targetRevision: v0.0.5
  syncPolicy:
    retry:
      limit: 3
    syncOptions:
    - CreateNamespace=true

Honestly, since this is just an issue with ArgoCD, I wouldn't change the code-base to fix a specific problem.

@bsctl feel free to close the issue if it's good enough.

[bsctl]

@prometherion thanks, I think yes. Closing it.