flux diff kustomization behaves incorrectly with sops-managed secrets of type dockerconfigjson

[rombert]

Describe the bug

(follow-up of #2363 (comment) )

I have a kustomization which includes a sops-managed secret of type kubernetes.io/dockerconfigjson . The flux diff command fails to execute with this setup.

Flux server-side status is fine:

$ flux get kustomization dev-fluxctl-diff 
NAME            	READY	MESSAGE                         	REVISION      	SUSPENDED 
dev-fluxctl-diff	True 	Applied revision: master/df608d2	master/df608d2	False

$ flux trace -n dev-fluxctl-diff secret docker.io

Object:        Secret/docker.io
Namespace:     dev-fluxctl-diff
Status:        Managed by Flux
---
Kustomization: dev-fluxctl-diff
Namespace:     flux-system
Path:          ./dev-fluxctl-diff
Revision:      master/df608d2f3e266d33f032e9fd86ffea83883d603b
Status:        Last reconciled at 2022-02-04 16:39:37 +0100 CET
Message:       Applied revision: master/df608d2f3e266d33f032e9fd86ffea83883d603b
---
GitRepository: flux-system
Namespace:     flux-system
URL:           MY-GIT-REPO
Branch:        master
Revision:      master/df608d2f3e266d33f032e9fd86ffea83883d603b
Status:        Last reconciled at 2022-02-02 22:45:57 +0100 CET
Message:       Fetched revision: master/df608d2f3e266d33f032e9fd86ffea83883d603b

Running flux build kustomization works as expected

$ flux build kustomization dev-fluxctl-diff --path ./dev-fluxctl-diff/
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: dev-fluxctl-diff
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: dev-fluxctl-diff
---
apiVersion: v1
data:
  .dockerconfigjson: KipTT1BTKio=
kind: Secret
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: dev-fluxctl-diff
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: docker.io
  namespace: dev-fluxctl-diff
type: kubernetes.io/dockerconfigjson

However, flux diff does not work

$ flux diff kustomization dev-fluxctl-diff --path ./dev-fluxctl-diff/
✗ Secret/dev-fluxctl-diff/docker.io invalid, error: data values must be of type string
► Secret/dev-fluxctl-diff/docker.io deleted

File setup:

$ cat dev-fluxctl-diff/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: dev-fluxctl-diff
resources:
  - namespace.yaml
  - ../shared/
$ cat dev-fluxctl-diff/namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: dev-fluxctl-diff
$ cat shared/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - docker.io-secret.yaml
$ cat shared/docker.io-secret.yaml
apiVersion: v1
kind: Secret
metadata:
    name: docker.io
type: kubernetes.io/dockerconfigjson
data:
    .dockerconfigjson: ENC[AES256_GCM,data:REMOVED,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: ageREMOVED
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            REMOVED
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2022-01-25T13:28:57Z"
    mac: ENC[AES256_GCM,data:REMOVED,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.7.1

Steps to reproduce

Execute flux diff with the above setup.

Expected behavior

flux diff should work

Screenshots and recordings

No response

OS / Distro

openSUSE Tumbleweed

Flux version

v0.26.1

Flux check

► checking prerequisites
✔ Kubernetes 1.22.5 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.16.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.20.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.21.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.21.1
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

I think I figured out what's going on. For encrypted secrets the diff command detects only changes in the secret keys and ignores values as it can't decrypt them. The CLI just strip the SOPS metadata, but the Kubernetes API tries to convert the value to json (being a kubernetes.io/dockerconfigjson) and obviously fails. Guess we need to place a dummy json in the value to avoid the API error...