If kubernetes-dashboard-cert secret is not having any data cert-check-cronjob is not restarting openunison.

[shnigam2]

We have observed if kubernetes-dashboard-cert secret is not having any data, cert-check-cronjob is not restarting openunison. As checking secret exists so no action taken. Want to get this bug fix, because on redeployment of kubernetes-dashboard, kubernetes-dashboard-cert secret comes with 0 data. And we are managing that secret through openunison, but what cert-check-job is doing not restarting openunison because secret exists but issue is secret is not having certificate.

[shnigam2]

@mlbiam Could you please look into this

[shnigam2]

Logs from cert-cronjob-pod

Processing key 'kubernetes-dashboard'
Checking if kubernetes secret exists
Secret exists
Key 'kubernetes-dashboard' finished

Yaml for secret which come first time deployment of kubernetes-dashboard happened:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"dashboard","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"kubernetes-dashboard","app.kubernetes.io/version":"2.3.1","argocd.argoproj.io/instance":"dashboard","helm.sh/chart":"kubernetes-dashboard-5.0.2"},"name":"kubernetes-dashboard-certs","namespace":"kubernetes-dashboard"},"type":"Opaque"}
  creationTimestamp: "2023-02-03T08:25:02Z"
  labels:
    app.kubernetes.io/instance: dashboard
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kubernetes-dashboard
    app.kubernetes.io/version: 2.3.1
    argocd.argoproj.io/instance: dashboard
    helm.sh/chart: kubernetes-dashboard-5.0.2
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
  resourceVersion: "206084874"
  uid: 5f99e33f-51ee-4924-ba89-c939e7a4071f
type: Opaque

But as we restarted orchestra openuison object through annotation data get updated and two labels get updated in secret yaml:

apiVersion: v1
data:
  dashboard.crt: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
  dashboard.key: xxxxxxxxxxxxxxxxxxxxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: "2023-02-03T08:32:57Z"
  labels:
    operated-by: openunison-operator
    tremolo_operator_created: "true"
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
  resourceVersion: "206086929"
  uid: d00d5aa6-1ea8-4c5d-993d-69efbe6e85b1
type: Opaque

Now the expectation is when kubernetes-dashboard-cert is managed by openunison it should check data exists or not also inside that secret. And if data not exist then also Restart of openunison triggered to get data updated in the secret.

[mlbiam]

The cert-check cron job isn't designed to handle this use case. We'll add it as an enhancement in our next release cycle (should be by end of February). The current recommendation is to add an annotation to the openunison orchestra object to trigger the operator to create the certificate or to update your kubernetes dashboard rollouts to not include the tls Secret so it doesn't get overwritten.

[shnigam2]

Hi @mlbiam ,

Is there any update on new release about adding this new enhancement, by when we expect new operator release.

Regards
Shobhit

[mlbiam]

We're in the process of writing a new operator that will make it much easier to handle this, and other, use cases. It will also be much smaller and will be backwards compatible. Expect to see something that can be tested in the next week or so.

[shnigam2]

Thanks for an update @mlbiam , if possible can we do cert check frequency in every hour or if possible for secrets which are not a part of openunison can be monitored in more frequent manner. Just a thought because redeployment of other product like dashboard can be promoted separately, so if possible please increase the frequency of cert-check or if possible separate cert-check-jobs for secrets which are not a part of openunison namespace.

[mlbiam]

The goal is to combine the logic of the operator and check certs into a single binary that is nateively compiled and much smaller. In theory, if you wanted we can update the charts so the CronJob runs as often as you'd like.

[shnigam2]

Hi @mlbiam is there any update on the new release and yes if possible please update chart so that we run cronjob as we want.

[mlbiam]

The latest openunison operator has been released. the latest charts have the updated images. Also, we added the usecase that if the kubernetes-dashboard certs get overwritten, the check-certs cronjob will catch it and redeploy them.