Helm watches ignores version of GVK and runs charts on all requests for GK

[gbmeuk]

Bug Report

What did you do?

Created a controller with two helm charts helm-charts/v1alpha1 and helm-charts/v1alpha2 and a watches file with entries for both, then created a CR for just v1alpha1 but the controller runs both charts and keeps flipping between the two.

Watches:

- group: devops.mesoform.com
  version: v1alpha1
  kind: TestingWatches
  chart: helm-charts/v1alpha1
- group: devops.mesoform.com
  version: v1alpha2
  kind: TestingWatches
  chart: helm-charts/v1alpha2

v1alpha1 chart just creates a namespace as:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    version: v1alha1
  name: {{ .Release.Name }}

v1alpha2 is slightly different

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    version: v1alpha2
    new.property: {{ .Values.newProperty }}
  name: {{ .Release.Name }}

on the CRD, v1alpha1 is served and stored and v1alpha2 is just served.

What did you expect to see?

Just the v1alpha1 chart is ran with just these in the logs:

{"level":"info","ts":"2024-11-09T18:45:27Z","logger":"helm.controller","msg":"Watching dependent resource","ownerApiVersion":"devops.mesoform.com/v1alpha1","ownerKind":"TestingWatches","apiVersion":"v1","kind":"Namespace"}
{"level":"info","ts":"2024-11-09T18:45:27Z","logger":"helm.controller","msg":"Installed release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1"}

And the resources looking as:

Name:         watch1
Namespace:    default
Labels:       <none>
Annotations:  helm.sdk.operatorframework.io/reconcile-period: 300s
              operator-sdk/primary-resource: /watch1
              operator-sdk/primary-resource-type: Namespace
API Version:  devops.mesoform.com/v1alpha1
Kind:         TestingWatches
Metadata:
  Creation Timestamp:  2024-11-09T18:45:23Z
  Finalizers:
    helm.sdk.operatorframework.io/uninstall-release
  Generation:        1
  Resource Version:  172196431
  UID:               9db799c9-8394-45d9-b77e-80e52b87732f
Spec:
  Old Property:  test1
Status:
  Conditions:
    Last Transition Time:  2024-11-09T18:45:23Z
    Status:                True
    Type:                  Initialized
    Last Transition Time:  2024-11-09T18:45:27Z
    Reason:                UpgradeSuccessful
    Status:                True
    Type:                  Deployed
  Deployed Release:
    Manifest:  ---
# Source: TestingWatch/templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    version: v1alha1
  name: watch1

    Name:  watch1
Events:    <none>

Name:         watch1
Labels:       app.kubernetes.io/managed-by=Helm
              helm.sdk.operatorframework.io/chart=TestingWatch
              hnc.x-k8s.io/included-namespace=true
              kubernetes.io/metadata.name=watch1
              watch1.tree.hnc.x-k8s.io/depth=0
Annotations:  meta.helm.sh/release-name: watch1
              meta.helm.sh/release-namespace: default
              version: v1alha1
Status:       Active

No resource quota.

No LimitRange resource.

What did you see instead? Under which circumstances?

Both charts are ran, and the resources flip between the attributes above and v1alpha2 ones:

Name:         watch1
Namespace:    default
Labels:       <none>
Annotations:  helm.sdk.operatorframework.io/reconcile-period: 300s
              operator-sdk/primary-resource: /watch1
              operator-sdk/primary-resource-type: Namespace
API Version:  devops.mesoform.com/v1alpha1
Kind:         TestingWatches
Metadata:
  Creation Timestamp:  2024-11-09T18:45:23Z
  Finalizers:
    helm.sdk.operatorframework.io/uninstall-release
  Generation:        1
  Resource Version:  172196461
  UID:               9db799c9-8394-45d9-b77e-80e52b87732f
Spec:
  Old Property:  test1
Status:
  Conditions:
    Last Transition Time:  2024-11-09T18:45:23Z
    Status:                True
    Type:                  Initialized
    Last Transition Time:  2024-11-09T18:45:27Z
    Reason:                UpgradeSuccessful
    Status:                True
    Type:                  Deployed
  Deployed Release:
    Manifest:  ---
# Source: TestingWatch/templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    version: v1alha2
    new.property: 
  name: watch1

    Name:  watch1
Events:    <none>

Name:         watch1
Labels:       app.kubernetes.io/managed-by=Helm
              helm.sdk.operatorframework.io/chart=TestingWatch
              hnc.x-k8s.io/included-namespace=true
              kubernetes.io/metadata.name=watch1
              watch1.tree.hnc.x-k8s.io/depth=0
Annotations:  meta.helm.sh/release-name: watch1
              meta.helm.sh/release-namespace: default
              version: v1alha2
Status:       Active

No resource quota.

No LimitRange resource.

Logs showing the upgrades and errors:

{"level":"info","ts":"2024-11-09T18:45:11Z","logger":"cmd","msg":"Version","Go Version":"go1.21.7","GOOS":"linux","GOARCH":"amd64","helm-operator":"v1.34.1","commit":"edaed1e5057db0349568e0b02df3743051b54e68"}
{"level":"info","ts":"2024-11-09T18:45:11Z","logger":"cmd","msg":"Watching all namespaces"}
{"level":"info","ts":"2024-11-09T18:45:11Z","logger":"helm.controller","msg":"Watching resource","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","reconcilePeriod":"1m0s"}
{"level":"info","ts":"2024-11-09T18:45:11Z","logger":"helm.controller","msg":"Watching resource","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","reconcilePeriod":"1m0s"}
{"level":"info","ts":"2024-11-09T18:45:11Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-11-09T18:45:11Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false}
{"level":"info","ts":"2024-11-09T18:45:11Z","msg":"starting server","kind":"health probe","addr":"[::]:8081"}
I1109 18:45:11.999650       1 leaderelection.go:250] attempting to acquire leader lease testingwatches-system/testingwatches...
I1109 18:45:12.081545       1 leaderelection.go:260] successfully acquired lease testingwatches-system/testingwatches
{"level":"info","ts":"2024-11-09T18:45:12Z","msg":"Starting EventSource","controller":"testingwatches-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-11-09T18:45:12Z","msg":"Starting Controller","controller":"testingwatches-controller"}
{"level":"info","ts":"2024-11-09T18:45:12Z","msg":"Starting EventSource","controller":"testingwatches-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-11-09T18:45:12Z","msg":"Starting Controller","controller":"testingwatches-controller"}
{"level":"info","ts":"2024-11-09T18:45:12Z","msg":"Starting workers","controller":"testingwatches-controller","worker count":2}
{"level":"info","ts":"2024-11-09T18:45:12Z","msg":"Starting workers","controller":"testingwatches-controller","worker count":2}
{"level":"error","ts":"2024-11-09T18:45:27Z","logger":"helm.controller","msg":"Release failed","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1","error":"failed to install release: release: already exists","stacktrace":"github.com/operator-framework/operator-sdk/internal/helm/controller.HelmOperatorReconciler.Reconcile\n\t/workspace/internal/helm/controller/reconcile.go:245\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-11-09T18:45:27Z","msg":"Starting EventSource","controller":"testingwatches-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-11-09T18:45:27Z","logger":"helm.controller","msg":"Watching dependent resource","ownerApiVersion":"devops.mesoform.com/v1alpha1","ownerKind":"TestingWatches","apiVersion":"v1","kind":"Namespace"}
{"level":"info","ts":"2024-11-09T18:45:27Z","logger":"helm.controller","msg":"Installed release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1"}
{"level":"error","ts":"2024-11-09T18:45:27Z","logger":"helm.controller","msg":"Failed to update status after install release failure","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1","error":"Operation cannot be fulfilled on testingwatches.devops.mesoform.com \"watch1\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/operator-framework/operator-sdk/internal/helm/controller.HelmOperatorReconciler.Reconcile\n\t/workspace/internal/helm/controller/reconcile.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-11-09T18:45:27Z","msg":"Reconciler error","controller":"testingwatches-controller","object":{"name":"watch1","namespace":"default"},"namespace":"default","name":"watch1","reconcileID":"1f8a1462-effa-447e-a91a-eae9917a95c6","error":"failed to install release: release: already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-11-09T18:45:28Z","logger":"helm.controller","msg":"Installed release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1"}
{"level":"error","ts":"2024-11-09T18:45:28Z","logger":"helm.controller","msg":"Release failed","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1","error":"failed to install release: release: already exists","stacktrace":"github.com/operator-framework/operator-sdk/internal/helm/controller.HelmOperatorReconciler.Reconcile\n\t/workspace/internal/helm/controller/reconcile.go:245\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-11-09T18:45:28Z","msg":"Reconciler error","controller":"testingwatches-controller","object":{"name":"watch1","namespace":"default"},"namespace":"default","name":"watch1","reconcileID":"30dc1dea-918e-41a0-89b7-967c170b1159","error":"failed to install release: release: already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-11-09T18:45:29Z","logger":"helm.controller","msg":"Release failed","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1","error":"failed to install release: release: already exists","stacktrace":"github.com/operator-framework/operator-sdk/internal/helm/controller.HelmOperatorReconciler.Reconcile\n\t/workspace/internal/helm/controller/reconcile.go:245\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-11-09T18:45:29Z","logger":"helm.controller","msg":"Installed release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1"}
{"level":"error","ts":"2024-11-09T18:45:29Z","msg":"Reconciler error","controller":"testingwatches-controller","object":{"name":"watch1","namespace":"default"},"namespace":"default","name":"watch1","reconcileID":"cb92b651-00d0-4269-a50c-7a782d47db5b","error":"failed to install release: release: already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-11-09T18:45:30Z","msg":"Starting EventSource","controller":"testingwatches-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-11-09T18:45:30Z","logger":"helm.controller","msg":"Watching dependent resource","ownerApiVersion":"devops.mesoform.com/v1alpha2","ownerKind":"TestingWatches","apiVersion":"v1","kind":"Namespace"}
{"level":"info","ts":"2024-11-09T18:45:30Z","logger":"helm.controller","msg":"Installed release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1"}
{"level":"error","ts":"2024-11-09T18:45:30Z","logger":"helm.controller","msg":"Release failed","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1","error":"failed to install release: release: already exists","stacktrace":"github.com/operator-framework/operator-sdk/internal/helm/controller.HelmOperatorReconciler.Reconcile\n\t/workspace/internal/helm/controller/reconcile.go:245\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-11-09T18:45:30Z","logger":"helm.controller","msg":"Installed release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1"}
{"level":"error","ts":"2024-11-09T18:45:30Z","logger":"helm.controller","msg":"Failed to update status after install release failure","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1","error":"Operation cannot be fulfilled on testingwatches.devops.mesoform.com \"watch1\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/operator-framework/operator-sdk/internal/helm/controller.HelmOperatorReconciler.Reconcile\n\t/workspace/internal/helm/controller/reconcile.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-11-09T18:45:30Z","msg":"Reconciler error","controller":"testingwatches-controller","object":{"name":"watch1","namespace":"default"},"namespace":"default","name":"watch1","reconcileID":"6f3ebbb8-599c-456b-ab90-c463568320ac","error":"failed to install release: release: already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-11-09T18:45:31Z","logger":"helm.controller","msg":"Upgraded release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1","force":false}
{"level":"info","ts":"2024-11-09T18:45:32Z","logger":"helm.controller","msg":"Reconciled release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1"}
{"level":"info","ts":"2024-11-09T18:45:32Z","logger":"helm.controller","msg":"Upgraded release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1","force":false}
{"level":"info","ts":"2024-11-09T18:45:34Z","logger":"helm.controller","msg":"Reconciled release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1"}
{"level":"info","ts":"2024-11-09T18:45:34Z","logger":"helm.controller","msg":"Upgraded release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1","force":false}
{"level":"info","ts":"2024-11-09T18:45:35Z","logger":"helm.controller","msg":"Reconciled release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha1","kind":"TestingWatches","release":"watch1"}
{"level":"info","ts":"2024-11-09T18:45:36Z","logger":"helm.controller","msg":"Upgraded release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1","force":false}
{"level":"info","ts":"2024-11-09T18:45:37Z","logger":"helm.controller","msg":"Reconciled release","namespace":"default","name":"watch1","apiVersion":"devops.mesoform.com/v1alpha2","kind":"TestingWatches","release":"watch1"}
...

Environment

Operator type:

/language helm

Kubernetes cluster type:

GKE

Dockerfile:

FROM quay.io/operator-framework/helm-operator:v1.37.0

ENV HOME=/opt/helm
COPY watches.yaml ${HOME}/watches.yaml
COPY helm-charts  ${HOME}/helm-charts
WORKDIR ${HOME}

Possible Solution

Haven't found anything obvious in code.

Additional context

N/A

[acornett21]

Hi @gbmeuk If you need to support multiple api's, you need to write a go operator so you can implement a conversion webhook to support the multiple apis. More on info on webhooks.

[gbmeuk]

Hi @gbmeuk If you need to support multiple api's, you need to write a go operator so you can implement a conversion webhook to support the multiple apis. More on info on webhooks.

Thanks, @acornett21. I think Documentation may need clarifying for Helm as it's misleading to think the operator SDK will watch a version when it won't? Will the webhook approach work for Ansible Operator as well? In some testing with Ansible, I was looking to use standard Admission Webhooks to take the apiVersion from CREATE/UPDATE requests and add it as an annotation where Ansible could use that to decide whether to run a given role (CRD version) or not.

P.S. both links are the same.