[Openshift] Repeated install/uninstall and removal of CamelCatalog leads to re-creation of builder image

[lfabriko]

Following scenario leads to issue in CK operator log and creating new builder image:

1. Install and run an Integration
2. reset the Integration and kit (kamel reset)
3. Delete the CamelCatalog custom resource (oc delete camelcatalog --all)
4. run again the integration

Result of oc get images is incremented by another builder image image-registry.openshift-image-registry.svc:5000/lfabriko2/camel-k-runtime-quarkus-builder@sha256:*

Log of CK operator:
{"level":"info","ts":1689252480.1975944,"logger":"camel-k.controller.catalog","msg":"Could not find Camel K builder container"}

[squakez]

Yes, this may happen in Openshift because the S2I will check an ImageStream which was cleaned by the uninstallation problem. Thanks for reporting it's something we need to look at, though it should not be harmful in a "normal" environment where the user does not uninstall continuously.

[gansheer]

The owner of the ImageStream is the CamelCatalog resource which seems to be the right design in the end because it is its controller that triggers the build.

I think it goes down to two possibilities (I don't know if any are valid in openshift):

• ensures that the removal of the ImageStream also removes the images declared in it
• when creating the builder image check if the Image is present in the repository instead of only the ImageStream

[gansheer]

I am looking into this issue.

[gansheer]

The openshift image registry is composed of multiple custom ressources. Some of them are namespaced - those we use are ImageStream, ImageStreamTag and ImageTag, some are not namespaced - like Image.

Image resource is clearly not meant to be modified easily:

• there is no notion of image tag in its datas so there is no way to know if any camel-k-runtime-quarkus-builder image is the right one

dockerImageMetadataVersion: "1.0"
dockerImageReference: image-registry.openshift-image-registry.svc:5000/default/camel-k-runtime-quarkus-builder@sha256:873a5580b0ffed32321e7020da4ae6906d9046dc713f3a9019987393284bf868
kind: Image
metadata:
  annotations:
    image.openshift.io/dockerLayersOrder: ascending
    image.openshift.io/manifestBlobStored: "true"
    openshift.io/image.managed: "true"
  creationTimestamp: "2023-07-19T09:11:39Z"
  name: sha256:873a5580b0ffed32321e7020da4ae6906d9046dc713f3a9019987393284bf868
  resourceVersion: "57031"
  uid: 4457cf52-debd-4acb-b849-d7d8517b7a46

• access to the Image resources is restricted to a level when even extending the openshift specific role of the operator is not enough to list them
  

  camel-k/config/rbac/openshift/operator-role-openshift.yaml

  Lines 52 to 60 in 55cc551

  	- apiGroups:
  	- ""
  	- "image.openshift.io"
  	resources:
  	- imagestreamimages
  	- imagestreammappings
  	- imagestreams
  	- imagestreams/secrets
  	- imagestreamtags

• adding specific camel labels metadata is not available in Openshift CLIs
• adding specific owner reference in metadata is ignored when the referenced object is deleted

All these observations leads me to the conclusion that the two possibilities I listed are not valid in openshift. If we were in a normal production use case the ImagePruners would do the job of cleaning the image registry.

I believe trying to add any code to manage this use case described would do more harm than good as the only modification that would work is removing the owner reference from the ImageStream.

[gansheer]

For those using the Camel-k CLI you could avoid this issue with using the flag --skip-camel-catalogs in your command: kamel uninstall --skip-camel-catalogs ..... It will keep the CamelCatalog custom resource and all the resource it is owner of.

[squakez]

Thanks @gansheer. Taking in consideration your analysis, I'd be inclined to close this issue and let the user avoid cleaning the catalog when it is not required by their processes.

[squakez]

Documented in #4611