Support SHA Digests for container image in the device plugin CRD

[chaitanya1731]

Details -

We found a design difference between the requirements of the upstream community operator and the requirements of RedHat certification. Specifically, the way that container images are referenced in the operator bundle is different.

As per the upstream community operator design, image tags are used to reference all container images in the operator bundle, However, RedHat certification requires that all images be referenced by SHA digest instead of image tags since SHAs are a more secure way of identifying images. Refer Digest Pinning

Reason as per RedHat: the reason a image SHA is used instead of a tag is because a tag can be pushed to a disconnected mirror and be overwritten, making it challenging to diagnose issues in knowing what content is being run.
By identifying containers by their hash, it helps to ensure that the operator bundle (and therefore the operator) are always referring to the exact containers that the bundle was certified with.

Other links for reference -
Image SHA digest vs Image tags
More details here #03270409

Solution suggestions -

To resolve the issue and comply with RedHat certification requirements, the Intel Device Plugin Operator should support SHA Digests validation in the device plugin CRD.

To workaround this issue, we are currently asking users to replace the SHAs by image tags while deploying the device plugin CRs. For example - Change intel-sgx-plugin@sha256:c0423b149b909472460f84a299b087a5104ed40a3572687d6b450609e3bb3076 to intel-sgx-plugin:0.24.0 manually.

Proposed Tasklist

• update validatePluginImage to check expectedMinVersion only if the version is a semver AND if not a semver, check versionStr is a sha256 regexp. Other conditions return fail
• enhance image upgrade (or optionally some better place) to upgrade images if <Kind>_IMAGE or <Kind>_INITIMAGE (e.g., SGXDEVICEPLUGIN_IMAGE) env vars are set and if differs. If not, default to current semver based upgrade.
• (optional) add a way to make bundle with an overlay that adds the env variables

[mythi]

To resolve the issue and comply with RedHat certification requirements, the Intel Device Plugin Operator should support SHA Digests validation in the device plugin CRD.

This alone won't be enough because it breaks "seamless upgrades". My question how it's done with SHAs has not been answered but after doing some reading I found what OLM documentation suggests as the answer. What this doc does not answer is how to get these automatically changed when the images are automatically built for UBI base image updates.

This will be a bit of work for us to get all 10+ images to be added but let me draft a detailed plan and see what can be done.

[uMartinXu]

@mythi Can we split this into two steps:
First step: let's allow the SHA digests to pass the check of the webhook. So it will allow a user to use SHA digest to specify the image which is the requirement of the certifying process. You know we can't release the operator bundle image on OCP without this fix to pass the certification.
Second Step: So as to seamlessly upgrading with SHA digest, we'd like to fix it in the next release(Intel Technology Enabling for Openshift project-1.1)? Currently, the Seamless Upgrading is not our target feature for this Comming 1.0 Release. And we are planning to work on it for the next release 1.1.

So with this two steps plan, we will not block the coming GA 1.0 Release of Intel Technology Enabling for Openshift project. And also we have enough time to work on the Seamless Upgrading features.

BTW, we also working on figuring out who the owner from Red Hat can help with this.

[mythi]

First step: let's allow the SHA digests to pass the check of the webhook.

I think it should be possible to deliver this as a stand-alone fix/feature. See "Proposed Tasklist" I added earlier this afternoon. It has the same split.