test/e2e: Add nydus-snapshotter workaround

There is an issue in containerd around the caching of snapshotter
images. Chengyu has created a workaround in nydus-snapshotter,
so let's try installing that fork to get the CI working.

Signed-off-by: stevenhorsman <[email protected]>

src/cloud-api-adaptor/hack/nydus-snapshotter-fork-deploy.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+# Copyright (c) 2024 IBM Corporation
+
+# This is temp hack to try out Chengyu's nydus snapshotter fix in https://github.com/containerd/nydus-snapshotter/pull/593
+# before it has been merged, released and the CoCo operator updated to consume it.
+
+nydus_snapshotter_install_dir="/tmp/nydus-snapshotter"
+nydus_snapshotter_url=https://github.com/ChengyuZhu6/nydus-snapshotter.git
+nydus_snapshotter_version=fix-error
+git clone -b "${nydus_snapshotter_version}" "${nydus_snapshotter_url}" "${nydus_snapshotter_install_dir}"
+pushd "$nydus_snapshotter_install_dir"
+
+# Use sed rather than yq as we don't know which version might be installed
+sudo sed -i -e 's/^\(\s*FS_DRIVER: \).*$/\1\"proxy\"/g' "misc/snapshotter/base/nydus-snapshotter.yaml"
+sudo sed -i -e 's/^\(\s*ENABLE_CONFIG_FROM_VOLUME: \).*$/\1\"false\"/g' "misc/snapshotter/base/nydus-snapshotter.yaml"
+sudo sed -i -e 's/^\(\s*ENABLE_SYSTEMD_SERVICE: \).*$/\1\"true\"/g' "misc/snapshotter/base/nydus-snapshotter.yaml"
+sudo sed -i -e 's/^\(\s*ENABLE_RUNTIME_SPECIFIC_SNAPSHOTTER: \).*$/\1\"true\"/g' "misc/snapshotter/base/nydus-snapshotter.yaml"
+
+sudo sed -i -e 's%^\(\s*image: \)\("ghcr.io\/containerd/nydus-snapshotter:latest"\)%\1\"quay.io/chengyu_zhu/nydus-snapshotter:0.13.17\"%g' "misc/snapshotter/base/nydus-snapshotter.yaml"
+
+kubectl create -f "misc/snapshotter/nydus-snapshotter-rbac.yaml"
+kubectl apply -f "misc/snapshotter/base/nydus-snapshotter.yaml"
+
+kubectl rollout status DaemonSet nydus-snapshotter -n nydus-system --timeout 5m
+
+pods_name=$(kubectl get pods --selector=app=nydus-snapshotter -n nydus-system -o=jsonpath='{.items[*].metadata.name}')
+kubectl logs "${pods_name}" -n nydus-system

src/cloud-api-adaptor/install/README.md

@@ -133,6 +133,21 @@ Alternatively the manual approach, is:
     kubectl logs ds/peerpodconfig-ctrl-caa-daemon -n confidential-containers-system
     ```
 
+## Workaround: Updating nydus-snapshotter to a fork
+
+There is an issue with containerd not caching images with snapshotter labels and peer-pods using the nydus-snapshotter to implement
+the image pull on guest.
+Chengyu's has implemented a workaround in the nydus-snapshotter in https://github.com/containerd/nydus-snapshotter/pull/593
+and we want to try it out before it has been merged, released and the CoCo operator updated to consume it to get peer pods working.
+To try this temporary experimental feature run:
+```
+./hack/nydus-snapshotter-fork-deploy.sh
+```
+After completion you can check it has worked with:
+```
+kubectl get pod -o json -n nydus-system | jq .items[].spec.containers[].image
+```
+
 ## Building custom cloud-api-adaptor image
 
 * Set CLOUD_PROVIDER

src/cloud-api-adaptor/test/provisioner/provision.go

@@ -613,6 +613,14 @@ func (p *CloudAPIAdaptor) Deploy(ctx context.Context, cfg *envconf.Config, props
 		return err
 	}
 
+	// Temp workaround to update the nydus-snapshotter with Chengyu's branch
+	cmd = exec.Command("./hack/nydus-snapshotter-fork-deploy.sh")
+	stdoutStderr, err = cmd.CombinedOutput()
+	log.Tracef("%v, output: %s", cmd, stdoutStderr)
+	if err != nil {
+		return err
+	}
+
 	return nil
 }