e2e-cos-device-plugin-gpu job fails

[bart0sh]

What happened:

https://testgrid.k8s.io/sig-node-containerd#e2e-cos-device-plugin-gpu job fails continuously since at least Jan 22 2024.

What you expected to happen:

The device plugins test cases are running locally just fine, so the job should run too.

Anything else we need to know?:
It looks like the job fails to start api server. Here is what I see in the buildlog.txt:

Last output from querying API server follows:
-----------------------------------------------------
*   Trying 34.168.10.172:443...
* connect to 34.168.10.172 port 443 failed: Connection refused
* Failed to connect to 34.168.10.172 port 443 after 39 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to 34.168.10.172 port 443 after 39 ms: Couldn't connect to server

[bart0sh]

/cc @swatisehgal @ffromani @SergeyKanzhelev
/sig node
/sig testing

[kannon92]

@dims could this be related to the cloud provider deprecation? I remember seeing this GPU list in that job but didn't see if the job was successful or not before.

[dims]

@kannon92 please see below:

Snippet from:

Feb 07 13:36:48.024915 bootstrap-e2e-master containerd[1191]: time="2024-02-07T13:36:48.024858788Z" level=error msg="PullImage \"registry.k8s.io/kube-apiserver-amd64:v1.30.0-alpha.1.107_052bce26f4f48b\" failed" error="rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver-amd64:v1.30.0-alpha.1.107_052bce26f4f48b\": failed to resolve reference \"registry.k8s.io/kube-apiserver-amd64:v1.30.0-alpha.1.107_052bce26f4f48b\": registry.k8s.io/kube-apiserver-amd64:v1.30.0-alpha.1.107_052bce26f4f48b: not found"

Discussion thread:
https://cloud-native.slack.com/archives/GGEQHJ0AE/p1707284414739809?thread_ts=1707230141.026879&cid=GGEQHJ0AE

containerd PR that just merged:
containerd/containerd#9779

artifacts got built:
https://prow.k8s.io/?repo=containerd%2Fcontainerd&type=periodic

Now we have wait for the CI jobs to recover, feel free to kick them off now. Hopefully we get past this point (and not see this specific problem). Hoping that is enough.

(NOTE: not related to cloud provider deprecation, just a hiccup in upstream containerd)

[bart0sh]

The job started to run. Thank you for fixing it! Closing the issue.

[aojea]

love it, thanks all