Packaging job on ci.jenkins.io never completes - retries fail

[MarkEWaite]

Service(s)

ci.jenkins.io

Summary

Packaging job on ci.jenkins.io has failed to allocate an agent for last 8 hours

Reproduction steps

1. Open the packaging job, confirm that the agent is not allocated

[dduportal]

The pipeline logs shows errors like the following, indicating that the pod agents failed to be started (created in Kubernetes but never reached the "Running" state with an active connection to the controller):

06:17:06  ERROR: Failed to launch packaging-packaging-master-329-9hqpp-g6p96-v3n93
06:17:06  io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [1000000] milliseconds for [Pod] with name:[packaging-packaging-master-329-9hqpp-g6p96-v3n93] in namespace [jenkins-agents].
06:17:06  	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939)
06:17:06  	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921)
06:17:06  	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97)
06:17:06  	at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185)
06:17:06  	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
06:17:06  	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
06:17:06  	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
06:17:06  	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
06:17:06  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
06:17:06  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
06:17:06  	at java.base/java.lang.Thread.run(Unknown Source)
06:17:06  Pod jenkins-agents/packaging-packaging-master-329-9hqpp-g6p96-v3n93 was just deleted

[dduportal]

FWIW, last successful build was the 10 of May 2024. Since then we've moved the ci.jenkins.io Linux container agent workloads from AWS/DigitalOcean to Azure: might (or might not be related).

Currently checking the pipeline agent label and setup, along with other errors in the controller and pods.

[dduportal]

Oh: this pipeline uses a custom pod template definitions: https://github.com/jenkinsci/packaging/blob/3bb91b8dcb84387f4c04497d118109f029681fe2/Jenkinsfile#L12 (https://github.com/jenkinsci/packaging/blob/master/KubernetesPod.yaml).

This is most probably broken since #3954. Let's see if we can reuse the "all in one container image" here (cc @smerle33 FYI)

[dduportal]

Confirmed the issue: the "custom" pod template definition is missing tolerations which forbids it to be scheduled.

As it uses the jenkinsciinfra/packaging Docker image, using the "All in one" image might be counter-productive as this job should have the same behavior in both CI and (private) CD.

We might have to update the definition to inherit from existing pod template (to get the admin-defined parameters) and only override what is needed (at least the image).

[dduportal]

PR opened in jenkinsci/packaging#465. I can't tell if it fails because I'm not a maintainer of the repo, or if it is an issue due to my PR.

[dduportal]

PR merged, agents are running