ci-kubernetes-conformance-kind-ga-only is experiencing pod scheduling timeout

[hasheddan]

What happened:

ci-kubernetes-conformance-kind-ga-only is experiencing pod scheduling timeout since #18471

What you expected to happen:

For the pod to be scheduled.

Please provide links to example occurrences, if any:

https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-conformance-kind-ga-only/1288101820190691330

Anything else we need to know?:

@spiffxp I am having trouble determining what is causing the scheduling timeout based on looking at the metrics. Do you have any insight?

/assign

[hasheddan]

Pod was scheduled successfully this morning: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-conformance-kind-ga-only/1288464839429066754

[BenTheElder]

#18530

[RobertKielty]

For ref conformance-all#conformance-ga-only

[spiffxp]

https://prow.k8s.io/?job=ci-kubernetes-conformance-kind-ga-only

• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-conformance-kind-ga-only/1290097710933741573
  • There are no nodes that your pod can schedule to - check your requests, tolerations, and node selectors (0/22 nodes are available: 19 Insufficient cpu, 3 node(s) had taints that the pod didn't tolerate.)
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-conformance-kind-ga-only/1290279157099401216
  • Job execution failed: Pod scheduling timeout.

Perhaps we should bring the CPU limit down a bit? Or should we somehow be forcing the cluster to be larger?

/assign @BenTheElder

[hasheddan]

@spiffxp also seeing this same issue with https://testgrid.k8s.io/sig-release-master-blocking#kind-master-parallel. It appears both of these jobs are requesting a full node (i.e. 7300m CPU). Is this necessary?

[BenTheElder]

The full node is to avoid noisy neighbors.
We cannot schedule for disk IO
I'm looking at moving some things into memory but for now yes, the goal is an isolated node. This is comparable to the boskos resources created by a cloudy e2e

The cluster should be bigger

[spiffxp]

/close

https://prow.k8s.io/?job=ci-kubernetes-conformance-kind-ga-only
https://prow.k8s.io/?job=ci-kubernetes-kind-e2e-parallel

Neither have had an error status (grey triangle) since #18637 landed

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close

https://prow.k8s.io/?job=ci-kubernetes-conformance-kind-ga-only
https://prow.k8s.io/?job=ci-kubernetes-kind-e2e-parallel

Neither have had an error status (grey triangle) since #18637 landed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.