Optimize log output when resources are not enough

[hwdef]

What would you like to be added:

Optimize the scheduler's logs when resources are low by printing exactly what kind of resources are low.

Why is this needed:

easier to identify the reasons for insufficient resources.

kubernetes scheduler VS volcano

kubernetese scheduler:

volcano:

volcano just print NotEnoughResources

[srikanth-iyengar]

Hello @hwdef, I'd like to inquire if I could select this particular issue as my initial task for contributing to Volcano:

Is it feasible to determine the type of resource scarcity causing a job blockage? One potential approach could involve quering the Kubernetes cluster for available memory and CPU. By comparing these metrics with the job's resource allocation requests, we could potentially pinpoint the source of failure in resource allocation. Your guidance on whether this is an appropriate first task would be greatly appreciated.

[hwdef]

Yes you can fix this, our goal is that the error prompts in case of insufficient resources are consistent with the native scheduler of kubernetes.

[srikanth-iyengar]

@hwdef Can you please confirm if my approach is correct or not
i.e. quering the Kubernetes cluster for available memory and CPU. By comparing these metrics with the job's resource allocation requests we can find which is causing the issue and will log the same

[hwdef]

@hwdef Can you please confirm if my approach is correct or not i.e. quering the Kubernetes cluster for available memory and CPU. By comparing these metrics with the job's resource allocation requests we can find which is causing the issue and will log the same

By and large this is correct, but I don't think you need to actively query the k8s for available cpu, memory remaining, volcano should already be maintaining these in cache. And we don't just need to know the cpu and memory ones, there are also extended resources such as nvidia.com/gpu

[srikanth-iyengar]

Ohk got it,
Will start working on this

[srikanth-iyengar]

/assign

[lowang-bh]

Hi @srikanth-iyengar , you can refer to the following display, deployment with 2 pods.

pod first scheduled failed

another pod's message:

podgroup message:

[srikanth-iyengar]

Will this solution resolve the issue of resource unavailability log?

I am covering all the resources by looping through the resource required like this

[srikanth-iyengar]

Hi @srikanth-iyengar , you can refer to the following display, deployment with 2 pods.

pod first scheduled failed

another pod's message:

podgroup message:

Thanks for the info @lowang-bh

[srikanth-iyengar]

Hey @hwdef and @lowang-bh, could one of you provide a heads-up if everything is looking good in the PR?

[srikanth-iyengar]

/assign