Consider LimitRanges when calculating Workload usage

[alculquicondor]

What would you like to be added:

Administrators can setup LimitRanges per namespace to set default requests for Pods.

We should consider these defaults when creating a Workload.
It can be done in the Workload webhook so that it applies to any custom job, similar to #316.

Caveat: if a LimitRange is created after the Workload object is created, we will not have an accurate calculation of requests.

This issue is an spinoff from #485

Why is this needed:

LimitRanges are a common tool for admins to set defaults for a namespace.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[trasc]

/assign

[mcariatm]

/assign

[trasc]

@alculquicondor Maybe the workload web-hook is not the best way to add this, since we'll need tho adapt the fix for #590.
There will a lot easier if:
a. the defaulting (both for "limits to requests" and LimitsRanges ) is done in the job web-hook
or
b. the job to workload equality ignores the resources needs of the containers

[alculquicondor]

a. This means that we would have to do the same for every custom job API (mpi, ray, etc), which adds more complexity to integrations.
b. We cannot completely ignore the resources, because that means that we are not calculating the most up-to-date quota.

You can work independently from #590. Whichever PR is ready later will have to rebase.

[alculquicondor]

cc @kerthcet

After some offline discussion, we believe it's better to have a "totalRequests" field in the status. This can be calculated during admission and added in the same API call. This means we don't need to recreate or update the Workloads if LimitRanges or RuntimeClass changes.

[kerthcet]

So we'll only update the totalRequests in admission, and after admission, we'll skip the calculation to avoid querying the limitRange, right? Then we can also solve the bug here #590

[alculquicondor]

That is correct, that would solve #590 too. After admission, we would use the calculated requests in the cache.

[kerthcet]

This is solved? What about the field totalRequests? Or this is closed by mistake.

[trasc]

It is solved.
This however has two follow-up's #611 and #612, In the PR for 611 we'll need to be able to record the requests at the time of admission, hence we will need some API change for that.