Cost per job

[cmelone]

Closes #75

Computes and stores the following metrics:

• cpu_cost: cost of using CPU resources on the node, based on the CPU request of the job
• mem_cost
• cpu_penalty: penalty factor that represents the over or under allocation of CPU resources
• mem_penalty

---

To normalize the cost of resources within instance types, we'll define cost per resource metrics.

$$\text{Cost per CPU}_i = \frac{C_i \times 0.5}{\text{CPU}_i}$$ $$\text{Cost per RAM}_i = \frac{C_i \times 0.5}{\text{RAM}_i}$$

• $C_i$ is the cost of the node over the life of the job
• $CPU_i$ is the number of CPUs available on node $i$
• $RAM_i$ is the amount of RAM available on node $I$
• $C_i$ should be divided in half when computing cost per resource numbers, as we assume that CPU and RAM represent the two halves of the cost of the node.

$$\text{Job Cost} = (\text{CPU}_{\text{usage}} \times \text{Cost per CPU}_i + \text{RAM}_{\text{usage}} \times \text{Cost per RAM}_i)$$

Using this base cost per job metric, jobs are rewarded for minimizing usage and wall time. However, it does not penalyze them for disruptions to the cluster caused by misallocation.

Underallocation can potentially slow down other jobs on the same node, and overallocation delays scheduling of other jobs. A penalty factor would be useful for quantifying negative impacts to the CI system and encourage better resource requests.

$$\text{P}_{\text{CPU}} = |\text{CPU}_{usage}-\text{CPU}_{request}|$$ $$\text{P}_{\text{RAM}} = |\text{RAM}_{usage}-\text{RAM}_{request}|$$

With the penalty, cost per job would be:

$$((\text{CPU}_{\text{usage}} + \text{P}_{\text{CPU}}) \times \text{Cost per CPU}_i + (\text{RAM}_{\text{usage}} + \text{P}_{\text{CPU}}) \times \text{Cost per RAM}_i )$$

Job cost and $P$ are stored separately as the former represents "true" cost, while the latter can be used to measure the efficiency of its resource requests via an artificial penalty. When analyzing costs, node instance type should be controlled for because cost per job is influenced by $\text{Cost per CPU}_i$ and $\text{Cost per RAM}_i$, which will vary among instance types.

---

For example:

• instance costs 100 cents per hour
• instance has 100GB memory and 100 cores
• job duration was 30 minutes
• resource requests: 2GB memory, 2 cores
• mean usage: 1GB memory, 5 cores

$C_i$ = 50 cents (cost of the instance while the job ran)

$$\text{Cost per CPU}_i = \frac{50 \times 0.5}{\text{100}} = 0.25 cents$$ $$\text{Cost per RAM}_i = \frac{50 \times 0.5}{\text{100}} = 0.25 cents$$

therefore,

$$\text{Cost for RAM}_i = 0.25 \times 1 = 0.25 cents$$ $$\text{Cost for CPUs}_i = 0.25 \times 5 = 2.5 cents$$

computing the penalties:

$$\text{P}_{\text{CPU}} = |5-2|= 3$$ $$\text{P}_{\text{RAM}} = |1-2| = 1$$

In this case, we penalize the job for using more CPU than it requested, which could have crowded out other jobs. We also penalize the job for using less RAM than requested because when k8s scheduled the job, it blocked those resources from being scheduled for other work.

"total" cost:

$$((5 + 3) \times 0.25 + (1 + 1) \times 0.25) = 2.5$$

[cmelone]

requesting @tgamblin review of cost formula not code