Fewest nodes strategy

[sylr]

Description

What problem are you trying to solve?

It seems that Karpenter instance scheduling is solely based on EC2 economics.

Nodes will be scheduled based on instance type price, so, given node type constraints and resources needed to be scheduled, we can end up with, for example, 2 c7.xlarge instances instead of 1 c7.2xlarge.

This model does not work when using paying services which are billed per host (e.g.: DataDog, outcoldsolutions, orca.security ... etc).

It would really be nice to have a fewest-nodes strategy for scheduling.

How important is this feature to you?

Given the price of the aforementioned services, it is really important :)

[sylr]

We could also consider adding possibility to define custom costs which would be considered during scheduling, e.g.:

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: dedicated-ingress-arm64
spec:
  # In the same currency as the instance price
  costs: 
  - name: datadog per host cost
    perHostMonthlyPrice: 25
  - name: orca.security per host cost
    perHostMonthlyPrice: 30

With this, considering that all resources needed for scheduling correspond to a m7.2xlarge we would have:

m7.xlarge price	m7.xlarge + costs	m7.2xlarge price	m7.2xlarge + costs	Result
45	(45 + 30) x 2 = 150	100	(100 + 30) x 1 = 130	1 m7.2xlarge
40	(40 + 30) x 2 = 140	100	(100 + 30) x 1 = 130	1 m7.2xlarge
35	(35 + 30) x 2 = 120	100	(100 + 30) x 1 = 130	2 m7.xlarge

[flbla]

hi,
I think it's the same as this one #5033