Option on Provisioner to set a minimum number of on-demand nodes

[nikoul]

Tell us about your request

I have a k8s deployment that currently benefits from spot instances using a karpenter Provisioner with both spot and on-demand capacity types . However for reliability reasons, I would like to set a minimum of X nodes on-demand, even when there is spot availability.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

My k8s deployment is composed of ML inference nodes that take 5 to 8 minutes to spin up.
Let's take a scenario where 100% of the deployment is made of spot instances. In case of a massive spot interruption from AWS (it happens sometimes), all the nodes in the deployment might go down, and no pods will serve requests until the on-demand fallback nodes spin up (which takes up to 8 minutes).

Are you currently working around this issue?

We're currently crossing fingers that at least 1 spot instance node will remain in case of a massive spot interruption.

Additional Context

No response

Attachments

No response

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[engedaam]

Have you considered maintaining multiple provisioners one for spot and one for on-demand?

[jonathan-innis]

In case of a massive spot interruption from AWS

Do you have native interruption handling enabled? If you do, in most cases Karpenter should have enough time to spin-up replacement nodes before the capacity gets completely reclaimed by EC2.

I would like to set a minimum of X nodes on-demand

There's an issue (#2050) covering manual node provisioning right now, so we're definitely hoping that we can have a way to deploy a static number of nodes in the future. Feel free to +1 that issue and add the context around your use-case.

[nikoul]

Have you considered maintaining multiple provisioners one for spot and one for on-demand?

I have quickly thought about it, but I'm not sure how it would solve this. From my understanding, at best it would randomly choose a on-demand node, but it wouldn't enforce it.

[ellistarn]

You may be interested in #757.

Also, #749 would let you pin a static number of on demand nodes.

[ellistarn]

Closing as a dup of #749