support custom device plugin resource

[lzjqsdd]

Tell us about your request

'Provision' CRD support custom device plugin resource which can be ignored when matching ec2 instances

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

in my case, pod need mount /dev/fuse. Now we build a device plugin for /dev/fuse, and register example/fuse resource to kubelet, and already deploy a daemonSet for this resource. this case is same like “nvidia.com/gpu” . But karpenter can't identify "example/fuse", and won't privisioning new k8s node.

if karpenter can ignore this custom resource ,and still provisioning new node , the "example/fuse" daemonSet will depoly soon, and pod can be scheduled immediately.

so, i think if karpenter Provisioner CRD can support custom device plugin resource, and ignore this resource when trying to match available ec2 instances.

Are you currently working around this issue?

Now i have to create a leading pod without this "example/fuse" resource. And force karpenter to provisioning new node. Then delete this leading pod when it's running, this means i already got enough resource for real pod. And the real pod which need "example/fuse" can be scheduled soon. this solution seems some kind of weird .

So If karpetner native support custom device plugin resource woluld be best, users just define custom Provisioner for his needed resource.

Additional Context

No response

Attachments

No response

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

Linking the relevant issues that are tracking similar requests"

1. karpenter doesn't scaleout when using time-slicing for GPUs #729
2. Mega Issue: Karpenter doesnt support custom resources requests/limit #751
3. Support hugepages-2Mi resource type aws/karpenter-provider-aws#3315

[jonathan-innis]

Closing this as a duplicate of #751. Do you mind adding your use-case to that larger issue?

[lzjqsdd]

Closing this as a duplicate of aws/karpenter-core#751. Do you mind adding your use-case to that larger issue?

ok