Add support for other kinds of Daemonset Kube Resource (CRD)

[clamoriniere]

component: cluster-autoscaler

Issue description

The cluster-autoscaler uses the Daemonsets spec to simulate the resources needed for the Daemonset’s pods on a new node.
We are looking to extend the capability of the cluster-autoscaler in terms of watching for other resources that implement an equivalent of a Daemonset (one Pod deployed on each Node).

We have the following established open-source project github.com/DataDog/ExtendedDaemonset that defines a CRD (ExtendedDaemonset) that replaces the Daemonset object for some of our applications that need to be deployed on all nodes. One of the reasons that we have worked on this project is to allow a canary deployment on a subset of nodes,as well as have more control over the rollout of an application (velocity, pausing…).

We made this previous contribution that allows the cluster-autoscaler controller to recognize the Pods created by the ExtendedDaemonset as Daemonset’s Pod.

Now the idea is to provide a way for the controller to consider other kinds of Daemonset resources when it estimates the Daemonsets’ pod resources usage in the GetDaemonSetPodsForNode() function

Proposed solutions

To make it possible, we identified two ways to do it:

1. Add the possibility for the cluster-autoscaler controller to look at PodTemplates resources in addition to the Daemonsets. To avoid querying PodTemplates that are not linked to our use case, we can limit the query with a LabelSelector on a dedicated label like cluster-autoscaler.kubernetes.io/daemonset-pod. Then we can simulate a pod creation like for the Daemonset.

2. Add the possibility to list any kind of resource that contains a PodTemplate in its spec section.

   We can imagine providing a list of resource metadata (kind, apiVersion) as controller parameters. Attach to each resource, also the path to the PodTemplate in the spec section.

   daemonsetResources:
   - kind: CustomeDaemonset
     apiVersion: foo.io/v1
     podTemplate:
       path: .spec.template
       kind: PodTemplate
       apiVersion: v1

   Then, using the dynamic goclient, we can query the api-server to get the list of resources, and then decoding the object, access the PodTemplate section.

Questions

• What do you think about this feature?

• Which implementation solution makes more sense for you? Do you see any other alternatives?

[elmiko]

What do you think about this feature?

this feature seems reasonable to me

Which implementation solution makes more sense for you? Do you see any other alternatives?

i like that option 1 uses labels to help further refine the search, i would be ok with something that could look at any resource with a PodTemplate in its spec, provided that the "anything" has a label or annotation which marks it as such.

i guess that means i like parts of both implementations presented. maybe this is another alternative, eg any object with cluster-autoscaler.kubernetes.io/daemonset-pod label, provided it contains a PodTemplate in the spec.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.