Cluster-autoscaler: AWS EC2 Spot Fleets support

[mumoshu]

Hi, thanks for developing cluster-autoscaler!

Is this something you'd like to add to cluster-autoscaler?

With spot fleets, we can easily mix various types of spot instances into a single logical group managed by AWS to benefit from reduced infrastructure cost and reduced operational burden including bidding, choosing which type of instance type to bid, increased availability via mixing mutliple spot instance types(assuming it will be rare that we lose to all the bids to all the different spot types at once).

A spot fleet has the notion of "target capacity" where "capacity" means number of units(a.k.a InstanceWeight) required to handle your workload e.g. 1 for r3.2xlarge, 4 for r3.8xlarge.
Basically I guess we can treat 1 target capacity = 1 instance weight as unit of autoscaling in cluster-autoscaler so that we can reuse a lot of current codebase to support spot fleets.

I'm both interested in using it and if necessary implementing it.
If you have any comments, advices, questions, etc., please let me know!

[andrewsykim]

I would be interested in helping as well, though as mentioned in #1921, the CA needs some sort of logic for choosing a node group which I assume would be a pre-requisite for this.

[mumoshu]

Hi @andrewsykim, I've been thinking of how we'd like cluster-autoscaler to select which node pool(asg or spot fleet) to expand for a few days.

I've now realized that the current "random" strategy plus "most-pods" and "least-waste" proposed in #2118 would work in most cases, as long as we choose the diversified strategy and launch specifications with enough "diversity" for our spot fleets, to keep our fleets up and running.

I'd rather like cluster-autoscaler to not select an autoscaling group which is suspended, or a spot fleet which is losing to bids for all the launch specifications.
However, that's an another issue IMHO.

[andrewsykim]

@mumoshu my org and I have built a new image wattpad/cluster-autoscaler:v1.2 with the changes in #2118. Trying to get as much testing in as possible, it would help this issue move forward. When you have time could you test it in any clusters you and your org are running?

[mumoshu]

@andrewsykim Thanks for sharing the image. Sure, I'd definitely like to test it!

[motymichaely]

@mumoshu How can we move this forward?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[zytek]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@zytek: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mumoshu]

I believe this is still relevant to every Kubernetes-on-AWS user who doesn't use a third-party solution like Spotinst.

[mumoshu]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@mumoshu: Reopening this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mumoshu]

The CA project has been moved to https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler. Could anyone open an another issue that replicates this there?

[zytek]

@mumoshu It's already opened: kubernetes/autoscaler#838
Please close this issue as I have asked for reopen by mistake. Sorry :)
/close