Added OnDemand and Spot Price models addressing #131 #486

mrcrgl · 2017-11-23T10:27:00Z

No description provided.

…vity of the offer provided by the aws pricing api

# Conflicts: # cluster-autoscaler/cloudprovider/aws/ec2_instance_types.go

mrcrgl · 2017-11-23T10:29:24Z

Re-opened PR #484
Related to: #131

mwielgus · 2017-11-23T11:00:25Z

cluster-autoscaler/cloudprovider/aws/api/converter.go

+
+import "strconv"
+
+// stringRefToFloat64 converts fields of type *float64 to float64 (fallback to zero value if nil)


string to float?

Yes, it's used when transforming AWS API DTO's to local ones.
For instance at this place: https://github.com/kubernetes/autoscaler/pull/486/files#diff-fc9a082c5fda90baeac6f105e7a018b9R61

Would you suggest to not put it in a function?

i think @mwielgus was referring to the comment.

To be more specific, I believe this is the change wanted:

of type *float64 to float64 => of type *string to float64

mwielgus · 2017-11-23T11:02:34Z

cluster-autoscaler/cloudprovider/aws/api/converter.go

+
+// stringRefToFloat64 converts fields of type *float64 to float64 (fallback to zero value if nil)
+// it's mostly used to handle aws api responds nicely
+func stringRefToFloat64(p *string) (float64, error) {


What is the point of using *string. Strings are already pointers (slices).

It's the way AWS API's are designed, all values are pointers in their DTO's to carry the information whether the value is provided or not.
do you think it's more transparent for the reader to make that conversion on caller side?

There is a package aws.String which lets you do all these type of conversion. You can use it instead of *string. https://docs.aws.amazon.com/sdk-for-go/api/aws/

mwielgus · 2017-11-23T11:51:23Z

cluster-autoscaler/cloudprovider/aws/api/ec2_launchconfiguration.go

+
+// EC2LaunchConfiguration holds AWS EC2 Launch Configuration information
+type EC2LaunchConfiguration struct {
+	HasSpotMarkedBid bool


Please add comments to the fields.

k8s-ci-robot · 2017-11-23T11:51:27Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://github.com/kubernetes/kubernetes/wiki/CLA-FAQ to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

mwielgus · 2017-11-23T12:19:22Z

This PR is HUGE and not reviewable in the current form.

Please:

Provide a detailed explanation what are you doing and how to get the price for both regular and spot instances, so that your code can be validated against the description.
Consider splitting the PR into smaller PRs. PRs that are <200 LOC are MUCH easier to understand.
Maybe spot / non spot. Maybe submit some utils first.

mwielgus · 2017-11-23T12:24:30Z

cluster-autoscaler/cloudprovider/aws/api/ec2_launchconfiguration_test.go

+}
+
+func newLCFakeService(
+	name string,


single line

mwielgus · 2017-11-23T12:25:16Z

cluster-autoscaler/cloudprovider/aws/api/ec2_launchconfiguration_test.go

+	}
+	type cases []testCase
+
+	var (


no need for var, just :=

mwielgus · 2017-11-23T12:25:37Z

cluster-autoscaler/cloudprovider/aws/api/ec2_autoscaling.go

+
+// EC2AutoscalingGroup holds AWS Autoscaling Group information
+type EC2AutoscalingGroup struct {
+	Name                    string


mwielgus · 2017-11-23T12:28:21Z

cluster-autoscaler/cloudprovider/aws/api/instance_info.go

+
+// InstanceInfo holds AWS EC2 instance information
+type InstanceInfo struct {
+	InstanceType  string


mwielgus · 2017-11-23T12:30:06Z

cluster-autoscaler/cloudprovider/aws/api/instance_info.go

+type instanceInfoService struct {
+	client httpClient
+	cache  instanceInfoCache
+	mu     sync.RWMutex


don't use single/two letter names for field in struct.
moreover you can do
type xxxx struct {
sync.RwMutex // <<<NO FIELD NAME.
}
and then just xxx.Lock().

mwielgus · 2017-11-23T12:30:54Z

cluster-autoscaler/cloudprovider/aws/api/instance_info.go

+
+type productOffers map[string]productOffer
+
+type productOffer struct {


Can we get these from some official library?

I didn't find anything about it, just started here (https://aws.amazon.com/blogs/aws/new-aws-price-list-api/) and moved over

Is the same of http://docs.aws.amazon.com/sdk-for-go/api/service/pricing/?

Good point, I'll check that

Has this been checked?

Not yet in detail, I need to find some time. Bug briefly checked that it could be replaced without too many code changes.

+1 on using the pricing API

mwielgus · 2017-11-23T12:31:18Z

cluster-autoscaler/cloudprovider/aws/api/instance_info.go

+}
+
+type productPriceDimension struct {
+	RateCode     string       `json:"rateCode"`


either all privte or all public.

Here and elsewhere.

In my opinion, the struct productPriceDimension is just a local DTO used to represent the json structure. Therefore, it's not part of packages api and doesn't need to be kept stable/backward compatible. Unfortunately, the marshaller is not able to read private fields of foreign structs.
This is the reason why I've decided to make it like this. Does that work for you?

ok, if we are super sure that this cannot be done with the official library.

mwielgus · 2017-11-23T12:33:07Z

cc: @mumoshu @sethpollack

mrcrgl · 2017-11-23T13:41:14Z

@mwielgus thank you for the feedback concerning the PR size. I consider to create some kind of documentation how the code was meant to be structured and how it should work.

mwielgus · 2019-03-01T11:41:53Z

@Jeffwan - How do we proceed on this?

mwielgus · 2019-03-22T01:12:17Z

Inactivity warning. Please conclude the review and PR fixes.

mwielgus · 2019-03-26T23:20:19Z

Second inactivity warning.

mrcrgl · 2019-03-27T07:39:25Z

@mwielgus I wrote you on Slack

mwielgus · 2019-03-27T09:42:08Z

@mrcrgl I answered you on Slack :).

ylallemant · 2019-04-03T08:07:59Z

@mwielgus we will attend the AWS Special Interest Group meeting next Friday to discus this PR and the necessary steps to get it merged

k8s-ci-robot · 2019-04-03T08:08:48Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

mwielgus · 2019-04-03T09:48:53Z

@ylallemant Cool! Good luck :).

happilymarrieddad · 2019-04-05T21:26:49Z

Is this going to be a thing? That would save so much money!

ghost · 2019-04-06T11:52:38Z

Hi @happilymarrieddad - we've discussed this yesterday at the sig-aws meeting. People seem interested in getting this merged (especially us) but it might require a bit more time, that being said we've been running this code in production for close to 2 years. As we mentioned in the meeting, the only major downside of running it right now is that you have to manually scale if an ASG would have the lowest price but the instance type is not available. This happens maybe 2-3 times a year.

bhack · 2019-04-06T12:41:10Z

In the meantime https://github.com/AutoSpotting/AutoSpotting

drewhemm · 2019-04-12T12:48:11Z

Hi guys,

I just opened PR 1886 to add support for ASG MixedInstancesPolicy. Whilst it doesn't attempt to perform calculations on the different costs of on-demand and spot instances, what it does allow is for a LaunchTemplate to have an instance type defined, but that can be overridden when creating the ASG.

There are more details in the PR conversation.

mwielgus · 2019-04-24T16:25:38Z

Inactivity warning. Please proceed with the review or this PR will be closed.

mwielgus · 2019-05-01T21:12:33Z

Inactivity warning. Please proceed with the review or this PR will be closed.

happilymarrieddad · 2019-05-02T14:41:12Z

Oh man, I hope someone, when they get a chance, gets this moving. I'm super excited about it!

mcrute · 2019-05-03T15:39:51Z

@mrcrgl I'm going to pick this up from @Jeffwan from the AWS side. Would you mind rebasing and updating this PR?

ghost · 2019-05-03T17:13:07Z

cc: @ylallemant

losipiuk · 2019-05-14T11:37:01Z

This PR is very stale. Closing for now due to inactivity. Feel free to reopen if there is followup work planned on this.

bhack · 2019-05-14T12:34:11Z

I don't understand aws sig approach on this.
Are Amazon aws sig members against this? Is this going to conflict the pricing model?

max-rocket-internet · 2019-05-14T13:15:35Z

A lot has changed since 2017. It's not really related to the cluster-autoscaler itself, but like @drewhemm mentioned, with AWS Launch Templates and instance type overrides on the autoscaling group, the usability of spot instances with k8s on AWS has increased a lot. With this configuration the ASG will choose a selection of the cheapest spot instance types and spread them across AZs. Here's example configuration in Terraform using AWS EKS. This won't make everyone happy but it is a vast improvement for many people who were are waiting for this feature 🙂

ghost · 2019-05-14T13:34:22Z

🙄

losipiuk · 2019-05-14T13:44:59Z

I don't understand aws sig approach on this.
Are Amazon aws sig members against this? Is this going to conflict the pricing model?

We are not. We just not have enough context on details of implementation of AWS cloud provider.
As I said. Feel free to reopen. But we need an active AWS CP contributor to take ownership of that and make actual in depth code review.

mrcrgl · 2019-05-14T21:10:35Z

I’m really disappointed about this. We’ve invested many days of work into this change and were more than patient with the review. At the end, you are telling that this PR is laying too long? But why is it so hard getting a review from someone who’s responsible?

This is not the way you earn motivated contributors.

Jeffwan · 2019-05-15T06:45:43Z

I don't understand aws sig approach on this.
Are Amazon aws sig members against this? Is this going to conflict the pricing model?

@bhack

We're not, we did encourage people to come to sig-autoscaling and sig-aws to bring more use cases and feedbacks. One concern was price model in this PR is a hack way and it may not work for all the cases, the other one was lot of users said they don't like to manage many spot ASGs.

At the same time, as @max-rocket-internet said, Launch Template provides capability to bring onDemand and Spot instances in one ASG group. ASG guarantees users have cheapest spot instance type all the time. PR changes is here. #1886. docs #1983 To achieve most of users want from this PR, MixedInstancePolicy is another option.

Jeffwan · 2019-05-15T07:01:19Z

I’m really disappointed about this. We’ve invested many days of work into this change and were more than patient with the review. At the end, you are telling that this PR is laying too long? But why is it so hard getting a review from someone who’s responsible?

This is not the way you earn motivated contributors.

I feel really sorry this PR stays here for long time and need rebase and rebase all the time. In the past, I think CA didn't have active AWS CP contributors.. AWS EKS only has <1 yr history, we are trying to put more efforts here and help address users problem. Community development definitely needs to engage powers from all contributors.

I am writing a doc to compare MixedInstancePolicy vs Native Spot vs Spot Fleet for this use case.
Let's see if there's a good solution to address all the spot requirements and I will share with all in two weeks. At the same time, please check PR #1886 and #1983 and see if they can meet your needs, If you need more info, I can record a demo for this.

mrcrgl added 4 commits November 22, 2017 22:41

Added OnDemand and Spot Price models addressing kubernetes#131

1e548ee

gofmt -s

5610570

Fixed broken instance-info test; Added areness of tenancy and effecti…

8a923bc

…vity of the offer provided by the aws pricing api

Merge branch 'master' into issue/131

c214eae

# Conflicts: # cluster-autoscaler/cloudprovider/aws/ec2_instance_types.go

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 23, 2017

mrcrgl changed the title ~~Issue/131~~ Added OnDemand and Spot Price models addressing #131 Nov 23, 2017

mwielgus reviewed Nov 23, 2017

View reviewed changes

Add comments to conversion functions to clarify the use

bb80b4f

mwielgus reviewed Nov 23, 2017

View reviewed changes

mrcrgl force-pushed the issue/131 branch from 41f42b5 to bb80b4f Compare November 23, 2017 11:51

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 23, 2017

Added field comments to EC2LaunchConfiguration

56d8fd3

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Nov 23, 2017

mwielgus reviewed Nov 23, 2017

View reviewed changes

Added comments and apot review suggestions

2924a1a

aleksandra-malinowska mentioned this pull request Feb 21, 2019

Ability to tag priorities to different ASG's for preferring particular array. #1709

Closed

viggeh mentioned this pull request Mar 8, 2019

[Feature Idea] A simple expander based on static precedence #1772

Closed

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 3, 2019

losipiuk closed this May 14, 2019


		import "strconv"

		// stringRefToFloat64 converts fields of type *float64 to float64 (fallback to zero value if nil)


		type productOffers map[string]productOffer

		type productOffer struct {

Added OnDemand and Spot Price models addressing #131 #486

Added OnDemand and Spot Price models addressing #131 #486

Conversation

mrcrgl commented Nov 23, 2017

mrcrgl commented Nov 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 23, 2017

mwielgus commented Nov 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwielgus commented Nov 23, 2017

mrcrgl commented Nov 23, 2017

mwielgus commented Mar 1, 2019

mwielgus commented Mar 22, 2019

mwielgus commented Mar 26, 2019

mrcrgl commented Mar 27, 2019

mwielgus commented Mar 27, 2019

ylallemant commented Apr 3, 2019

k8s-ci-robot commented Apr 3, 2019

mwielgus commented Apr 3, 2019

happilymarrieddad commented Apr 5, 2019

ghost commented Apr 6, 2019

bhack commented Apr 6, 2019

drewhemm commented Apr 12, 2019

mwielgus commented Apr 24, 2019

mwielgus commented May 1, 2019

happilymarrieddad commented May 2, 2019

mcrute commented May 3, 2019

ghost commented May 3, 2019

losipiuk commented May 14, 2019

bhack commented May 14, 2019

max-rocket-internet commented May 14, 2019

ghost commented May 14, 2019

losipiuk commented May 14, 2019

mrcrgl commented May 14, 2019

Jeffwan commented May 15, 2019 • edited Loading

Jeffwan commented May 15, 2019 • edited Loading

mrcrgl commented Nov 23, 2017 •

edited

Loading

mwielgus commented Nov 23, 2017 •

edited

Loading

Jeffwan commented May 15, 2019 •

edited

Loading

Jeffwan commented May 15, 2019 •

edited

Loading