aws: Allow rolling updates for ASGs

[radeksimko]

Once #1109 is fixed, I'd like to be able to use Terraform to actually roll out the updated launch configuration and do it carefully.

Whoever decides to not roll out the update and change the LC association only should still be allowed to do so.

Here's an example from CloudFormation:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatepolicy.html

How about something like this?

resource "aws_autoscaling_group" "test" {
  rolling_update_policy {
    max_batch_size = 1
    min_instances_in_service = 2
    pause_time = "PT0S"
    suspend_processes = ["launch", "terminate"]
    wait_on_resource_signals = true
  }
}

then if there's such policy defined, TF can use autoscaling API and shut down each EC2 instance separately and let ASG spin up a new one with an updated LC.

[pmoust]

👍

Our use case pretty much depends on this to be crystal clean. Other wise rolling updates require some minimal external intervention which I am working on making obsolete.

Huge +1 on this

[matthewford]

👍

[jessem]

👍 Likewise. Using a python script to handle this, which makes it a bit clunky to keep things simple with terraform.

[chrisferry]

👍

[phinze]

@radeksimko totally agreed that this is desirable behavior.

It's worth noting, though, that this is CloudFormation-specific behavior that's not exposed up at the AutoScaling API. I think the way we'd be able to achieve something similar would be to build some resources based on CodeDeploy:

https://docs.aws.amazon.com/codedeploy/latest/APIReference/Welcome.html

[johnrengelman]

My experience with code deploy is that it's limited to installing software on running instances...so it can roll out updates to ASG instances but it doesn't know how to do a roll-out by 'terminating' instances like CF does.
This would be an awesome feature, but I don't really think it's something that fits into Terraform's current model very well. Perhaps if there was a separate lifecycle type hook that actions could be plugged into.

So something like:

resource 'aws_launch_configuration' 'main' {
}

resource 'aws_autoscaling_group' 'main' {
  lifecycle {
    on_update {
      properties ['launch_configuration']
      actions {
        action 'aws_autoscaling_group_terminate_instances' {
          batch_size = 1
        }
      }
    }
  }
}

These actions then could be a whole separate concept in Terraform.

[woodhull]

👍 just realized this. did not realize that this was implemented by AWS as a cloudformation primitive rather than an ASG primitive we could hook into.

[woodhull]

Is anyone experimenting with using other tools to hack around this terraform limitation? Even if we were to combine terraform with external tooling like http://docs.ansible.com/ec2_asg_module.html I'm not sure where we'd hook in. A local-exec provisioner seems like the right thing -- but those are only triggered on creation, not modification. Maybe as a placeholder before implementing a native rolling update solution terraform could offer some sort of hook for launch configuration changes that we could use to trigger an external process?

Otherwise, I think we'll need to manage AMI version updates externally via ansible or some homegrown tool and then use terraform refresh to pull them in before doing plan or apply runs. It's all starting to drift away from the single-command infrastructure creation and mutation dream we had while starting to use terraform.

As part of our migration to terraform/packer away from a set of legacy tools we had been planning a deployment model based on updating pre-baked AMIs created with packer on a rolling basis. Any other ideas for workarounds that we could use until terraform is able to do this sort of thing out of the box?

[johnrengelman]

I've been using bash around the AWS cli. It would be awesome to implement tasks in Go against the awslabs library and then just call them from terraform though.

[woodhull]

Over the weekend I wrote some code to handle blue/green deploys for our particular use case in ruby, adding to our ever growing wrapper of custom code needed to actually make terraform useful.

Rather than hooking into terraform events it's ended up as a huge hack: it dynamically modifies the terraform plan to do the create of the new ASG with new AMI alongside the existing ASG. Then it uses ruby to make sure the new ASG is up and working correctly before removing the old ASG, regenerates the terraform plan file to match the new state, and finally calls terraform refresh so that the tfstate matches the new reality we've created.

Would be great if this sort of workflow for mutating a running app when we've updated an AMI were built in or if there were at least easy ways to hook into terraform to add custom behavior like this beyond learning Go and Terraform internals. In our case, even if terraform could handle the ASG operations for us, we'd still like to be able to run our quick, custom sanity check script to make sure everything is working properly on the new instances before removing the old ASG from the pool.

[woodhull]

This feature might be the most straightforward (although awkwardly round-about) way to get rolling deploys working inside terraform: #1083

[ketzacoatl]

+1.. this would make a huge difference in some of my current workflows

[d4v3y0rk]

+1

[ajmath]

+1

[blewa]

+1

[nathanielks]

@woodhull that wrapper wouldn't be around somewhere we could take a looksee, would it? 😄

[woodhull]

@nathanielks I copy/pasted some fragments of the code here: https://gist.github.com/woodhull/c56cbd0a68cb9b3fd1f4

It's wasn't designed for sharing or reusability, sorry about that! I hope it's enough of a code fragment to help.

[nathanielks]

@woodhull you're a gentlemen, thanks for sharing!

[BrunoBonacci]

+1

[saliceti]

+100

[ryan0x44]

If anyone is trying to achieve Blue-Green deployments with AWS Auto Scaling and Terraform, I've made a note about how I did it here. Not sure if it's an ideal solution, but it's working for me and proving to be very reliable at the moment :)

[nathanielks]

Thanks @ryandjurovich!

[ernoaapa]

👍

[farridav]

+1 just about to look at how to do this, would love to avoid writing a script to down and up instances to use the new Launch Configuration will take a look at @ryandjurovich's script also :)

[robmorgan]

👍

[brikis98]

@rvangundy Ah, I have mine inlined with a heredoc, which is probably why I didn't hit that issue. Good to know, thanks!

[shuoy]

Regarding "Immagine a 10 nodes db cluster with 10TB of data, spinning up a complete new cluster will cause the full resync of the 10TB of data from the old cluster to the new cluster all at once, this might saturate the network link and cause denial of service, where maybe all you wanted was to increase the number of connections."

@BrunoBonacci , we are facing the same situation about rolling update. Imagine we want to bump up one version of the software running on a data node, we need the kind of rolling update "in-place". It looks like the rolling update with TF is not going to get you there. Maybe we should consider something like Ansible to deal with that?

[BrunoBonacci]

@shuoy Certainly Ansible is a way to "script" this behaviour but the idea of using terraform (at least for me) is to replace previous scripting tools such as Chef, puppet, Ansible as so on.

I've seen different approaches to rolling update around. Kubernetes allow you the set a grace time to wait between the update of a machine to the next. This certainly could solve some of the issues, however it would work only for quite short grace time.
If you have 10TB to replicate in a cloud environment it could take a while, specially if you throttle the speed to avoid network saturation. Having a grace time of 10-20 hours wouldn't be reasonable.

I think the right approach it would be to provide an attribute which can be set to choose if the rolling update has to be performed automatically (based on grace period) or in stages.

something like:

## just a suggestion
rolling_update = "auto"
grace_period = "3m"

Which it will wait 3 minutes between a successful update to the next.

While the stage rolling update could work as follow:

## just a suggestion
rolling_update = "staged"
instances_updated = 1

In this case terraform would look at the ASG for example of 10 machines, and just update one.
Then the operator would wait for the new instance to be fully in service before
bumping the instances_updated = 2 and repeat terraform apply.
In such way the rolling update could be performed in stages giving enough time
to the app to replicate necessary data and become fully operative.
Once all 10 instances have been updated then update can be considered successful.

Again, this is just a suggestion to allow a declarative (non scripted) approach to rolling update,
which would work for stateful clusters (such as databases).

[shuoy]

@BrunoBonacci "If you have 10TB to replicate in a cloud environment it could take a while, specially if you throttle the speed to avoid network saturation. Having a grace time of 10-20 hours wouldn't be reasonable."
-- My point is this. For a node that following into this category, Chef/Ansible is exactly the tool you will be using to have an "in-place update way"; the destroying the old node and bringing up a new node (immutable way) is exactly what you want to avoid.
-- In the in-place-update way (chef/ansible), you basically apt-get update your mysqld (which take 1-2 minutes); yet the "destroy old and bring a new one way" (immutable way) takes hours unless you are using EBS volume. But I think you meant to say non-EBS way, or maybeI missed your point above.

Do we have scenario that large volume of data reside on ephemeral disk? Yes, e.g., for people use EC2 as their Hadoop cluster capacity, data is saved on ephemeral disk from cost saving perspective (EBS gives you extra cost).

So, in short, I think Terraform is great at certain scenarios, but it's not accurate that Chef/Ansible can be totally replaced. Particularly in the update use case for stateful nodes.

[jdoss]

@brikis98 and others that are using the output example posted here, it seems to not work if you upgrade to 0.7.0-rc2. Here is the Error that is kicking out:

* Resource 'aws_cloudformation_stack.web_autoscaling_group' does not have attribute 'outputs.AsgName' for variable 'aws_cloudformation_stack.web_autoscaling_group.outputs.AsgName'

I am still trying to get outputs working but if anyone has any advice on how to get this working again with 0.7.x that would be awesome.

[brikis98]

@jdoss You may want to file a separate bug agains the 0.7 release for that. The changelog mentions that for terraform_remote_state they no longer use the .output map. Perhaps they changed something similar on aws_cloudformation_stack but didn't document it?

[phinze]

@jdoss on first glance, it looks like that reference is using the pre-0.7 map dot-index notation. in 0.7, maps are indexed like aws_instance.tag["sometag"]

So try changing this line to use square bracket indexing and see if that fixes it for you!

  autoscaling_group_name = "${aws_cloudformation_stack.autoscaling_group.outputs["AsgName"]}"

If that doesn't work I'd welcome a fresh bug report and we can dig in. 👍

[jdoss]

@phinze You da best! That was the issue. 😄

[moofish32]

@rvangundy -- did you keep the lifecycle hook create_before_destroy=true or remove it? I think I am thrashing between this error and this #3916

[rvangundy]

@moofish32 I have create_before_destroy=true on my aws_launch_configuration

[antonbabenko]

@brikis98 - Thank you for the piece of CloudFormation code which does rolling deployments. It does not work for me, because I use spot instances in my launch configuration and when I specify MinInstancesInService greater than zero (because I really want to not terminate all instances before rolling) I have this error from AWS:

Autoscaling rolling updates cannot be performed because the current launch configuration is using spot instances and MinInstancesInService is greater than zero.

I wonder if anyone has any ideas how to implement ASG rolling update for spot instances using less moving parts?

My initial idea was to make a shell script which would work similarly to aws-ha-release.

I prefer to use just Terraform and CloudFormation and avoid implementation of orchestration magic.

UPD: Having MinInstancesInService=0 and MaxBatchSize=1 (which is smaller than amount of instances in ASG) helped. I am happy.

[iadknet]

+1

[shaharmor]

Is there any plan to implement this in the near/far future?

[james-gonzalez]

+1

[w32-blaster]

+1

[kenoir]

+1

[sthulb]

@phinze How about implementing this as a data source? it doesn't seem like the worst way of implementing it using current code structure.

I immediately regret this suggestion.

I don't mind implementing this feature, but I'm kind of at a loss on how to implement it in HCL. As the comment above would suggest, I thought about using the data source format, but linking it back to the ASG resource (Something like a UpdatePolicy = "${data.aws_asg_policy.foo}"; this concept doesn't seem horrid, but leaves room for interpretation.

Please advise on the most idiomatic way this could be represented in HCL.

[prog893]

+1

[andrleite]

+1

[sudochop]

+1

[addisonbair]

+1

[radeksimko]

Hi folks,
we do appreciate the +1's if these don't generate notifications. 😅

Therefore it's more helpful for everyone to use reactions as we can then sort issues by the number of 👍 :
https://github.com/hashicorp/terraform/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc+label%3Aprovider%2Faws

The 👍 reactions do count and we're more than happy for people to use those and prefer over "+1" comments for the mentioned reasons.

Thanks.

[w32-blaster]

👍

[apparentlymart]

Hi everyone,

While this sort of sophisticated behavior would definitely be useful, we (the Terraform team) don't have any short-term plans to implement this ourselves since we're generally recommending that people with these needs consider other solutions such as EC2 Container Service and Nomad, which either have or are more likely to grow sophisticated mechanisms for safe rollout and are in a better position to do so due to the ability to manage such multi-step state transitions.

We're trying to prune stale feature requests (that aren't likely to be addressed soon) by closing them. In this case we're currently leaning towards not implementing significant additional behaviors on top of what the EC2 API natively supports, so I'm going to close this.

[radeksimko]

After chatting with @apparentlymart privately I just want to add a few more things.

We do not suggest everyone should use containers (nor that containers solve the problem entirely) and for those who prefer not to, there's a workaround - you can use aws_cloudformation_stack similar way the folks at Travis did.

Also I'm tracking this issue in my own TODO list, to not forget about it. I personally want to get this done, but it's currently pretty low on my list of priorities, PRs are certainly welcomed.