Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE REQUEST] Instances already terminated before ASG lifecycle hook runs #452

Open
modbender opened this issue Dec 7, 2024 · 3 comments

Comments

@modbender
Copy link

Is your feature request related to a problem? Please describe.

In our company Jenkins agent workspace has some important files that are required for debugging. I have put a lifecycle hook that copies files from workspace location to s3 before termination. Setup is simple, hook triggers SNS notification, and SNS triggers lambda. Lambda runs a send_command on EC2 instance and copies data to specified s3 bucket. But the send_command keeps giving me error saying instance is not in a valid state.
So when checked I found out that the instance is already in terminated state even though ASG page says Terminating:Wait.

I have already seen the duplicate issue here: #216

Describe the solution you'd like

So I have seen the code which calls terminatInstances. A very simple solution is to switch to using TerminateInstanceInAutoScalingGroup. In this you can specify which instance is to be terminated by it's instance ID and if ShouldDecrementDesiredCapacity is set to true then desired capacity is set accordingly by AWS.
https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_TerminateInstanceInAutoScalingGroup.html

Describe alternatives you've considered

Alternates I have considered are:

  1. Run s3 copy script every 15 mins using cron on EC2 instance.
  2. Experimenting with Cloud Status Interval in sec
@modbender
Copy link
Author

@GavinBurris42 I see that you are the one who last merged PR on this repo, could you please consider the above? It's a real simple change.

@modbender
Copy link
Author

@vineeth-bandi Could you check this request ^

@modbender
Copy link
Author

@pdk27 could you help with this request here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant