Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nuke EKS compute resources #234

Closed
wants to merge 2 commits into from
Closed

Conversation

yorinasub17
Copy link
Contributor

@yorinasub17 yorinasub17 commented Nov 12, 2021

Fixes #111 and #148

In my monthly AWS bill investigation, I keep having to manually nuke EKS clusters with clickops in our accounts that have Fargate Profiles or Node Groups and I don't want to do that anymore. This PR updates the EKS nuke routine to clean up those compute resources so that the EKS cluster can be nuked.

Note that in order to do so, I had to bump the AWS SDK version, which in turn meant that the mocks needed to be refreshed. That's where the changes to aws/mocks/EC2API.go is coming from.

@yorinasub17
Copy link
Contributor Author

Ran this in PHX and here is the output:

[cloud-nuke] INFO[2021-11-12T17:45:50-06:00] The following resource types will be nuked:
[cloud-nuke] INFO[2021-11-12T17:45:50-06:00] - ekscluster
[cloud-nuke] INFO[2021-11-12T17:45:51-06:00] Retrieving active AWS resources in [eu-north-1, ap-south-1, eu-west-3, eu-west-2, eu-west-1, ap-northeast-3, ap-northeast-2, ap-northeast-1, sa-east-1, ca-central-1, ap-southeast-1, ap-southeast-2, eu-central-1, us-east-1, us-east-2, us-west-1, us-west-2, global]
[cloud-nuke] INFO[2021-11-12T17:45:51-06:00] Checking region [1/18]: eu-north-1
[cloud-nuke] INFO[2021-11-12T17:45:52-06:00] Checking region [2/18]: ap-south-1
[cloud-nuke] INFO[2021-11-12T17:45:55-06:00] Checking region [3/18]: eu-west-3
[cloud-nuke] INFO[2021-11-12T17:45:56-06:00] Checking region [4/18]: eu-west-2
[cloud-nuke] INFO[2021-11-12T17:45:57-06:00] Checking region [5/18]: eu-west-1
[cloud-nuke] INFO[2021-11-12T17:46:01-06:00] Checking region [6/18]: ap-northeast-3
[cloud-nuke] INFO[2021-11-12T17:46:03-06:00] Checking region [7/18]: ap-northeast-2
[cloud-nuke] INFO[2021-11-12T17:46:05-06:00] Checking region [8/18]: ap-northeast-1
[cloud-nuke] INFO[2021-11-12T17:46:08-06:00] Checking region [9/18]: sa-east-1
[cloud-nuke] INFO[2021-11-12T17:46:09-06:00] Checking region [10/18]: ca-central-1
[cloud-nuke] INFO[2021-11-12T17:46:10-06:00] Checking region [11/18]: ap-southeast-1
[cloud-nuke] INFO[2021-11-12T17:46:13-06:00] Checking region [12/18]: ap-southeast-2
[cloud-nuke] INFO[2021-11-12T17:46:15-06:00] Checking region [13/18]: eu-central-1
[cloud-nuke] INFO[2021-11-12T17:46:16-06:00] Checking region [14/18]: us-east-1
[cloud-nuke] INFO[2021-11-12T17:46:17-06:00] Checking region [15/18]: us-east-2
[cloud-nuke] INFO[2021-11-12T17:46:19-06:00] Checking region [16/18]: us-west-1
[cloud-nuke] INFO[2021-11-12T17:46:20-06:00] Checking region [17/18]: us-west-2
[cloud-nuke] INFO[2021-11-12T17:46:21-06:00] Checking region [18/18]: global
[cloud-nuke] INFO[2021-11-12T17:46:21-06:00] The following 6 AWS resources will be nuked:
[cloud-nuke] INFO[2021-11-12T17:46:21-06:00] * ekscluster eks-service-catalog-xliopt eu-west-1
[cloud-nuke] INFO[2021-11-12T17:46:21-06:00] * ekscluster eks-service-catalog-xonfx6 eu-west-1
[cloud-nuke] INFO[2021-11-12T17:46:21-06:00] * ekscluster eks-service-catalog-dch7zl ap-northeast-1
[cloud-nuke] INFO[2021-11-12T17:46:21-06:00] * ekscluster eks-service-catalog-cjukfc us-east-1
[cloud-nuke] INFO[2021-11-12T17:46:21-06:00] * ekscluster eks-service-catalog-irj1b3 us-east-1
[cloud-nuke] INFO[2021-11-12T17:46:21-06:00] * ekscluster eks-service-catalog-fxvsuh us-east-1

THE NEXT STEPS ARE DESTRUCTIVE AND COMPLETELY IRREVERSIBLE, PROCEED WITH CAUTION!!!

Are you sure you want to nuke all listed resources? Enter 'nuke' to confirm (or exit with ^C): nuke
[cloud-nuke] INFO[2021-11-12T17:46:49-06:00] Terminating 2 resources in batches
[cloud-nuke] INFO[2021-11-12T17:46:49-06:00] Deleting 2 EKS clusters in region eu-west-1
[cloud-nuke] INFO[2021-11-12T17:49:52-06:00] Deleted Fargate Profile aws-auth-merger associated with cluster eks-service-catalog-xliopt
[cloud-nuke] INFO[2021-11-12T17:49:52-06:00] Deleted Fargate Profile aws-auth-merger associated with cluster eks-service-catalog-xonfx6
[cloud-nuke] INFO[2021-11-12T17:52:23-06:00] Deleted Fargate Profile control-plane-services associated with cluster eks-service-catalog-xonfx6
[cloud-nuke] INFO[2021-11-12T17:52:24-06:00] Deleted EKS cluster: eks-service-catalog-xliopt
[cloud-nuke] INFO[2021-11-12T17:54:56-06:00] Deleted EKS cluster: eks-service-catalog-xonfx6
[cloud-nuke] INFO[2021-11-12T17:54:56-06:00] [OK] 2 of 2 EKS cluster(s) deleted in eu-west-1
[cloud-nuke] INFO[2021-11-12T17:54:56-06:00] Terminating 1 resources in batches
[cloud-nuke] INFO[2021-11-12T17:54:56-06:00] Deleting 1 EKS clusters in region ap-northeast-1
[cloud-nuke] INFO[2021-11-12T17:57:59-06:00] Deleted Fargate Profile aws-auth-merger associated with cluster eks-service-catalog-dch7zl
[cloud-nuke] INFO[2021-11-12T17:58:00-06:00] Deleted Node Group eks-service-catalog-dch7zl-node_group associated with cluster eks-service-catalog-dch7zl
[cloud-nuke] INFO[2021-11-12T18:00:02-06:00] Deleted EKS cluster: eks-service-catalog-dch7zl
[cloud-nuke] INFO[2021-11-12T18:00:02-06:00] [OK] 1 of 1 EKS cluster(s) deleted in ap-northeast-1
[cloud-nuke] INFO[2021-11-12T18:00:02-06:00] Terminating 3 resources in batches
[cloud-nuke] INFO[2021-11-12T18:00:02-06:00] Deleting 3 EKS clusters in region us-east-1
[cloud-nuke] INFO[2021-11-12T18:03:04-06:00] Deleted Fargate Profile control-plane-services associated with cluster eks-service-catalog-cjukfc
[cloud-nuke] INFO[2021-11-12T18:03:04-06:00] Deleted Fargate Profile core-services associated with cluster eks-service-catalog-irj1b3
[cloud-nuke] INFO[2021-11-12T18:03:04-06:00] Deleted Fargate Profile control-plane-services associated with cluster eks-service-catalog-fxvsuh
[cloud-nuke] INFO[2021-11-12T18:05:37-06:00] Deleted EKS cluster: eks-service-catalog-cjukfc
[cloud-nuke] INFO[2021-11-12T18:05:37-06:00] Deleted EKS cluster: eks-service-catalog-irj1b3
[cloud-nuke] INFO[2021-11-12T18:05:37-06:00] Deleted EKS cluster: eks-service-catalog-fxvsuh
[cloud-nuke] INFO[2021-11-12T18:05:37-06:00] [OK] 3 of 3 EKS cluster(s) deleted in us-east-1

Copy link
Member

@brikis98 brikis98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment on lines +149 to +153
// We can't delete Fargate profiles in parallel, so we delete them sequentially, waiting until the profile has been
// deleted before moving on to the next one. This will make the delete routine very slow, but unfortunately, there
// is no other alternative.
// See https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteFargateProfile.html for more info on the serial
// requirement.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, what a bummer :(

@yorinasub17
Copy link
Contributor Author

Thanks for review! Merged as a part of #238

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants