Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the ann-benchmark AWS instance deletion resilient. #282

Merged
merged 1 commit into from
Dec 2, 2024

Conversation

jfrancoa
Copy link
Contributor

@jfrancoa jfrancoa commented Dec 2, 2024

It was noted that some of the AWS instances didn't cleanup properly, incurring in extra costs from our AWS EC2 account. This commit makes sure that the instance gets deleted and if for whatever reason it didn't after 5 minutes, the job will fail. So that the person monitoring the job can take a look at the AWS' instances or escalate the issue to SRE.

It was noted that some of the AWS instances didn't cleanup
properly, incurring in extra costs from our AWS EC2 account.
This commit makes sure that the instance gets deleted and if
for whatever reason it didn't after 5 minutes, the job will
fail. So that the person monitoring the job can take a look
at the AWS' instances or escalate the issue to SRE.
Copy link

@orca-security-eu orca-security-eu bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@jfrancoa
Copy link
Contributor Author

jfrancoa commented Dec 2, 2024

@jfrancoa
Copy link
Contributor Author

jfrancoa commented Dec 2, 2024

Now the job waits for the instance to be deleted:

Terminating instance i-00800f7df71c5b6a7
{
  "TerminatingInstances": [
    {
      "InstanceId": "i-00800f7df71c5b6a7",
      "CurrentState": {
        "Code": 32,
        "Name": "shutting-down"
      },
      "PreviousState": {
        "Code": 16,
        "Name": "running"
      }
    }
  ]
}
Waiting for instance to terminate...
Instance status: shutting-down
Instance status: shutting-down
Instance status: shutting-down
Instance status: shutting-down
Instance status: shutting-down
Instance status: shutting-down
Instance status: shutting-down
Instance successfully terminated
Deleting key pair key-ef97cc3b-65f5-4db2-bfa6-25b7e32df96c
{
  "Return": true,
  "KeyPairId": "key-0e0ab78ece2851ae2"
}
Deleting security group sg-0c39dc3db77059627

@jfrancoa jfrancoa merged commit 706db1e into main Dec 2, 2024
113 checks passed
@jfrancoa jfrancoa deleted the jose/ensure-aws-instances-deletion branch December 2, 2024 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants