Skip to content
This repository has been archived by the owner on Oct 15, 2024. It is now read-only.

Add another layer to the cache to keep regions separated #454

Merged
merged 1 commit into from
Jan 22, 2020

Conversation

sas-pemcne
Copy link
Contributor

This fix corrects an issue where the nuker does not correctly handle resources from multiple regions in a single run.

Currently, the nuker logic, specifically in the HandleWait function, is as follows:

  1. Try and get the item type out of the cache
  2. If the item type does not exist, list the resources and add to the cache
  3. Loop through cached resources and try to find a match for our item id
  4. If our item isn't in the cache, it has been deleted

The flaw in this design is that the item.List(), called in step 2 above, only lists items for a single region (assuming the resource is region specific). Because of this, the cache will not be accurate for any region outside of the region that the item was in.

Consider the following scenario, nuker runs on two regions: us-east-1 and us-east-2. us-east-1 has a single VPC and us-east-2 has a VPC with dependencies that cannot be deleted.

  1. In us-east-1, EC2VPC is processed. Given the logic above a cache is built
  2. Cache ends up being an empty array because the VPC was deleted successfully on first pass
  3. us-east-2 is processed next. Given the logic above, EC2VPC exists in the cache, the empty array is pulled, and no cache updates were performed
  4. The VPC does not exist in the cache, thus it assumes it was deleted
  5. Nuker completes successfully thinking all items were complete

Because of this, I was getting errors very similar to the following where the debug clearly shows a 400 response from AWS but the nuker thinks the resource has been removed because of the incorrect cache.

Action=DeleteVpc&Version=2016-11-15&VpcId=vpc-123

time="2019-12-20T11:18:56Z" level=debug msg="received AWS response:
< HTTP/1.1 400 Bad Request ... <Error><Code>DependencyViolation</Code><Message>The vpc 'vpc-123' has dependencies and cannot be deleted.</Message></Error> ..."

us-east-1 - EC2VPC - vpc-123 - [ID: "vpc-123", IsDefault: "false"] - removed

My fix was to simply add another layer onto the cache and have all cached items reside in their respective region. The following is the result of the cache structure after my fix:

{
  'us-east-1': {
    'EC2VPC': [],
    'EC2Instance': [],
    ...
  },
  'us-east-2': {
    'EC2Instance': [],
    ...
  },
  ...
}

@sas-pemcne sas-pemcne requested a review from a team January 21, 2020 12:53
@svenwltr svenwltr added the kind/enhancement New core feature or improvement of existing ones. label Jan 22, 2020
@svenwltr svenwltr merged commit 41e232c into rebuy-de:master Jan 22, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/enhancement New core feature or improvement of existing ones.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants