Upgrading to v1.19.0 causes errors with EIP tag update

[idm-ryou]

After upgraded to CDK v1.19.0, updating existing EIP for adding auto-generated Tag causes an error at CloudFormation

Reproduction Steps

1. Deploy stack with Vpc construct with CDK earlier v1.19.0
2. Upgrade to CDK v1.19.0 and deploy

Error Log

$ cdk deploy NetworkStackEnvStg

NetworkStackEnvStg 
NetworkStackEnvStg: deploying... 
NetworkStackEnvStg: creating CloudFormation changeset... 
 0/6 | 2:07:22 AM | UPDATE_IN_PROGRESS   | AWS::EC2::EIP                         | Vpc/PublicSubnet1/EIP (VpcPublicSubnet1EIPD7E02669)  
 1/6 | 2:07:24 AM | UPDATE_FAILED        | AWS::EC2::EIP                         | Vpc/PublicSubnet1/EIP (VpcPublicSubnet1EIPD7E02669) API: ec2:disassociateAddress You do not have permission to access the specified resource. 
    PublicSubnet.addNatGateway (/codebuild/output/src566679633/src/node_modules/@aws-cdk/aws-ec2/lib/vpc.ts:1538:21) 
    \_ NatGateway.configureNat (/codebuild/output/src566679633/src/node_modules/@aws-cdk/aws-ec2/lib/nat.ts:145:27) 
    \_ Vpc.createNatGateways (/codebuild/output/src566679633/src/node_modules/@aws-cdk/aws-ec2/lib/vpc.ts:1129:14) 
    \_ new Vpc (/codebuild/output/src566679633/src/node_modules/@aws-cdk/aws-ec2/lib/vpc.ts:1044:14) 
    \_ new NetworkStack (/codebuild/output/src566679633/src/src/network-stack.ts:20:16) 
    \_ newStack (/codebuild/output/src566679633/src/src/app.ts:52:28) 
    \_ Records.mapValues (/codebuild/output/src566679633/src/src/lib.ts:62:5) 
    \_ result.toEntries.map (/codebuild/output/src566679633/src/src/lib.ts:44:31) 
    \_ Array.map (<anonymous>) 
    \_ Function.mapValues (/codebuild/output/src566679633/src/src/lib.ts:43:43) 
    \_ Object.newStackGroup (/codebuild/output/src566679633/src/src/lib.ts:61:18) 
    \_ Object.<anonymous> (/codebuild/output/src566679633/src/src/app.ts:49:27) 
    \_ Module._compile (internal/modules/cjs/loader.js:778:30) 
    \_ Module.m._compile (/codebuild/output/src566679633/src/node_modules/ts-node/src/index.ts:536:23) 
    \_ Module._extensions..js (internal/modules/cjs/loader.js:789:10) 
    \_ Object.require.extensions.(anonymous function) [as .ts] (/codebuild/output/src566679633/src/node_modules/ts-node/src/index.ts:539:12) 
    \_ Module.load (internal/modules/cjs/loader.js:653:32) 
    \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12) 
    \_ Function.Module._load (internal/modules/cjs/loader.js:585:3) 
    \_ Function.Module.runMain (internal/modules/cjs/loader.js:831:12) 
    \_ main (/codebuild/output/src566679633/src/node_modules/ts-node/src/bin.ts:212:14) 
    \_ Object.<anonymous> (/codebuild/output/src566679633/src/node_modules/ts-node/src/bin.ts:470:3) 
    \_ Module._compile (internal/modules/cjs/loader.js:778:30) 
    \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10) 
    \_ Module.load (internal/modules/cjs/loader.js:653:32) 
    \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12) 
    \_ Function.Module._load (internal/modules/cjs/loader.js:585:3) 
    \_ Function.Module.runMain (internal/modules/cjs/loader.js:831:12) 
    \_ startup (internal/bootstrap/node.js:283:19) 
    \_ bootstrapNodeJSCore (internal/bootstrap/node.js:622:3) 
 1/6 | 2:07:25 AM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack            | NetworkStackEnvStg The following resource(s) failed to update: [VpcPublicSubnet1EIPD7E02669].  
 1/6 | 2:07:57 AM | UPDATE_IN_PROGRESS   | AWS::EC2::EIP                         | Vpc/PublicSubnet1/EIP (VpcPublicSubnet1EIPD7E02669)  
 
 ❌  NetworkStackEnvStg failed: Error: The stack named NetworkStackEnvStg is in a failed state: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [VpcPublicSubnet1EIPD7E02669]. ) 
The stack named NetworkStackEnvStg is in a failed state: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [VpcPublicSubnet1EIPD7E02669]. ) 
error Command failed with exit code 1. 
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. 

Environment

• CLI Version : 1.19.0 (build 5597bbe)
• Framework Version: 1.19.0
• OS : Linux(on a CodeBuild project environment)
• Language : TypeScript

Other

$ cdk diff NetworkStackEnvStg

Stack NetworkStackEnvStg 
Resources 
[~] AWS::EC2::EIP Vpc/PublicSubnet1/EIP VpcPublicSubnet1EIPD7E02669  
 └─ [+] Tags 
     └─ [{"Key":"Name","Value":"NetworkStackEnvStg/Vpc/PublicSubnet1"}] 

Related: aws-cloudformation/cloudformation-coverage-roadmap#84 (comment)

---

This is 🐛 Bug Report

[idm-ryou]

Workaround:

cdk.Tag.remove(this.vpc, "Name", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

[danly]

I am having the same issue. After this upgrade, my VPC subnets are adding a new tag to each subnet. I am having the:
API: ec2:disassociateAddress You do not have permission to access the specified resource.

The rollback then proceeds to fail as each subnet fails to update.

[crucialfelix]

"UPDATE_ROLLBACK_FAILED state and can not be updated" I had to initiate "Continue update rollback" from the console. I had to skip changing those resources.

So now they are out of sync.

I still get the same API: ec2:disassociateAddress error on subsequent changes. With the workaround #5469 (comment) at least it can roll back.

So I will return to 1.18.0 for now.

[gbooth27]

I am also seeing this bug after upgrading to 1.19.0

[crucialfelix]

I did get it to deploy still using 1.19.0 actually.

You also have to remove any other tags that might be assigned by parent constructs. In my case I had source: cdk and stage: staging so I now have code to remove all tags from EIP:

cdk.Tag.remove(this.vpc, "Name", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

cdk.Tag.remove(this.vpc, "source", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

cdk.Tag.remove(this.vpc, "stage", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

[idm-ryou]

Related Issue:
aws-cloudformation/cloudformation-coverage-roadmap#309

[mark-schaal]

I am also seeing this bug after upgrading to 1.19.0.

Update 1: Confirming the recommendation by @idm-ryou to manually remove the tag is a good temp workaround.

Update 2: In complex environments, this is causing cascading errors for all tagged resources. I've reached a point where a multi-VPC application cannot be deployed even when forcibly removing all tags attached with the following code:

 Tag.remove(vpc, "Name", {
        includeResourceTypes: [],
        priority: 300
});

Update 3: In complex or multi-stack environments, I found the easiest way to find all issues with EIP tags is to run the command cdk diff in your local project. In my case, I found that an EKS cluster further downstream was also modifying custom k8s tags on the Subnet, inadvertently impacting the EIP resource.

[nathanpeck]

@rix0rrr I also encountered this bug. I ended up just wiping my stack and recreating with 1.19.0

[lkoniecz]

@rix0rrr
Any ETA on this?

thanks

[njlynch]

From the associated issues, it looks like this was resolved by CloudFormation on Mar 6 2020 (15 months ago!).

I also just attempted to recreate it by creating a new CDK project on 1.18.0, adding a VPC (with EIP), then upgrading the project to 1.19.0 and deploying again. The diff showed the Tags being added, and the deployment succeeded as expected.

Based on the above, I believe this can be closed. I'm going to mark as "Closing Soon" and give folks a few days to comment if they're still impacted. If you're still being bit by this, please comment (ideally with a reproducible example) so we can continue troubleshooting.

[peterwoodworth]

Closing because the bot isn't working, please ping me if anyone is still impacted by this issue

-Peter

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.