pipelines: multiple synths in a 'for' loop with a custom lookup role stopped working in 1.103 #15130
Comments
mrpackethead
added
bug
|
@skinny85 , have sent you via private message in cdk.dev slack codebuild log. It contains customer senstive information so cant' post it publically. Accounts removed from post below. |
|
skinny85
added
@aws-cdk/pipelines
github-actions
bot
added
@aws-cdk/aws-codebuild
mrpackethead
changed the title
mrpackethead
changed the title
|
The reason this is happening is because there is context information missing from How did this use to work in the past? Your |
|
It absolutely did syntheisese and deploy correctly in versions lower than 1.102 and i am able to demonstrate that to you. It is the way that i showed you in our call much ealier this year. Happy to show you the pipleines that actually build and deploy. So, not helpful to say ' It could not deploy '. We've been deploying this way for months! Our code build is interating over the manifest, and synthing against each cloud assembly. It assumes a role in the remote account, and does a synth against the account using an account. Its EXACTLY what happens on the desktop but its automated. cdk.context. json The code build does a intital cdk synth, this creates all the tempates, but cdk.context.json was not populated. However all the manifests were created. This has caused me significnat pain, and i cna't find any release notes that talk about it in in vs1.103. I've had to go and pin a whole bunch of stacks today.. to get them to work, ( after spending the best part of a day to figure out what was going on ). |
|
My best guess is this is where things have gone badly. |
|
This issue is also affecting us, it seems we have a similar setup to @mrpackethead, where our pipeline account has the required cross account permissions for deployment and in general our devs do not. Ideally we don't want to give devs access to production accounts just so we can allow the pipeline to work. It was also working for us prior to the latest releases |
|
I see. You all have scripted a That is fine, but I would repeat that standing guidance was to:
You are correct. You are running afoul of a validation intended to protect against incomplete This didn't use to fail, and in your particular cases (with your A potential simple workaround would be:
But read on for a potential better solution. Your custom scripting may not be necessary anymore.
Sure. Would you consider giving them read-only access? In #14874 we introduced a new version of the bootstrap template, adding a This change introduces a new RecommendationOur current recommendation would be:
Fallback (not recommended but closer to your current setups)We know there are organizations who under no circumstances want to see Since Be aware that this setup opens you up to risk of nondeterminism! If any lookup value changes, this will immediately impact your infrastructure in a way you may not be prepared for! If AWS opens a new AZ tomorrow, your VPC will try to spread over more AZs and your deployments will get stuck because the existing subnets need to be replaced. If a new AMI gets released it will be immediately picked up and all of your instances will be replaced (make sure there's no important state on them!). This is the reason we recommend committing To be less susceptible to nondeterminism you should store the Let me know if this is guidance you cannot follow, for whatever reason. |
rix0rrr
changed the title
|
@rix0rrr, it appears in our case the issue occurs when the VPC construct is looking up the availability zones for the current region. Which isn't an explicit import in our case, but appears to trip up the new changes made to the pipeline. I do like the addition of the |
|
Not using lookups in pipelines would have rendered CDK in the useless pile to us.
A Validation htat never exisited before, and has introduced a breaking change. We coded around the KNOWN behavior.
How do we do that? I dont' have any control of how synth validates the code......
Even if i wanted, to, which i dont' ( and probalby cant ) ,its a deeper problem than that. Some of our apps dont' know what environments they are going to deploy to. Sounds crazy.? We make extensive use of Tags in our environments. A great example, we tag accounts in our organisation.. Our app will look up the tags attached to the acount to determine if something should be deployed in that account. ( for example, some roles ) . a
As described, it requires that (a) devs have access, and (b) that the environments are determined before synth time.
The permissions that the lookup role give don't provide enough permissions to do the lookups.. Why is it so restricted?
Our pipeline synths everything now, and commits cdk.context.json back to the repo, ( and tags it ) when its done.
WE discussed this months ago. WE do not risk nondterminism. MY lookups are all determinisistic. ( the ones that we do want to be at least ), they all get committed to cdk.context.json by the codebuild thats doing the synths.
We are going around in circles, and i'm really very dissapointed.. What i need really quickly, is a way to turn off the 'validation' so we can move forward quickly.. and then as a second step we can take our time to review these breaking changes. |
We are working on that right now. Hope to get it out later today. Stay tuned. |
In #14613, we introduced validation so that `cdk synth` would guard against incomplete context lookups. In the past, stacks with missing context would have successfully completed synthesis, propagated down the pipeline and caused confusing error messages during deployment. The new behavior was to error out early in case there was missing context. This broke people who had resorted to resolving context in the pipeline using multiple `synth` commands in `for`-loop: this used to work because the `synths` would be incomplete but silently succeed, but with the new validation the very first `cdk synth` would start failing and the `for` loop would never complete. This PR adds a `--no-validation` flag to `cdk synth` to stop the additional validation, so the `for` loop can complete successfully. The same behavior can be controlled with an environment variable, by setting `CDK_VALIDATION=false`. Fixes #15130.
|
We added |
In #14613, we introduced validation so that `cdk synth` would guard against incomplete context lookups. In the past, stacks with missing context would have successfully completed synthesis, propagated down the pipeline and caused confusing error messages during deployment. The new behavior was to error out early in case there was missing context. This broke people who had resorted to resolving context in the pipeline using multiple `synth` commands in `for`-loop: this used to work because the `synths` would be incomplete but silently succeed, but with the new validation the very first `cdk synth` would start failing and the `for` loop would never complete. This PR adds a `--no-validation` flag to `cdk synth` to stop the additional validation, so the `for` loop can complete successfully. The same behavior can be controlled with an environment variable, by setting `CDK_VALIDATION=false`. Fixes #15130.
|
In aws#14613, we introduced validation so that `cdk synth` would guard against incomplete context lookups. In the past, stacks with missing context would have successfully completed synthesis, propagated down the pipeline and caused confusing error messages during deployment. The new behavior was to error out early in case there was missing context. This broke people who had resorted to resolving context in the pipeline using multiple `synth` commands in `for`-loop: this used to work because the `synths` would be incomplete but silently succeed, but with the new validation the very first `cdk synth` would start failing and the `for` loop would never complete. This PR adds a `--no-validation` flag to `cdk synth` to stop the additional validation, so the `for` loop can complete successfully. The same behavior can be controlled with an environment variable, by setting `CDK_VALIDATION=false`. Fixes aws#15130.
In aws#14613, we introduced validation so that `cdk synth` would guard against incomplete context lookups. In the past, stacks with missing context would have successfully completed synthesis, propagated down the pipeline and caused confusing error messages during deployment. The new behavior was to error out early in case there was missing context. This broke people who had resorted to resolving context in the pipeline using multiple `synth` commands in `for`-loop: this used to work because the `synths` would be incomplete but silently succeed, but with the new validation the very first `cdk synth` would start failing and the `for` loop would never complete. This PR adds a `--no-validation` flag to `cdk synth` to stop the additional validation, so the `for` loop can complete successfully. The same behavior can be controlled with an environment variable, by setting `CDK_VALIDATION=false`. Fixes aws#15130.
My codebuild is breaking now when trying to synth in a cdkpipeline.
It appears that code build is trying to do some kind of cross accoutn lookup, using a cross account lookup role.
Its complaining it can not assume the role in the acconnt where the lookup is.
I found that a cross account lookup role has been introuduced in 1.108.. Are there any other changes that have occured in the way synths occur.?
edit:
I have tested various versions. THis code works find in versions up to and includeing 1.102. All versions higher fail with the same error.
https://github.com/aws/aws-cdk/pull/14874/files?file-filters%5B%5D=.yaml#diff-4fdac38426c4747aa17d515b01af4994d3d2f12c34f7b6655f24328259beb7bf
The text was updated successfully, but these errors were encountered: