GovCloud -> Specified bucket does not exist #1459

sentient-kshaffer · 2018-12-31T19:13:17Z

I'm trying to deploy to govcloud using cdk version 0.21.0, and I can get to the point of creating CloudFormation changesets, (the assets/cdk meta is in s3), but then when creating the changeset I get an error: " ❌ TestApi failed: ValidationError: S3 error: The specified bucket does not exist
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html".

Here's the verbose cdk deploy output.

Command line arguments: { _: [ 'deploy' ],
  trace: false,
  strict: false,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  verbose: true,
  v: true,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  version: false,
  help: false,
  h: false,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  '$0': 'cdk',
  app: undefined,
  context: undefined,
  plugin: undefined,
  rename: undefined,
  profile: undefined,
  proxy: undefined,
  'require-approval': undefined,
  'toolkit-stack-name': undefined,
  STACKS: [] }
Determining whether we're on an EC2 instance.
Does not look like EC2 instance.
cdk.json: {
  "app": "node -r dotenv/config cloudformation.js",
  "requireApproval": "never"
}
Setting "aws:cdk:toolkit:default-region" context to us-gov-west-1
Resolving default credentials
Retrieved account ID 12345678987654321 from disk cache
Setting "aws:cdk:toolkit:default-account" context to 12345678987654321
context: { 'aws:cdk:toolkit:default-region': 'us-gov-west-1',
  'aws:cdk:toolkit:default-account': '12345678987654321',
  'aws:cdk:enable-path-metadata': true }
outdir: /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdkMytzxN
outfile: /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdkMytzxN/cdk.out
{ version: '0.19.0',
  stacks: 
   [ { name: 'TestApi',
       environment: [Object],
       template: [Object],
       metadata: [Object] } ],
  runtime: 
   { libraries: 
      { '@aws-cdk/cdk': '0.21.0',
        '@aws-cdk/cx-api': '0.21.0',
        '@aws-cdk/aws-ec2': '0.21.0',
        '@aws-cdk/aws-iam': '0.21.0',
        '@aws-cdk/aws-s3': '0.21.0',
        '@aws-cdk/aws-kms': '0.21.0',
        '@aws-cdk/aws-s3-notifications': '0.21.0',
        '@aws-cdk/aws-codepipeline-api': '0.21.0',
        '@aws-cdk/aws-events': '0.21.0',
        '@aws-cdk/aws-dynamodb': '0.21.0',
        '@aws-cdk/aws-applicationautoscaling': '0.21.0',
        '@aws-cdk/aws-autoscaling-common': '0.21.0',
        '@aws-cdk/aws-cloudwatch': '0.21.0',
        '@aws-cdk/assets': '0.21.0',
        '@aws-cdk/aws-serverless': '0.21.0',
        '@aws-cdk/aws-lambda': '0.21.0',
        '@aws-cdk/aws-sqs': '0.21.0',
        '@aws-cdk/aws-sns': '0.21.0',
        'jsii-runtime': 'node.js/v8.9.2' } } }
Removing outdir /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdkMytzxN
Stack name not specified, so defaulting to all available stacks: TestApi
Using default AWS SDK credentials for account 12345678987654321
Waiting for stack CDKToolkit to finish creating or updating...
TestApi: deploying...
Preparing assets
 - /Users/ME/repos/Test-api/_build/api (zip)
Preparing asset {"path":"/Users/ME/repos/Test-api/_build/api","id":"TestApiTestApiApiLambdaAsset696E6B90","packaging":"zip","s3BucketParameter":"TestApiApiLambdaAssetS3Bucket6941F9D0","s3KeyParameter":"TestApiApiLambdaAssetS3VersionKeyF34988D2"}
Preparing zip asset from directory: /Users/ME/repos/Test-api/_build/api
zip archive: /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdk-assetsCHbeiP/archive.zip
Preparing file asset: /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdk-assetsCHbeiP/archive.zip
Using default AWS SDK credentials for account 12345678987654321
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/assets/TestApiTestApiApiLambdaAsset696E6B90/c992d8450f3a4eb2c9b070d8d3803f6eb84f13c3f61a17ab61c7dcc565a9437f.zip: checking if already exists
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/assets/TestApiTestApiApiLambdaAsset696E6B90/c992d8450f3a4eb2c9b070d8d3803f6eb84f13c3f61a17ab61c7dcc565a9437f.zip: uploading
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/assets/TestApiTestApiApiLambdaAsset696E6B90/c992d8450f3a4eb2c9b070d8d3803f6eb84f13c3f61a17ab61c7dcc565a9437f.zip: upload complete
S3 url for _build/api: s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/assets/TestApiTestApiApiLambdaAsset696E6B90/c992d8450f3a4eb2c9b070d8d3803f6eb84f13c3f61a17ab61c7dcc565a9437f.zip
Updated: _build/api (zip)
 - /Users/ME/repos/Test-api/_build/worker (zip)
Preparing asset {"path":"/Users/ME/repos/Test-api/_build/worker","id":"TestApiTestApiWorkerLambdaAsset968F7F09","packaging":"zip","s3BucketParameter":"TestApiWorkerLambdaAssetS3Bucket8738621F","s3KeyParameter":"TestApiWorkerLambdaAssetS3VersionKey6C50483F"}
Preparing zip asset from directory: /Users/ME/repos/Test-api/_build/worker
zip archive: /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdk-assets2JeUzB/archive.zip
Preparing file asset: /var/folders/by/fdfp2zzj3rz9kqp819l6kpvm0000gn/T/cdk-assets2JeUzB/archive.zip
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/assets/TestApiTestApiWorkerLambdaAsset968F7F09/55a8234937ca9ef83080cf6c81e5ae6fbae765a1cf9e189b551a8efcb17a43d4.zip: checking if already exists
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/assets/TestApiTestApiWorkerLambdaAsset968F7F09/55a8234937ca9ef83080cf6c81e5ae6fbae765a1cf9e189b551a8efcb17a43d4.zip: uploading
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/assets/TestApiTestApiWorkerLambdaAsset968F7F09/55a8234937ca9ef83080cf6c81e5ae6fbae765a1cf9e189b551a8efcb17a43d4.zip: upload complete
S3 url for _build/worker: s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/assets/TestApiTestApiWorkerLambdaAsset968F7F09/55a8234937ca9ef83080cf6c81e5ae6fbae765a1cf9e189b551a8efcb17a43d4.zip
Updated: _build/worker (zip)
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/cdk/TestApi/d905fa8a01da407c101c7b5d822cab437f39fe58c8aaf09d94df947b500d4690.yml: checking if already exists
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/cdk/TestApi/d905fa8a01da407c101c7b5d822cab437f39fe58c8aaf09d94df947b500d4690.yml: uploading
s3://cdktoolkit-stagingbucket-1i4zjzlrfs10/cdk/TestApi/d905fa8a01da407c101c7b5d822cab437f39fe58c8aaf09d94df947b500d4690.yml: upload complete
Stored template in S3 at: https://cdktoolkit-stagingbucket-1i4zjzlrfs10.s3.amazonaws.com/cdk/TestApi/d905fa8a01da407c101c7b5d822cab437f39fe58c8aaf09d94df947b500d4690.yml
Attempting to create ChangeSet CDK-861eb256-d225-432f-93c0-8d05b583f523 to create stack TestApi
TestApi: creating CloudFormation changeset...

 ❌  TestApi failed: ValidationError: S3 error: The specified bucket does not exist
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
S3 error: The specified bucket does not exist
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
ValidationError: S3 error: The specified bucket does not exist
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
    at Request.extractError (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18)```

The text was updated successfully, but these errors were encountered:

sentient-kshaffer · 2018-12-31T19:24:18Z

It appears this is due to the way that AWS names the govcloud buckets and has aliases to resources...
Since govcloud s3 is actually region based: s3-us-gov-west-1.amazonaws.com instead of just s3.amazonaws.com.

rix0rrr · 2019-01-02T09:23:57Z

Interesting. From a cursory code review, this shouldn't happen.

Could you go into CloudFormation and check the outputs of the CDKToolkit stack for me please?

sentient-kshaffer · 2019-01-02T13:52:01Z

@rix0rrr
BucketName: cdktoolkit-stagingbucket-1i4zjzlrfs10
BucketDomainName: cdktoolkit-stagingbucket-1i4zjzlrfs10.s3.amazonaws.com

And if I'm understanding correctly, the BucketDomainName should be: cdktoolkit-stagingbucket-1i4zjzlrfs10.s3-us-gov-west-1.amazonaws.com, right?

rix0rrr · 2019-01-02T13:59:24Z

Yes! The silly thing is, it seems to be a CloudFormation issue. The content of the CDKToolkit template should be this:

Ouputs:
  BucketDomainName:
    Value: { "Fn::GetAtt": ["StagingBucket", "DomainName"] }

Which means we're asking CloudFormation for the domain name, and it's returning the wrong name.

sentient-kshaffer · 2019-01-02T14:01:21Z

@rix0rrr Is there anyway for me to override this such that I can actually use cdk to do my deployment? I've tried using cdk synth to create my templates and using the aws cli directly, but the synthesized template is not valid and needs a lot of TLC...

Thanks for the quick responses!

rix0rrr · 2019-01-02T14:09:50Z

You can try changing the CDKToolkit template and change it to something like the following:

Ouputs:
  BucketDomainName:
    Value: { "Fn::Join": [ ".", [{ "Fn::Ref": "StagingBucket" }, "s3-us-gov-west-1.amazonaws.com"]] }

I.e., construct the right Output value manually.

It might be that this change will not deploy since CloudFormation usually doesn't deploy templates if the only thing that changed are the Outputs. In that case, add an AWS::CloudFormation::WaitConditionHandle to the list of Resources.

Resources:
  DummyToDeploy1:
    Type: AWS::CloudFormation::WaitConditionHandle

sentient-kshaffer · 2019-01-02T14:11:42Z

@rix0rrr Would it be possible to use the RegionalDomainName for that value in CDK? Not sure how that affects non-govcloud s3 buckets, since they're normally not region specific...
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html

Also, I'll give that a go, thanks!

rix0rrr · 2019-01-02T14:14:48Z

Yeah you can also try using RegionalDomainName. If that works for you, I'll be happy to update the toolkit to use that instead.

rix0rrr · 2019-01-02T14:27:26Z

Ah, no that's internal. Pass --no-version-reporting to cdk deploy (or set versionReporting: false in ckd.json).

This is part of some internal infrastructure that we unfortunately haven't been able to deploy to GovCloud yet.

sentient-kshaffer · 2019-01-02T14:29:26Z

Okay thanks! Sorry I deleted the comment, because I thought I did something wrong. Let me give that a go!

sentient-kshaffer · 2019-01-02T14:34:06Z

Cool, that got me beyond the issues I made this ticket for. Now i'm just getting some about my API Gateway spec, time to debug!

Thanks a bunch @rix0rrr.

And just as an overview of this thread:

The DomainName from CloudFormation for the CDKToolkit BucketDomainName doesn't return the region correctly when in a region based s3 region, but using RegionalDomainName does.
In order to avoid CDK::MetaData resources issues for now, pass in versionReporting: false to cdk.json.

Just as a question for how updating on this repository goes, are you going to make a change for the RegionalDomainName in the toolkit? Or is that something I should try to create a pull request for? This is an amazing utility and I want to help make it better however I can (even with small changes!)

Thanks again for your assistance!

rix0rrr · 2019-01-02T14:46:51Z

Thanks for the kind words :).

I made a ticket for the first issue: #1469

It's probably a matter of changing the CFN code in this file: https://github.com/awslabs/aws-cdk/blob/master/packages/aws-cdk/lib/api/bootstrap-environment.ts, but warrants some additional testing.

Can't promise we'll get around to it quickly, if you want to give it a shot you're more than welcome to!

sentient-kshaffer · 2019-01-02T14:50:45Z

@rix0rrr Awesome thanks!

And yes this definitely warrants testing in different regions... S3 buckets can be strange!

Thanks for the workaround in the meantime though!

rix0rrr · 2019-01-02T14:54:19Z

I think I will close this issue, as it seems there is no actionable work left. Feel free to reopen (or file a new one) if you run into more GovCloud issues.

Unfortunately, I predict some more issues with ServicePrincipals similar to the issues in China.

dz902 · 2019-07-28T17:15:40Z

@rix0rrr Any planned updates on this issue?

This is still blocking Chinese regions users to use CDK, unless change the file by hand, which is bad practice as CDK is iterating rather rapidly.

dz902 · 2019-07-28T17:31:23Z

Editing, then, rerun cdk bootstrap is required to make this temporary workaround work. I don't see any local temporary files generated by cdk bootstrap, does this mean when cdk deploy, it will look for a stack named "CDKToolkit" and just use it?

beary · 2019-08-02T13:42:27Z

@rix0rrr
I also meet this problem, it's China region.

cdk bootstrap, everything's ok

cdk deploy -v

Stored template in S3 at: https://toolkit-stagebucket.s3.amazonaws.com/cdk/XUserStack/123123.yml
XUserStack: creating CloudFormation changeset...

  ❌  XUserStack failed: ValidationError: S3 error: The specified bucket does not exist

I checked my S3 bucket found the 123123.yml had been uploaded. But the Object URL above is wrong, it should be
https://toolkit-stagebucket.s3.cn-north-1.amazonaws.com.cn/cdk/XUserStack/123123.yml

Is there any way to change this URL in the code or via a plugin? Like some serverless plugin, I read the API document but not found any methods.

whimzyLive · 2019-09-30T06:37:37Z

@rix0rrr Facing same issue as @beary with cdk 1.9.0

rix0rrr added bug This issue is a bug. package/tools Related to AWS CDK Tools or CLI labels Jan 2, 2019

rix0rrr closed this as completed Jan 2, 2019

beary mentioned this issue Aug 2, 2019

Toolkit: toolkit output domain name needs to use RegionalDomainName for GovCloud and CN compatibility #1469

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GovCloud -> Specified bucket does not exist #1459

GovCloud -> Specified bucket does not exist #1459

sentient-kshaffer commented Dec 31, 2018 •

edited

Loading

sentient-kshaffer commented Dec 31, 2018

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019 •

edited

Loading

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

dz902 commented Jul 28, 2019

dz902 commented Jul 28, 2019

beary commented Aug 2, 2019

whimzyLive commented Sep 30, 2019

GovCloud -> Specified bucket does not exist #1459

GovCloud -> Specified bucket does not exist #1459

Comments

sentient-kshaffer commented Dec 31, 2018 • edited Loading

sentient-kshaffer commented Dec 31, 2018

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019 • edited Loading

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

sentient-kshaffer commented Jan 2, 2019

rix0rrr commented Jan 2, 2019

dz902 commented Jul 28, 2019

dz902 commented Jul 28, 2019

beary commented Aug 2, 2019

whimzyLive commented Sep 30, 2019

sentient-kshaffer commented Dec 31, 2018 •

edited

Loading

sentient-kshaffer commented Jan 2, 2019 •

edited

Loading