Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

s3-deployment: allow disabling "sync --delete" #953

Closed
eladb opened this issue Oct 17, 2018 · 6 comments · Fixed by #8263
Closed

s3-deployment: allow disabling "sync --delete" #953

eladb opened this issue Oct 17, 2018 · 6 comments · Fixed by #8263
Assignees
@iliapolo
Labels
@aws-cdk/aws-s3-deployment effort/small Small work item – less than a day of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on.

Comments

@eladb
Copy link
Contributor

eladb commented Oct 17, 2018

The current behavior is aws sync --delete, which means that destination files that don't exist in source will be deleted. In some scenarios (see #936) it might be useful to retain old files for the transition period when an app is updated.
eladb pushed a commit that referenced this issue Oct 19, 2018
feat(aws-s3-deployment): bucket deployments
d5b8940 
This new library allows deploying contents to S3 buckets as part of a CDK app from a .zip file in another bucket or from a local asset (directory or a .zip file).

This is how it's used:

    new BucketDeployment(this, 'DeployMe', {
      source: Source.asset('/local/directory'),
      destinationBucket: dest
    });

"source" can either be a local directory, a local zip file or another
bucket+key.

This library is backed by a custom CloudFormation resource which uses the
AWS CLI to invoke "aws s3 sync". 

By default, when the resource is deleted, the contents of the destination are deleted. The `RetainOnDelete` option can be used to disable this behavior.

Fixes #952.


Future plans:

- The current version will always use "--delete" which means destination files will be deleted if they don't exist on the source (will be addressed in #953).
- Another mode of operation that we plan to add is blue/green deployments (#954).

�
@eladb eladb mentioned this issue Oct 19, 2018
Merged
eladb pushed a commit that referenced this issue Oct 19, 2018
feat(aws-s3-deployment): bucket deployments (#971)
84d6876 
This new library allows deploying contents to S3 buckets as part of a CDK app from a .zip file in another bucket or from a local asset (directory or a .zip file).

This is how it's used:

    new BucketDeployment(this, 'DeployMe', {
      source: Source.asset('/local/directory'),
      destinationBucket: dest
    });

"source" can either be a local directory, a local zip file or another
bucket+key.

This library is backed by a custom CloudFormation resource which uses the
AWS CLI to invoke "aws s3 sync". 

By default, when the resource is deleted, the contents of the destination are deleted. The `RetainOnDelete` option can be used to disable this behavior.

Fixes #952.

Misc:
- Change travis CI environment to allow running python3 (as well as node.js).
- Add "!!!!!" markers when build/test fails so it will be easier to locate errors in build logs.

Future plans:

- The current version will always use "--delete" which means destination files will be deleted if they don't exist on the source (will be addressed in #953).
- Another mode of operation that we plan to add is blue/green deployments (#954).
eladb pushed a commit that referenced this issue Oct 19, 2018
v0.13.0
cb187b9 
### Highlights

 - __A new construct library for AWS Step Functions__
   ([docs](https://github.com/awslabs/aws-cdk/blob/master/packages/%40aws-cdk/aws-stepfunctions/README.md)).
   The library provides rich APIs for modeling state machines by exposing a
   programmatic interface for [Amazon State
   Language](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html).
 - __A new construct library for Amazon S3 bucket deployments__
   ([docs](https://github.com/awslabs/aws-cdk/blob/master/packages/%40aws-cdk/aws-s3-deployment/README.md)).
   You can use now automatically populate an S3 Bucket from a .zip file or a
   local directory. This is a building block for end-to-end support for static
   websites in the AWS CDK.

### Bug Fixes

* **aws-apigateway:** make LambdaRestApi proxy by default ([#963](#963)) ([a5f5e2c](a5f5e2c)), closes [#959](#959)
* **aws-cdk:** Allow use of assumed roles behind a proxy ([#898](#898)) ([f2b1048](f2b1048))
* **aws-cdk:** Auto-delete stacks that failed creating before new attempt ([#917](#917)) ([2af8309](2af8309))
* **aws-cloudfront:** expose distributionId ([#938](#938)) ([f58d98c](f58d98c))
* **aws-dynamodb:** don't emit empty array properties ([#909](#909)) ([841975a](841975a))
* **docs:** use ..code to display file structure in "writing constructs" ([#935](#935)) ([b743362](b743362))

### Features

* **assets:** isZipArchive indicates if this is a zip asset ([#944](#944)) ([65190f9](65190f9))
* **aws-cdk:** deploy supports CloudFormation Role ([#940](#940)) ([393be6f](393be6f)), closes [#735](#735)
* **aws-cloudformation:** allow specifying custom resource type ([#943](#943)) ([9de3a84](9de3a84))
* **aws-cloudformation:** correctly handle the templateConfiguration property in the CreateUpdateStack Pipeline Action. ([#923](#923)) ([d251a46](d251a46))
* **aws-cloudfront:** add support for "webAclId" ([#969](#969)) ([3ec9d76](3ec9d76))
* **aws-codedeploy:** add auto rollback configuration to server Deployment Group. ([#925](#925)) ([7ee91cf](7ee91cf))
* **aws-codedeploy:** add instance tag filter support for server Deployment Groups. ([#824](#824)) ([e6e8c51](e6e8c51))
* **aws-codedeploy:** add support for setting CloudWatch alarms on a server Deployment Group. ([#926](#926)) ([27b26b1](27b26b1))
* add support for Step Functions ([#827](#827)) ([81b533c](81b533c))
* **aws-lambda:** add grantInvoke() method ([#962](#962)) ([1ee8135](1ee8135)), closes [#961](#961)
* **aws-lambda:** improvements to the code and runtime APIs ([#945](#945)) ([36f29b6](36f29b6)), closes [#902](#902) [#188](#188) [#947](#947) [#947](#947) [#664](#664)
* **aws-logs:** extractMetric() returns Metric object ([#939](#939)) ([5558fff](5558fff)), closes [#850](#850)
* **aws-s3:** initial support for website hosting ([#946](#946)) ([2d3661c](2d3661c))
* **aws-s3-deployment:** bucket deployments ([#971](#971)) ([84d6876](84d6876)), closes [#952](#952) [#953](#953) [#954](#954)
* **docs:** added link to CloudFormation concepts ([#934](#934)) ([666bbba](666bbba))

### BREAKING CHANGES

* **aws-apigateway:** specifying a path no longer works. If you used to
provide a '/', remove it. Otherwise, you will have to supply `proxy: false`
and construct more complex resource paths yourself.
* **aws-lambda:** The construct `lambda.InlineJavaScriptLambda` is no longer supported. Use `lambda.Code.inline` instead; `lambda.Runtime.NodeJS43Edge` runtime is removed. CloudFront docs [stipulate](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html#lambda-requirements-lambda-function-configuration) that you should use node6.10 or node8.10. It is always possible to use any value by instantiating a `lambda.Runtime` object.
@eladb eladb mentioned this issue Oct 19, 2018
Merged
eladb pushed a commit that referenced this issue Oct 19, 2018
v0.13.0 (#975)
bf73b09 
### Highlights

 - __A new construct library for AWS Step Functions__
   ([docs](https://github.com/awslabs/aws-cdk/blob/master/packages/%40aws-cdk/aws-stepfunctions/README.md)).
   The library provides rich APIs for modeling state machines by exposing a
   programmatic interface for [Amazon State
   Language](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html).
 - __A new construct library for Amazon S3 bucket deployments__
   ([docs](https://github.com/awslabs/aws-cdk/blob/master/packages/%40aws-cdk/aws-s3-deployment/README.md)).
   You can use now automatically populate an S3 Bucket from a .zip file or a
   local directory. This is a building block for end-to-end support for static
   websites in the AWS CDK.

### Bug Fixes

* **aws-apigateway:** make LambdaRestApi proxy by default ([#963](#963)) ([a5f5e2c](a5f5e2c)), closes [#959](#959)
* **aws-cdk:** Allow use of assumed roles behind a proxy ([#898](#898)) ([f2b1048](f2b1048))
* **aws-cdk:** Auto-delete stacks that failed creating before new attempt ([#917](#917)) ([2af8309](2af8309))
* **aws-cloudfront:** expose distributionId ([#938](#938)) ([f58d98c](f58d98c))
* **aws-dynamodb:** don't emit empty array properties ([#909](#909)) ([841975a](841975a))
* **docs:** use ..code to display file structure in "writing constructs" ([#935](#935)) ([b743362](b743362))

### Features

* **assets:** isZipArchive indicates if this is a zip asset ([#944](#944)) ([65190f9](65190f9))
* **aws-cdk:** deploy supports CloudFormation Role ([#940](#940)) ([393be6f](393be6f)), closes [#735](#735)
* **aws-cloudformation:** allow specifying custom resource type ([#943](#943)) ([9de3a84](9de3a84))
* **aws-cloudformation:** correctly handle the templateConfiguration property in the CreateUpdateStack Pipeline Action. ([#923](#923)) ([d251a46](d251a46))
* **aws-cloudfront:** add support for "webAclId" ([#969](#969)) ([3ec9d76](3ec9d76))
* **aws-codedeploy:** add auto rollback configuration to server Deployment Group. ([#925](#925)) ([7ee91cf](7ee91cf))
* **aws-codedeploy:** add instance tag filter support for server Deployment Groups. ([#824](#824)) ([e6e8c51](e6e8c51))
* **aws-codedeploy:** add support for setting CloudWatch alarms on a server Deployment Group. ([#926](#926)) ([27b26b1](27b26b1))
* add support for Step Functions ([#827](#827)) ([81b533c](81b533c))
* **aws-lambda:** add grantInvoke() method ([#962](#962)) ([1ee8135](1ee8135)), closes [#961](#961)
* **aws-lambda:** improvements to the code and runtime APIs ([#945](#945)) ([36f29b6](36f29b6)), closes [#902](#902) [#188](#188) [#947](#947) [#947](#947) [#664](#664)
* **aws-logs:** extractMetric() returns Metric object ([#939](#939)) ([5558fff](5558fff)), closes [#850](#850)
* **aws-s3:** initial support for website hosting ([#946](#946)) ([2d3661c](2d3661c))
* **aws-s3-deployment:** bucket deployments ([#971](#971)) ([84d6876](84d6876)), closes [#952](#952) [#953](#953) [#954](#954)
* **docs:** added link to CloudFormation concepts ([#934](#934)) ([666bbba](666bbba))

### BREAKING CHANGES

* **aws-apigateway:** specifying a path no longer works. If you used to
provide a '/', remove it. Otherwise, you will have to supply `proxy: false`
and construct more complex resource paths yourself.
* **aws-lambda:** The construct `lambda.InlineJavaScriptLambda` is no longer supported. Use `lambda.Code.inline` instead; `lambda.Runtime.NodeJS43Edge` runtime is removed. CloudFront docs [stipulate](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html#lambda-requirements-lambda-function-configuration) that you should use node6.10 or node8.10. It is always possible to use any value by instantiating a `lambda.Runtime` object.
@rix0rrr rix0rrr added the feature-request A feature should be added or improved. label Nov 6, 2018
@eladb eladb added the @aws-cdk/aws-s3 Related to Amazon S3 label Sep 8, 2019
@eladb eladb self-assigned this Sep 8, 2019
@eladb
Copy link
Contributor Author

eladb commented Oct 2, 2019

Still relevant.

@eladb eladb assigned iliapolo and unassigned eladb Jan 23, 2020
@iliapolo iliapolo added the effort/small Small work item – less than a day of effort label Mar 9, 2020
@martinnabhan
Copy link

Still relevant for us too!

We're trying to set different headers for only index.html, but currently it doesn't seem to be possible. We've tried using two deployments, but since we can't disable sync --delete it's always one deployment or the other.

    new BucketDeployment(this, 'BucketDeployment', {
      sources: [Source.asset('../cms/dist', { exclude: ['.git', 'cdk.out', 'node_modules', 'index.html' })],
      destinationBucket: bucket,
      cacheControl: [CacheControl.fromString('max-age=31536000,public,immutable')],
    });

    new BucketDeployment(this, 'HTMLBucketDeployment', {
      sources: [Source.asset('../cms/dist', { exclude: ['**', '!index.html'] })],
      destinationBucket: bucket,
      cacheControl: [CacheControl.fromString('max-age=0,no-cache,no-store,must-revalidate')],
      contentType: 'text/html; charset=utf-8',
    });

Is there any other way of setting headers for specific objects?

@abbottdev
Copy link

I'm currently using an S3 bucket to host both my static website, along with some JSON data files which are loaded via HTTP requests. (They're in the same bucket to make CORS management easier!).

The JSON files are updated by Lambdas with a scheduled job - the current --delete means I have to manually re-run these after a deployment - would be much easier to specify a different behaviour.

Really useful feature though so far, thanks all 😄

@iliapolo iliapolo added @aws-cdk/aws-s3-deployment and removed @aws-cdk/aws-s3 Related to Amazon S3 labels May 8, 2020
@iliapolo iliapolo mentioned this issue May 8, 2020
Closed
@jangau
Copy link

jangau commented May 28, 2020

Still relevant here too.

@iliapolo iliapolo mentioned this issue May 28, 2020
Merged
@SomayaB SomayaB added the in-progress This issue is being actively worked on. label Jun 1, 2020
@mergify mergify bot closed this as completed in #8263 Jul 5, 2020
mergify bot pushed a commit that referenced this issue Jul 5, 2020
@iliapolo
feat(s3-deployment): prune - keep missing files on destination bucket (…
57914c7 
…#8263)

Introduce a `prune` property that eventually controls the `--delete` flag when invoking the `aws s3 sync` command.

Resolves #953 

In addition, migrate the module from `nodeunit` to `jest`.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@IslamWahid
Copy link

@eladb with that prune option, how will the old files be deleted eventually? it's necessary to keep them until the deployment is done but afterwards, we still want to remove them.

@FirstWhack
Copy link

FirstWhack commented Nov 1, 2021 •
edited
Loading

@IslamWahid I think I am looking for the same thing as you.

Something like pruneDelay: Duration would accomplish my needs. I need to ensure the non-pruned objects are eventually deleted.

I'm not sure this is a CDK issue though, it's not possible outside of CDK either as far as I know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iliapolo iliapolo

Labels
@aws-cdk/aws-s3-deployment effort/small Small work item – less than a day of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(s3-deployment): prune - keep missing files on destination bucket aws/aws-cdk
9 participants
@rix0rrr @eladb @iliapolo @abbottdev @FirstWhack @martinnabhan @jangau @SomayaB @IslamWahid