Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: don't upload the same asset multiple times #1011

Merged
merged 2 commits into from
Nov 1, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions packages/@aws-cdk/assets/test/integ.multi-assets.expected.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"Parameters": {
"SampleAsset1S3Bucket469E18FF": {
"Type": "String",
"Description": "S3 bucket for asset \"aws-cdk-multi-assets/SampleAsset1\""
},
"SampleAsset1S3VersionKey63A628F0": {
"Type": "String",
"Description": "S3 key for asset version \"aws-cdk-multi-assets/SampleAsset1\""
},
"SampleAsset2S3BucketC94C651A": {
"Type": "String",
"Description": "S3 bucket for asset \"aws-cdk-multi-assets/SampleAsset2\""
},
"SampleAsset2S3VersionKey3A7E2CC4": {
"Type": "String",
"Description": "S3 key for asset version \"aws-cdk-multi-assets/SampleAsset2\""
}
}
}
23 changes: 23 additions & 0 deletions packages/@aws-cdk/assets/test/integ.multi-assets.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import cdk = require('@aws-cdk/cdk');
import path = require('path');
import assets = require('../lib');

class TestStack extends cdk.Stack {
constructor(parent: cdk.App, name: string, props?: cdk.StackProps) {
super(parent, name, props);

// Check that the same asset added multiple times is
// uploaded and copied.
new assets.FileAsset(this, 'SampleAsset1', {
path: path.join(__dirname, 'file-asset.txt')
});

new assets.FileAsset(this, 'SampleAsset2', {
path: path.join(__dirname, 'file-asset.txt')
});
}
}

const app = new cdk.App();
new TestStack(app, 'aws-cdk-multi-assets');
app.run();
11 changes: 7 additions & 4 deletions packages/@aws-cdk/aws-lambda/lib/code.ts
Original file line number Diff line number Diff line change
Expand Up @@ -143,10 +143,13 @@ export class AssetCode extends Code {
}

public bind(lambda: Func) {
this.asset = new assets.Asset(lambda, 'Code', {
path: this.path,
packaging: this.packaging
});
// If the same AssetCode is used multiple times, retain only the first instantiation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test for this?

if (!this.asset) {
this.asset = new assets.Asset(lambda, 'Code', {
path: this.path,
packaging: this.packaging
});
}

if (!this.asset.isZipArchive) {
throw new Error(`Asset must be a .zip file or a directory (${this.path})`);
Expand Down
39 changes: 29 additions & 10 deletions packages/aws-cdk/lib/api/toolkit-info.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ export interface Uploaded {
}

export class ToolkitInfo {
/**
* A cache of previous uploads done in this session
*/
private readonly previousUploads: {[key: string]: Uploaded} = {};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename key to something meaningful


constructor(private readonly props: {
sdk: SDK,
bucketName: string,
Expand Down Expand Up @@ -60,17 +65,31 @@ export class ToolkitInfo {
return { filename, key, changed: false };
}

debug(`${url}: uploading`);
await s3.putObject({
Bucket: bucket,
Key: key,
Body: data,
ContentType: props.contentType
}).promise();

debug(`${url}: upload complete`);
const uploaded = { filename, key, changed: true };

// Upload if it's new or server-side copy if it was already uploaded previously
const previous = this.previousUploads[hash];
if (previous) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just reuse the same file that was already uploaded?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're organizing the data in folders, by Asset instance.

This is currently so that we can give permissions on the folder to constructs that require access to the asset, so they will be able to download all versions of it, so that even if we upload a new version and the Execution Role is updated to only allow downloading the latest version of the asset, old runs/definitions will not stop working. This is not required for Lambdas which have a point-in-time handover but it is required for CodeBuild script assets, for example, which download the file at an arbitrary later point in time.

If we reused the same file between multiple assets, we'd either have to fall back to "only permissions to the latest version of the asset" or "permission to all assets in the bucket".

debug(`${url}: copying`);
await s3.copyObject({
Bucket: bucket,
Key: key,
CopySource: `${bucket}/${previous.key}`
}).promise();
debug(`${url}: copy complete`);
} else {
debug(`${url}: uploading`);
await s3.putObject({
Bucket: bucket,
Key: key,
Body: data,
ContentType: props.contentType
}).promise();
debug(`${url}: upload complete`);
this.previousUploads[hash] = uploaded;
}

return { filename, key, changed: true };
return uploaded;
}

}
Expand Down