Using File Collection S3 storage adapter, where to put bucket

[anthonymanzo]

Hi,
I have this code in the kitchen json:
"collections" : [ { "name":"employee_pictures", "type":"file_collection", "storage_adapters":["s3"] },
And I've set my S3 access keys in my settings.json inside of the env property. However, I have no place to put the s3 bucket name and hence meteor throws an error when I compile:

W20170124-10:26:11.768(-8)? (STDERR) Error: FS.Store.S3 you must specify the "bucket" option
W20170124-10:26:11.768(-8)? (STDERR) at new FS.Store.S3 (packages/cfs_s3.js:114:11)

Where should I put this info?
Also, I haven't been able to find examples of how to use a fs collection in the examples, yet the s3 and fs storage adapters are mentioned in the API. Is this just a stub, or is it supported?

Thanks again.
Tony

[perak]

@anthonymanzo

Hi Tony,

To be honest I never used S3 storage adapter with file collection, but let's modify kitchen to allow this. Can you please let me know what/where to add/change to make S3 works properly? Maybe you can provide minimal .json file and modify generated code to allow S3 works - that would be perfect.

[perak]

(example-upload, generated code modified manually to allow S3 storage is perfect - maybe I can modify and deploy fixed kitchen today/tomorrow)

[anthonymanzo]

HI Perak, I think we’ll need to add a few things: 1. An area to define bucket, ‘store’ and any transform options information in your collection def. (disregard the accesskey and secret acesskey, those I define on server env safely) — see example from https://github.com/CollectionFS/Meteor-CollectionFS/tree/master/packages/s3: var imageStore = new FS.Store.S3("images", { region: "my-s3-region", //optional in most cases accessKeyId: "account or IAM key", //required if environment variables are not set secretAccessKey: "account or IAM secret", //required if environment variables are not set bucket: "mybucket", //required ACL: "myValue", //optional, default is 'private', but you can allow public or secure access routed through your app URL folder: "folder/in/bucket", //optional, which folder (key prefix) in the bucket to use fileKey: function(fileObj) { return new Date().getTime() + "-" + fileObj.name(); } // The rest are generic store options supported by all storage adapters transformWrite: myTransformWriteFunction, //optional transformRead: myTransformReadFunction, //optional maxTries: 1 //optional, default 5 }); Images = new FS.Collection("images", { stores: [imageStore] }); 2. A way to use things like <img> tags in the kitchen field definitions — I’ve tried to use custom fields and display helpers but they don’t display properly in the dataview. After publishing on server and subscribing on client it would look something like this: {{#each images}} {{> image}} {{/each}} <template name="image"> <div class="row"> <div class="col-md-12"> <a href="{{url}}" target="_blank" class="thumbnail"> <img src="{{url}}" /> </a> </div> </div> </template> 3. And probably a way in the publication to define queries that search on key or filename? BTW, here is an example of my settings.json where the env information that AWS S3 needs: "env":{ "AWS_ACCESS_KEY_ID”:"xxxx", "AWS_SECRET_ACCESS_KEY”:"xxxx" } Thanks for your help! Tony On Jan 24, 2017, at 11:56 AM, Petar Korponaić <[email protected]<mailto:[email protected]>> wrote: @anthonymanzo<https://github.com/anthonymanzo> Hi Tony, To be honest I never used S3 storage adapter with file collection, but let's modify kitchen to allow this. Can you please let me know what/where to add/change to make S3 works properly? Maybe you can provide minimal .json file and modify generated code to allow S3 works - that would be perfect. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#50 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AH95W3dfp1ZV6U8hD_g_C6Wi2fYQGLtRks5rVlddgaJpZM4LsoC5>.

[perak]

@anthonymanzo OK, in next version 0.9.78 I added this:

Storage adapters now can be defined in storage_adapter_options instead storage_adapters (both works for backward compatibility) like this:

{
	"name": "files",
	"type": "file_collection",
	"storage_adapter_options": { 
		"s3": {
			"bucket": "mybucket",
                        ...
		}
	}
}

And resulting code is:

this.Files = new FS.Collection("files", {
	stores: [
		new FS.Store.S3("files", {
			bucket: "mybucket",
                	...
		})
	]
});

storage_adapter_options is json object with "s3", "dropbox", "filesystem" or "gridfs" in the root, and these objects can contain anything. JSON is converted to javascript and inserted into code.

I hope this helps.

Unfortunately, I'll not able to deploy latest version tonight, I hope I'll deploy tomorrow ~24 hours after now.

[anthonymanzo]

Awesome, I’ll give a try when it’s deployed. Thanks. On Jan 24, 2017, at 2:45 PM, Petar Korponaić <[email protected]<mailto:[email protected]>> wrote: @anthonymanzo<https://github.com/anthonymanzo> OK, in next version 0.9.78 I added this: Storage adapters now can be defined in storage_adapter_options instead storage_adapters (both works for backward compatibility) like this: { "name": "files", "type": "file_collection", "storage_adapter_options": { "s3": { "bucket": "mybucket", ... } } } And resulting code is: this.Files = new FS.Collection("files", { stores: [ new FS.Store.S3("files", { bucket: "mybucket", ... }) ] }); storage_adapter_options is json object with "s3", "dropbox", "filesystem" or "gridfs" in the root, and these objects can contain anything. JSON is converted to javascript and inserted into code. I hope this helps. Unfortunately, I'll not able to deploy latest version tonight, I hope I'll deploy tomorrow ~24 hours after now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#50 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AH95W2QtwwHd3EMHB0a79ZgN5eP1YN_Uks5rVn7xgaJpZM4LsoC5>.

[perak]

@anthonymanzo version 0.9.78 is deployed few minutes ago.

[perak]

BTW, storage_adapter_options can only be edited directly in json ("source" tab in a GUI) - I didn't implemented "fancy" editor - not yet.

[anthonymanzo]

Storage adapter code is now showing up in the subscriptions - thanks.
Minor problem, I can't render the images with the data-view component - I need a way to return tags -- would use {{{ employeePictureToImgTag employee_picture }}} but the generated code for th data view is only using the standard handlebars double {{ }} which escapes html.

I am a noob at handlebars/blaze so please forgive my ignorance if this is not an issue.

[perak]

@anthonymanzo triple braces: already reported here perak/kitchen-site#363 will be fixed in next release.

[anthonymanzo]

Hi,
Just wanted to give an update that I haven't been able to test this as I'm having trouble with the latest version of cfs:s3 not working for me, I had the old version on lockdown.
I'm going to close this issue for you until then!

[anthonymanzo]

Hi, another comment on this. Got it working (more or less) and want to know how to define multiple stores for a collection (ie a largthumb, smallthumb set). Any ideas?

[anthonymanzo]

Here's my code. Kitchen generator is only picking up the first store:
"collections" : [

		{
			"name": "employee_thumbs",
			"type": "file_collection",
			"storage_adapter_options": { 
				"s3": {
					"bucket": "salk-saw-assets",
					"folder":"thumbs",
					"transformWrite":"function(fileObj, readStream, writeStream) {

// Transform the image into a 30x thumbnail
gm(readStream, fileObj.name()).resize('77', '77').stream().pipe(writeStream);
}",
},
"s3": {
"bucket": "salk-saw-assets",
"folder":"large_thumbs",
"transformWrite":"function(fileObj, readStream, writeStream) {
// Make the image a predictable 225px thumbnail
gm(readStream, fileObj.name()).resize('225', '225').stream().pipe(writeStream);
}",
}
}

		},

[perak]

@anthonymanzo yes, bug is: kitchen enumerates "distinct" storage adapters, instead each one. Will fix that for next release, but I'll not be able to deploy it next ~7 days. I suggest you to use copy_files and write code manually.