Files that already exist on S3 are still uploaded

[kranzky]

We have many more than 1000 files in our S3 bucket, but the default value for max-keys in fog is 1000. This means that AssetSync::Storage::get_remote_files only returns the first 1000 files found, meaning that, although local_files_to_upload should be empty, it ends up containing many thousands of files.

[kranzky]

I believe that the same bug means that requesting delete instead of keep will only delete the first 1000 files.

[davidjrice]

Is this a bucket with 1000 static assets in it?

Are they all used in your HTML / CSS and JS build ?

[kranzky]

yes, we've got more than 1000 static assets in the bucket, partly because we're not deleting old ones and partly because more are being uploaded than necessary; it looks like Fog only returns the first 1000 no matter what; I've forked and am playing around with a fix

[kranzky]

This is a bug in Fog

[davidjrice]

Thanks! Version 0.1.10 should solve this. Let me know if there are any further issues.

[scottkf]

It seems like I still have this issue, without about 3000 items in the bucket. In a rails app, the app.js and app.css are processed last, and don't get changed.

[davidjrice]

@scottkf really? Are you on the latest version (0.2.4). I've had no problems like this.

It is likely still related to Fog.

If you want to put together a test case or give us access to a basic rails app that exposes this problem I'd be glad to give you a hand. Not the easiest one to replicate...

[scottkf]

@davidjrice I'm using 0.2.4, I haven't had time to put a test case yet, but I'll let you know, thanks. What's weird though, is that if I choose to use the manifest.yml and limit the amount of files, the files listed still don't get updated.

[scottkf]

@davidjrice I still didn't figure out why it wasn't working, but I deleted application.js and application.css, compiled the assets again and it was re-uploaded, and now syncs properly.

[davidjrice]

Great. Well definitely upgrade to 0.2.5 as well to be sure. There was an issue with 0.2.4

On Wednesday, 18 January 2012 at 20:32, Scott Tesoriere wrote:

@davidjrice I still didn't figure out why it wasn't working, but I deleted application.js and application.css, compiled the assets again and it was re-uploaded, and now syncs properly.

---

Reply to this email directly or view it on GitHub:
#16 (comment)