Asset Sync Incorrectly Reading Bucket as Empty and Uploading All Files Every Sync.

[Undistraction]

I am using Asset Sync 0.5.0 with an S3 Bucket. Every time I sync it uploads every file, regardless of whether it already exists in the bucket.

Logging inside storage.rb I can see that ignore_existing_remote_files? is definitely set to false, and get_remote_files is definitely called, however despite the fact my bucket is full of files, bucket.files has a count of zero, so all files are uploaded as if the bucket was empty.

At this point there is the following comment:

fixes: #16

(work-around for fog/fog#596)

Is this possibly a regression?

[davidjrice]

You still having this issue? Am not able to replicate this at present.

Perhaps you can fire over some more detail, like:

• Are you using with AWS?
• Are there more than 1000 files in your bucket?
• If you start with a fresh bucket does it still happen?

[Undistraction]

Yes. I'm using it with AWS
No there are less than 1000 files.
Yes I've tried a new bucket and have the same problem.

[nicolasgarnil]

+1 having the same problem with version 0.5.0... I'm using middleman-sync which uses asset_sync and my entire website is being synced each time.
Is there a workaround?

[davidjrice]

@1ndivisible @negarnil having trouble recreating this. Any chance either of you could do the following to help me recreate?

• The easiest way would be granting access to said app. Which I understand, might be a security risk.
• The other option would be creating a "dummy" app that reproduces the problem.

[nicolasgarnil]

Hello david, I'm facing the same issue when I sync my middleman blog (using the middleman-sync gem which uses asset_sync) with my S3 bucket. I run $> middleman build (generates static files in the build directory) and then $> middleman sync and instead of only uploading new/modified files it uploads the entire build directory. And if I run $> middleman sync without running $> middleman build it uploads again the entire build directory instead of not uploading anything.

My blog is small in size but I'm working in other middleman blog for a client and every single sync to S3 uploads ~ 1 gb of data and it takes 1 hour or more no matter how many files I modified.

I can't give you access to my personal blog repository but can send you a zip file (without my S3 keys) to [email protected].

[stationkeeping]

Hi David,

Sorry for missing this. Can send you everything but my AWS keys if it would be useful? I've tried using a fresh bucket and a bucket on an alternative S3 Account and problem is replicated across both.

[davidjrice]

@negarnil @stationkeeping sure, if you can provide an archive of a sample app via email / dropbox that's cool.

You could setup a user on AWS using IAM that has access to only that bucket. Or a representative sample of the assets in the original bucket.

[dougc84]

+1 on this - we have the same issue with heroku and a major production app. we use s3 as well.

i cannot provide source or access to repo.