Add "sync" command

[taraspos]

Would be nice to be able to have a smart sync command like s3cmd sync or aws s3 sync, which will upload the files that changed and remove the deleted ones.

[igungor]

Thanks for the feature request. It'll be a great addition to the current feature set.

Even though it's not the same as "sync", you can synchronize two directories (update if a file differs) with s5cmd cp -u -s --parents s3://<bucket>/<folder> .. Please keep in mind that it's not going to remove an existing object/file.

[kjaercasper]

A aws cli sync feature would be rally useful, so object are remove at bucket. Any news on a sync feature?

[missedone]

looking forward for a faster version of s3 sync as well

[missedone]

just got this project https://github.com/sethkor/s3kor supports fast sync

[favoyang]

just got this project https://github.com/sethkor/s3kor supports fast sync

Here I quote from the s3kor repository page

Sync only compares mod timestamps and sizes as the only true way to get a MD5 or some other checksum of an S3 object is to download it comletley first. An objects ETag is not the MD5 of the object.

Then it's very similar to s5cmd cp -u -s, except it did one more scan to remove non-existed keys. Sounds like an acceptable workaround, better than nothing.

[timureh]

Dear developers, do you plan to implement synchronization? For example like https://github.com/sethkor/s3kor (without checking MD5). I cannot use s3kor because I have a non-AWS endpoint. s5cmd there is just an opportunity to use the --endpoint-url flag. that's what i need. orient us please

[ilkinulas]

Hi, sync is on our roadmap, please follow this issue to be notified about any updates regarding s5cmd sync implementation.

[vladimirsvicevicsrb]

Any update our new release version that includes sync operator? Thanks!

[sonmezonur]

s5cmd v2.0.0-beta is released. It includes sync command. Happy to receive your feedback and contributions!

[eminegit]

Hi, sync command works great with few files. I am trying to sync 2 million files into s3 bucket which has more than 15 million files but I stops after 5 minutes with "killed" error message