Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync command issue with many files. #447

Closed
eminegit opened this issue Jun 20, 2022 · 4 comments · Fixed by #483
Closed

Sync command issue with many files. #447

eminegit opened this issue Jun 20, 2022 · 4 comments · Fixed by #483
Assignees
@kucukaslan @boraberke
Labels
good first issue
Milestone
v2.1.0

Comments

@eminegit
Copy link

Hi, the sync command works great with a few files. I am trying to sync 2 million files into the s3 bucket which has more than 15 million files but it stops after 5 minutes with a "killed" error message. I use AWS s3 sync command; it works fine but it is just slower than the s5cmd command.
@sonmezonur
Copy link
Member

@eminegit Hello! Could you share your s5cmd version please? We have released a new version to address this issue (v2.0.0-beta.2)

@eminegit
Copy link
Author

Hi, the version I have is v2.0.0-beta.2-c978e4a. I used 's5cmd_2.0.0-beta.2_Linux-64bit.tar.gz' file to install s5cmd.

@igungor igungor added this to s5cmd Jun 29, 2022
@igungor igungor added this to the v2.1.0 milestone Jun 29, 2022
@misuto
Copy link

misuto commented Jul 6, 2022

Sounds like this could be the same issue as in #441

@kucukaslan kucukaslan moved this to Todo in s5cmd Jul 29, 2022
@kucukaslan kucukaslan mentioned this issue Aug 1, 2022
Closed
@kucukaslan
Copy link
Contributor

Sounds like this could be the same issue as in #441

I agree that this is related to it.

I've tried to reproduce error. I've tried to sync from a directory with huge number of files. I received
fatal error: runtime: out of memory not a "killed" error in Linux. The errors probably means the same thing.

I've also outlined some of the possible solutions at #441 (comment).

@kucukaslan kucukaslan mentioned this issue Aug 9, 2022
Merged
@kucukaslan kucukaslan moved this from Todo to In Progress in s5cmd Aug 16, 2022
igungor pushed a commit that referenced this issue Jun 16, 2023
@kucukaslan
command: use external sort for comparison in sync (#483)
f0ce87d 
It uses external sort instead of in-memory sort to dramatically reduce memory usage in the expense of the speed.
It uses encoding/gob format to store to disk and restore from there.

Fixes #441
Fixes #447
@github-project-automation github-project-automation bot moved this from In Progress to Done in s5cmd Jun 16, 2023
This was referenced Jul 26, 2023
Closed
Merged
@kabilar kabilar mentioned this issue Oct 25, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kucukaslan kucukaslan

@boraberke boraberke

Labels
good first issue
Projects
s5cmd
Archived in project
Milestone
v2.1.0
Development

Successfully merging a pull request may close this issue.

command: use external sort for comparison in sync
7 participants
@igungor @ilkinulas @misuto @sonmezonur @kucukaslan @eminegit @boraberke