Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upload failing on dataset with large files #429

Closed
yairl opened this issue Feb 13, 2024 · 1 comment · Fixed by #430
Closed

upload failing on dataset with large files #429

yairl opened this issue Feb 13, 2024 · 1 comment · Fixed by #430

Comments

@yairl
Copy link
Contributor

yairl commented Feb 13, 2024

My dataset - ~640GB - is failing on dagshub upload.
This is apparently due to some extremely large files (~70-80 MB) being grouped together to a single upload step of ~7GB.
@dagshub DagsHub
Copy link

dagshub bot commented Feb 13, 2024

Join the discussion on DagsHub!

@yairl yairl mentioned this issue Feb 13, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

dagshub upload: variable batch size. yairl/dagshub-client
1 participant
@yairl