Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Uploading large XML Files #123

Closed
rnekk2 opened this issue Oct 10, 2018 · 6 comments
Closed

Uploading large XML Files #123

rnekk2 opened this issue Oct 10, 2018 · 6 comments
Assignees
@azhavoro
Labels
bug Something isn't working
Milestone
0.5.0 - Beta

Comments

@rnekk2
Copy link

rnekk2 commented Oct 10, 2018

I am having issues uploading large XML files. I created a task with 130K images. I uploaded the annotations XML file around 120MB. It says file being processed. Once it says annotations are being saved, it fails. Changed the DATA_UPLOAD_MAX_MEMORY_SIZE value to 300 MB from the default 100MB but still no luck.
@nmanovic nmanovic added the bug Something isn't working label Oct 10, 2018
@nmanovic nmanovic self-assigned this Oct 10, 2018
@nmanovic nmanovic modified the milestones: Alpha 0.3.0, Beta 0.3.0 Oct 10, 2018
@nmanovic
Copy link
Contributor

Hi @rnekk2 ,

Could you please archive your XML file and attach to the issue (or share with us somehow)? We will try to reproduce the problem locally and improve upload functionality somehow.

@rnekk2
Copy link
Author

rnekk2 commented Oct 10, 2018

Hi @nmanovic ,

                         Attached is the zip file with contains the XML

6_Annotations-2.xml.zip

nmanovic added a commit that referenced this issue Nov 10, 2018
@nmanovic
Fix #123 (Uploading large XML Files)
20cf266 
Need to test the patch.
@nmanovic nmanovic mentioned this issue Nov 16, 2018
Merged
nmanovic added a commit that referenced this issue Nov 24, 2018
@nmanovic
Fix upload big xml files for tasks (#199)
1a040a0 
* Fix #123 (Uploading large XML Files)
* Integrated SILK for profiling.
* Don't try to call save_job without objects.
@nmanovic nmanovic mentioned this issue Dec 27, 2018
Closed
@bsekachev
Copy link
Member

This fix works for annotation files which contain large number of shapes.
However there is other case when annotation file contains some tracks each of them is defined for wide range of frames. So, annotation have a short of time to save and we get 504 code again.

@bsekachev bsekachev reopened this Jan 17, 2019
@bsekachev bsekachev modified the milestones: 0.3.0 - Beta, 0.4.0 - Alpha Jan 17, 2019
@nmanovic nmanovic modified the milestones: 0.4.0 - Alpha, 0.4.0 - Beta Jan 17, 2019
@bsekachev bsekachev mentioned this issue Apr 12, 2019
Merged
@nmanovic
Copy link
Contributor

nmanovic commented May 6, 2019

It should be fixed now. Need to check.

@nmanovic nmanovic modified the milestones: 0.5.0 - Alpha, 0.5.0 - Beta Jun 25, 2019
@nmanovic nmanovic assigned azhavoro and unassigned nmanovic Jul 22, 2019
@nmanovic
Copy link
Contributor

@azhavoro , could you please check that it works now with your new changes?

@nmanovic
Copy link
Contributor

nmanovic commented Aug 2, 2019

We completely reimplemented upload annotation method. It should work with huge annotation files.

@nmanovic nmanovic closed this as completed Aug 2, 2019
@azhavoro azhavoro mentioned this issue Feb 4, 2021
Merged
TOsmanov pushed a commit to TOsmanov/cvat that referenced this issue Aug 23, 2021
Save all task data in voc always (cvat-ai#123)
d08be07 
* Save all task data in voc always

* update changelog
@snyk-bot snyk-bot mentioned this issue Oct 19, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@azhavoro azhavoro

Labels
bug Something isn't working
Projects
None yet
Milestone
0.5.0 - Beta
Development

No branches or pull requests
4 participants
@rnekk2 @bsekachev @nmanovic @azhavoro