File Upload: Uploading larger zip files never complete, no feedback to user.

[kcondon]

Begin with an unpublished draft, no files.

Upload two zip files, 1 22MB with 60 non tabular files, 1 325MB with 225 non tabular files. When I upload them, they seem to upload relatively quickly but then stay on the upload in progress view and never complete, no clear error in logs.

Those two files are: SoilGrids-for-DSSAT-10km v1.0 (by country).zip (312MB; 225 files inside) & SoilGrids-for-DSSAT-10km v1.0 (uncertainty estimates).zip (22MB; 60 files inside) for IFPRI. See RT 228427 for files and details.

The zip upload does seem to partly work since some files are uploaded. When canceling and trying again, noticed duplicated files were uploaded.

Deleted files and tried manually uploading them. This also leaves files on the file system, so manually uploading complains about duplicates on the file system. They need to be cleared out from the server side before you can continue.

I marked this as critical because it really prevents a user from uploading files in this scenario and they don't seem to be able to reset and try again. Also, uploading as zip seems likely to be the way to upload for users with lots of files.

I checked the server file system and there were no files saved to the files directory for the dataset but there were a bunch of files in the temporary files upload directory in the generated directory. It looks files are stranded there whenever there is an error combined with the duplicate file checking code must look there rather than in the dataset files directory. Also need to check during file upload transaction so most likely it uses generated for that rather than in memory list.

This explains why delete does not affect duplicates, delete only deletes things successfully uploaded, cannot remove the stranded/orphan files in the generated directory, cancel neither.

I think there may be a more fundamental issue with file upload progress bar never completing. When I dragged and dropped the 60 files from the smaller zip rather than unploading the zip itself, it uploaded 6 files successfully, then all the rest showed the progress bar reach the end, one after another, but never close. The files are in the generated directory. So, whatever causes the progress bar not to complete prevents the files from appearing in the uploaded list and likely leads to stranded files in the generated directory.

From the server log, it looks like it may be generating thumbnails. Tried uploading individual files and found that many of the files in the smaller zip file are tiff images. The larger tiff images, 2.9MB for example, do not complete uploading and then appear to block the rest?

Also, at one point I thought we were uploading the smaller files first for efficiency. On a home network or slower network, this would be really useful since now I'm stuck watching several large files upload.

[kcondon]

Consolidating to #2482, closing.