How to ensure zipWriter.close() waits for streamed data to finish downloading?

[mbaquerizo]

Excuse the question because I'm not sure exactly what my issue is. But what my problem seems to be is that zipWriter.close() is happening before a file that I'm fetching is completely fetched. I'm trying to download multiple files using fetch, and passing the response.body (a ReadableStream) to zipWriter.add. Then I'm awaiting the Promise.all for these before calling zipWriter.close().

This is working for several small files, but when I have a large-ish file (20mb) selected, it seems like this file is still being downloaded after zipWriter.close() is called, and a Fetch failed error is thrown. If I fetch the same file without adding it to the zipWriter, there is no error; I can fetch it independently of the zip logic just fine.

Update: fixed code below. This works as described above.

type Category = 'cat1' | 'cat2' | 'cat3' | 'cat4' | 'cat5';

type FileData = {
  name: string;
  url: string;
};

export const createZip = async (
  mapEntryMetaDataPromises: (
    zipWriterAdd: typeof ZipWriter.prototype.add,
  ) => Promise<EntryMetaData>[],
) => {
  console.log('Creating zip...');

  const blobWriter = new BlobWriter('application/zip');
  const zipWriter = new ZipWriter(blobWriter);

  try {
    const promises = mapEntryMetaDataPromises(zipWriter.add.bind(zipWriter));

    await Promise.all(promises);

    console.log('Closing zipWriter...');
    return zipWriter.close();
  } catch (error) {
    console.error('Error in createZip: ', error);
    throw new Error('Error in createZip');
  }
};

export const fetchSelectedFilesZipDownload = async (
  selectedFiles: { [key in Category]?: FileData[] },
  fileName: string,
) => {
  const { cat1, cat2, cat3, cat4, cat5 } = selectedFiles;
  try {
    const zipBlob = await createZip((addFileData) => {
      const processFileCategory = (
        category: string,
        files: FileData[] = [],
      ) => {
        console.log('Processing Category: ', category);
        return files.map(async (file: FileData) => {
          console.log('Processing File: ', file.name);

          const response = await fetch(file.url);

          // const fileContent = new BlobReader(await response.blob());
          const fileContent = response.body || undefined;
          return addFileData(`${category}/${file.name}`, fileContent);
        });
      };

      return [
        ...processFileCategory('Category 1', cat1),
        ...processFileCategory('Category 2', cat2),
        ...processFileCategory('Category 3', cat3),
        ...processFileCategory('Category 4', cat4),
        ...processFileCategory('Category 5', cat5),
      ];
    });

    console.log('ZIP BLOB:::', zipBlob);
    saveBlobToClient(zipBlob, fileName);
  } catch (error) {
    console.log('Error in fetchSelectedFilesZipDownload:', error);
    let errorMessage = 'Something went wrong';
    if (error instanceof Error) {
      errorMessage = error.message;
    }
    throw new Error(errorMessage);
  }
};

export const saveBlobToClient = (blob: Blob, fileName: string) => {
  const zipUrl = URL.createObjectURL(blob);

  const tempLink = document.createElement('a');
  tempLink.href = zipUrl;
  tempLink.ariaHidden = 'true';
  tempLink.setAttribute('download', fileName);

  document.body.appendChild(tempLink);
  tempLink.click();

  document.body.removeChild(tempLink);
  URL.revokeObjectURL(zipUrl);
};

I did find this discussion (#442) which is the for same error I'm seeing, but I haven't confirmed if it was due to the disk being full. I'm also not really sure what to make of it, or if it means that my machine just can't handle the file size.

My internet speeds have varied during my testing, but even when download was relatively faster, the error still came up, just quicker, so I don't think it has to do with the speed of the download.

I guess I want to know (if anyone has some insight)

• What is causing this?
• Is my approach is ok, or is there a better/correct way to work with multiple fetch requests?

---

Screenshots

Logs in failed state

Note: I don't get 'Closing zipWriter...' here, so it seems like we're not making it to the zipWriter.close() call. BUT we then see 'Cannot close a ERRORED writable stream'. Not sure what I can tell from this.

Network tab

File is fetched completely (21.4mb), but this happens well after the above errors.

Logs in success state (multiple smaller files)

Network tab

[gildas-lormeau]

I did not test the code (yet) but you are supposed to call await zipWriter.close(); just before return readable;. The call to zipWriter.close() is mandatory (and not present in the code).

[mbaquerizo]

RIght, I did have that but accidentally removed it when removing some more sensitive stuff in my example. Edited my post.

[gildas-lormeau]

Can you please test this code and confirm you can reproduce the issue? It cannot be run as is because you are calling await zipWriter.close() in a synchronous lambda function. You must replace await with return. Can you also add more logs to verify your assertion? If this is the case, I need to know an approximate number of entries in the zip file and an approximate size of the resulting zip file.

Finally, can you reproduce the issue (with the same files) with this demo? https://gildas-lormeau.github.io/zip.js/demos/demo-create-file.html

[mbaquerizo]

Sorry to waste your time, yeah I didn't really check over the pasted code much as I'm trying a bunch of different things and changing it around. I'll be able to update this tomorrow. Sorry about that.

For some more info though, this fails when I try to select a single 20mb pdf file (unzipped) and fetching that.

When I download a different set of 3 smaller files (4.9mb total unzipped), I successfully get my archive downloaded (4.7mb zipped).

[mbaquerizo]

@gildas-lormeau I have updated the code above and added screenshots of logs. I am also seeing the same error in the demo you linked when using the same file that errored out on my end:

[gildas-lormeau]

Thank you for the additional info. It looks like the issue is indeed similar to the one you mentioned (#442). If you think there is enough space on your disk then another possible explanation could be data corruption associated to your profile in Chrome. You could try to use a new profile to verify if it's the cause of the issue. You can use the switch --user-data-dir=path when launching Chrome for this.

[nenge123]

          const response = await fetch(file.url);
          // const fileContent = new BlobReader(await response.blob());
          const fileContent = response.body || undefined;

this is error!
if failed fetch response===undefined
so

const response = await fetch(file.url).catch(e=>undefined);
if(response){
      return addFileData(`${category}/${file.name}`, response.body);
}

[mbaquerizo]

Thanks for pointing that out, I'll update that for sure

[mbaquerizo]

@nenge123 I took a look at this and while I should probably handle failed fetch, response.body can be null, which is not accepted by zipWriter.add. undefined is accepted by zipWriter.add, which is why it works. I'm not sure what the use case for passing undefined to zipWriter.add is. But strangely, when I check for response.body existence, before calling addFileData, I get this error:

TypeError: Failed to execute 'getWriter' on 'WritableStream': Cannot create writer when WritableStream is locked
    at writeData (zip-writer.js:1088:32)
    at createFileEntry (zip-writer.js:548:9)
    at getFileEntry (zip-writer.js:392:21)
    at async addFile (zip-writer.js:322:15)
    at async ZipWriter.add (zip-writer.js:115:11)

[nenge123]

your error is response.body is locked!
response.body only read once!!!
do not await response.blob(); ........ more code ..... await response.blob();
if you read more you can response2 = response.clone();

const response = await fetch(file.url).catch(e=>e.message);
if(response instanceof Response){
       //if no-cros  response 404 500
       if(response.status!==200){
             return ; 
      }
     //or check mime
     if(response.headers.get('content-type') != 'image/png'){
        return;
     }
      return addFileData(`${category}/${file.name}`, response.body);
}
console.log(response );

[nenge123]

 if(response.headers.get('content-length')<xxx){
    return;
 }

[nenge123]

if you not need the progress,
I sussguest you use return addFileData(${category}/${file.name}, await response.blob());

    const response = await fetch(url).catch(e=>undefined);
    if(response){
        const maxsize = parseInt(response.headers.get('content-length')||0);
        const reader  = response.body.getReader();
        const chunk = [];
        let havesize = 0;
        while(true){
            const { done, value } = await reader.read();
            if(done){
                break;
            }
            havesize += value.byteLength;
            progress(havesize,maxsize,value.byteLength);
            chunk.push(value);
        }
        return await (new Blob(chunk)).arrayBuffer();
    }