Large files being split in output

[petewritescode]

Hi, this is a really useful library, so thank you!

I've hit an issue when concatenating large files, whereby the file contents get split up and mixed in with the content of other files.

Issue reproduction

I've created a codesandbox to reproduce the issue:

https://codesandbox.io/p/sandbox/clever-heisenberg-fkfxrt

The two input files in ./data are a.txt (containing thousands of lines of "a") and b.txt (containing a single "b"). There's a build script which runs globcat and outputs to ./dist/output.txt.

Expected outcome - I expect output.txt to contain all the content from a.txt followed by the content from b.txt.
Actual outcome - The content from b.txt is in the middle if the output, with thousands of "a"s before and after it.

Possible cause

I have limited experience with Node streams but I think the issue might lie here: https://github.com/smonn/globcat/blob/main/src/lib/files-to-stream.ts. Specifically this bit (shortened for clarity):

for (const file of files) {
  const stream = fs.createReadStream(file)
  passthrough = stream.pipe(passthrough, { end: false })
}

As I understand it, because this happens within a loop, it means that files are being streamed to passthrough simultaneously, so their order is not guaranteed. I imagine this is fine as long as each file is below the highWaterMark (64KiB by default) and is therefore processed as a single chunk.

However, files larger than this will be broken into chunks, and those chunks may be delayed and processed after chunks from a different file, making it possible for file content to be jumbled up as in my reproduction.

Possible fix

I've fixed this locally by writing chunks synchronously. As I say, I'm no expert on Node streams so I don't know whether there are any issues with this solution, but I've included it in case it's helpful:

for (const file of files) {
  const stats = await fsPromises.stat(file)
  if (stats.isFile()) {
    const stream = fs.createReadStream(file)
    stream.once('end', () => {
      queueSize--
      if (queueSize === 0) {
        passthrough.end()
      }
    })
    for await (const chunk of stream) {
      passthrough.write(chunk)
    }
  } else {
    throw new Error('Not a file: ' + file)
  }
}

Thanks again,

Pete

[Jason3S]

@smonn,

Any movement on this issue?

I got bit by this bug when upgrading from 2.0.1 to 3.0.1.

There is also and issue with the limit on the number of active streams.

[smonn]

@Jason3S Thanks for this very detailed issue! I'll dive into it and get it sorted out.

[smonn]

Fixed in 3.1.0, let me know if it's still causing troubles. Thanks!