streams api for upload/download?

[ronag]

Support for node streams in an upload/download API would be useful.

[JCMais]

This could be added to the Curl and/or curly wrappers. But I don't have time right now to implement it. If anyone is willing to work on this feature feel free to comment here

[JCMais]

Recently I've been giving some thought to this feature. What do you think would be the best way to implement this @ronag ?

Here are some alternatives I've been thinking:

Curl.prototype.perform would return a Duplex stream when the Streamable feature is enabled (this would override all other feature flags we have so far).

curl.enable(CurlFeature.Streamable)
const duplex = curl.perform()

// upload and/or download data

Making Curl itself a Duplex stream would be harder, as it's already extending EventEmitter and emitting their own events: data, header, end and error.

This implementation would have a disvantage, which is not being possible to easily receive headers / statusCode before the download, a solution for that would be to still emit the data event after the headers are received, then the client would have the option to only start reading the stream in there.

This API would probably be confusing for consumers, as it does not really make sense to write to the Duplex after you started reading it.

Separated Methods

Having this separately probably makes more sense.

download

curl.enable(CurlFeature.StreamResponse)

Then the end event would be emitted immediately after all the headers are received, and the data parameter would be a ReadableStream.

curl.on('end', (statusCode, data, headers) => {
  // data is a ReadableStream
})
curl.perform()

upload

curl.setUploadStream(readableStream)

Then a custom READFUNCTION would be added automatically to handle passing the stream contents to libcurl.

[JCMais]

I've decided to go with the latter, here is the proposed API (showing both download and upload):

curly

  const inFilePath = './some.in.file.zip'
  const outFilePath = './some.out.file.zip'

  const readableStreamToUpload = fs.createReadStream(inFilePath)

  const { statusCode, data: streamToDownload, headers } = await curly.get<Readable>(
    'http://some-url.com/api/upload',
    {
      // Upload related options
      // PUT upload
      upload: true,
      // so we do not need to specify the payload size
      httpHeader: ['Transfer-Encoding: chunked'],
      // smallest buffer possible to cause the max amount of buffering
      bufferSize: 16 * 1024,
      // the stream to upload
      curlyStreamUpload: readableStreamToUpload,
      // Download as a stream
      curlyStreamResponse: true,
      // show libcurl default progress bar
      curlyProgressCallback() {
        return CurlProgressFunc.Continue
      },
    },
  )

  const writableStream = fs.createWriteStream(outFilePath)

  // then we can just pipe it
  streamToDownload.pipe(writableStream)

  // or we can use async iterators:
  for await (const chunk of streamToDownload) {
     writableStream.write(chunk)
  }
  writableStream.end()

Curl

  const inFilePath = './some.in.file.zip'
  const outFilePath = './some.out.file.zip'

  const readableStreamToUpload = fs.createReadStream(inFilePath)

  const curl = new Curl()

  curl.enable(CurlFeature.StreamResponse)

  curl.setOpt('URL', 'http://some-url.com/api/upload')
  // PUT upload
  curl.setOpt('UPLOAD', true)
  // so we do not need to specify the payload size
  curl.setOpt('HTTPHEADER', ['Transfer-Encoding: chunked'])
  // smallest buffer possible to cause the max amount of buffering
  curl.setOpt(Curl.option.UPLOAD_BUFFERSIZE, 16 * 1024)

  // the stream to upload
  curl.setUploadStream(readableStreamToUpload)

  // show libcurl default progress bar
  curl.setStreamProgressCallback(() => {
    return CurlProgressFunc.Continue
  })

  curl.on('end', (statusCode, data) => {
    console.log('\n'.repeat(5))
    console.log(
      `curl - end - status: ${statusCode} - data length: ${data.length}`,
    )
    curl.close()
  })

  curl.on('error', (error, errorCode) => {
     console.log('\n'.repeat(5))
     console.error('curl - error: ', error, errorCode)
     curl.close()
  })

  curl.on('stream', async (streamToDownload, _statusCode, _headers) => {
    const writableStream = fs.createWriteStream(outFilePath)
    
    // pipe
    streamToDownload.pipe(writableStream)
    // or any other usage...
  })

  curl.perform()

Notes

• When using curlyStreamResponse: true, if an error is thrown in the Curl instance after the curly call has been resolved (it resolves as soon as the stream is available) it will cause an error event to be emitted on the stream itself, so the consumer can handle these, if they need want to.
• If the readableStreamToUpload is destroyed before libcurl finishes uploading it, the error Curl upload stream was unexpectedly destroyed (Code 42) will be emitted in the Curl instance. If the stream was destroyed with an error, this error will passed to the event instead.
• Calling streamToDownload.destroy() will always cause the Curl instance to emit the error event. Even if an error argument was not supplied to stream.destroy().

I should be able to make this available later this week.

[JCMais]

Closed via #252