Streaming types / APIs in standard library

[Chriscbr]

Use Case

• Downloading a file from a bucket and saving it directly to disk
• Downloading a file from an HTTP response, and performing processing on its output
• Reading a file from disk as a stream, and processing it piece by piece (e.g. to count how many lines it has)
• Streaming a response back from an API endpoint written by the user

Proposed Solution

Dedicated streaming APIs that don't require the entire contents of an API response or file or etc. to be loaded in memory at once.

(side note: maybe file streams and HTTP streams should be treated separately? I'm not sure)

Implementation Notes

References:

• https://streams.spec.whatwg.org/#readablestream
• https://developer.mozilla.org/en-US/docs/Web/API/Streams_API
• https://docs.python.org/3/library/io.html
• https://docs.python.org/3/library/asyncio-stream.html
• https://hexdocs.pm/file_streams/file_streams/file_stream.html

Component

SDK

Community Notes

• Please vote by adding a 👍 reaction to the issue to help us prioritize.
• If you are interested to work on this issue, please leave a comment.
• If this issue is labeled needs-discussion, it means the spec has not been finalized yet. Please reach out on the #dev channel in the Wing Discord.

[Chriscbr]

Related: #5785

[skyrpex]

Use Case

• Downloading a file from a bucket and saving it directly to disk
• Downloading a file from an HTTP response, and performing processing on its output
• Reading a file from disk as a stream, and processing it piece by piece (e.g. to count how many lines it has)
• Streaming a response back from an API endpoint written by the user

• Uploading files by parts, in parallel
• Returning pages of a file from an API endpoint

(side note: maybe file streams and HTTP streams should be treated separately? I'm not sure)

I believe they are the same in the end but I'm no expert in that field so I can't really add much. You already shared Node's streams and the browser API's streams so we have good references there.

[Chriscbr]

Sharing another use case posted by a community member on our Discord:

I need to process a streaming response from a http request and forward the chunks to the client via websockets. Doing it using fetch I could do something like this:

    const aiApiUrl = 'https://api.example.com/ai-stream';
    const response = await fetch(aiApiUrl, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(requestBody)
    });

    if (!response.ok) {
      ws.send(JSON.stringify({ error: 'AI API request failed' }));
      return;
    }

    const reader = response.body
      .pipeThrough(new TextDecoderStream()) // Decode the text stream
      .getReader();

    try {
      // Read the stream and forward each chunk via WebSocket
      while (true) {
        const { done, value } = await reader.read();
        if (done) {
          ws.send(JSON.stringify({ event: 'end' })); // Notify client that the stream ended
          break;
        }
        
        // Send the chunk to the WebSocket client
        ws.send(value);
      }
    } catch (err) {
      console.error('Error during stream processing:', err);
      ws.send(JSON.stringify({ error: 'Stream processing error' }));
    } finally {
      reader.releaseLock();
    }

How can I process a streaming response with Wing via the http library?
If not any suggestions on how I can accomplish this?