fix(core): use TextDecoderStream for stream response

[reosablo]

Closes #1492

Description

This PR changes the implementation of streamResponse function to use TextDecoderStream rather than to use TextDecoder for each chunk from streaming response.

The current implementation sometimes break the response non-ASCII characters and this PR will fix it. see #1492 for more detail.

Checklist

• The base branch of this PR is preview, rather than main

[reosablo]

Note:

const stream = (ReadableStream as any).from(response.body);

yield* stream.pipeThrough(new TextDecoderStream("utf-8"));

• response is not an instance of globalThis.Response and response.body is not ReadableStream. So we need to convert response.body into ReadableStream to use TextDecoderStream.
• TypeScript doesn't recognize that ReadableStream has from static method so as any is required.

[sestinj]

Thanks for the contribution! This is quite important : )

[vvvlad]

I believe this causes the following error:
ReadableStream.from is not a function

[reosablo]

I believe this causes the following error:
ReadableStream.from is not a function

I found that ReadableStream.from() seems to be only supported in the latest VSCode.

MDN says ReadableStream.from() is supported Node.js v20.6.0+ and the VSCode built-in Node.js version seems to be like the bellow.

VSCode version	Electron version	Node.js version
May 2024 (version 1.90)	29	20.9.0
April 2024 (version 1.89)	28	18.18.2 (ReadableStream.from() is NOT available)
March 2024 (version 1.88)	28	18.18.2 (ReadableStream.from() is NOT available)

[reosablo]

The bellow code can be a remedy of the error if you want to support older VSCode versions.

// This triple slash directive doesn't seem to be recognized by TypeScript on compile time so we can't use this instead of `as any`.
// /// <reference lib="DOM.AsyncIterable" />

export async function* streamResponse(
  response: Response,
): AsyncGenerator<string, void, undefined> {
  if (response.status !== 200) {
    throw new Error(await response.text());
  }

  if (!response.body) {
    throw new Error("No response body returned.");
  }

  // Use ReadableSteam constructor instead of Readable.from() function
  const body = response.body;
  const stream = new ReadableStream<Uint8Array>({
    async start(controller) {
      for await (const chunk of body as any) {
        controller.enqueue(chunk);
      }
      controller.close();
    },
  }) as any;

  yield* stream.pipeThrough(new TextDecoderStream("utf-8"));
}