useChat chunk

[BrianHung]

#3225

Let's clients access chunks that are proceeded from the data stream via useChat onChunk. Useful for cache-ing responses state-fully on the client and then replaying them.

// parrot.ts
export async function POST(req: Request) {
  const { chunks } = await req.json();
  if (chunks) {
    const parts = chunks.map(({ type, value }) => formatStreamPart(type, value));
    const stream = convertArrayToReadableStream(parts)
      .pipeThrough(new TextEncoderStream())
    const response = new Response(stream, {
      headers: {
        contentType: 'text/plain; charset=utf-8',
        dataStreamVersion: 'v1',
      }
    });
    return response;
  }
  // If no chunks, call normal API
}

[lgrammel]

My thoughts:
a) not sure if we want to expose all chunk types like this
b) would need to be implemented in svelte, vue, solid (but good to hold off until a) is clear)

[BrianHung]

not sure if we want to expose all chunk types like this

What’s an alternative if we want to store chunks themselves? Is there a higher level API that’ll be more immune to changes if stream parts were changed?

[lgrammel]

@BrianHung the stream parts are standardized in the stream data protocol (but additions might happen). you could also record using a fetch interceptor for now (just write the stream parts as string to disk and then read them as strings and send them back).

[BrianHung]

Realized onResponse is a way to implement this too:

import { readDataStream } from 'ai';

// Passed into `useChat`
onResponse: async (resp) => {
	const onChunk = async () => {
		const response = resp.clone();
		if (!response.body) return;
		const reader = response.body.getReader()
		for await (const chunk of readDataStream(reader)) {
			console.log("onChunk", { chunk });
		}
	}
	onChunk();
},