How to use stream to return a message even instead of waiting until everything has been processed

[pslxx]

How to use stream to return a message even instead of waiting until everything has been processed

[MaximeThoonsen]

hey @pslxx , there are dedicated method in the ChatInterface like generateStreamOfText.
Does it help you?

[messi89]

@MaximeThoonsen the generateStreamOfText function use guzzle request (not async request), so theoretically the method waits for the whole response from the ollama api.

Is there any way to read streamed stream from ollama response directly ?

[iztok]

+1 Looking to iterate through each stream chunk but stream methods return StreamInterface that doesn't allow this (#78 (comment))

[iztok]

If anyone finds this helpful:

        $streamToIterator = function (StreamInterface $stream): Generator {
            while (!$stream->eof()) {
                yield $stream->read(32); // Adjust the chunk size as needed
            }
        };
        $iteratorStream = $streamToIterator($stream);

        foreach ($iteratorStream as $chunk) {
            // chunks are not token based anymore!
        }

[MaximeThoonsen]

hello @ezimuel, how are you?

It seems there is a lot of questions around streaming. Can we still do streaming with StreamInterface and LLPhant? What is the "clean/simple" working example?

@iztok the code your provided is working for you to get a stream?

[iztok]

@iztok the code your provided is working for you to get a stream?

Yes, this returns an iterateable stream I can use the same as I used the stream from the OpenAI library. One caveat is that this stream's chunks are not tokens but strings of size 32 bytes. I'm then broadcasting these chunks over the WebSocket to my chat clients.

[prykris]

Yes, this returns an iterateable stream I can use the same as I used the stream from the OpenAI library. One caveat is that this stream's chunks are not tokens but strings of size 32 bytes. I'm then broadcasting these chunks over the WebSocket to my chat clients.

@iztok I see how that is a caveat. Does that make any difference in your use case or does it seriously impact the end-user experience?

I am trying to understand the pitfalls I might run into while trying to implement something similar.

[ryanisn]

with ollama, $stream->read(32) waits on the whole response completed before returning the "chunk"

[mortensi]

Thanks for the hints. However, I must be doing something wrong. This does not work for me and I get all the answer at once.

       $qa = new QuestionAnswering(
           $redisVectorStore,
           $embeddingGenerator,
           new OpenAIChat($config)
       );

        $stream = $qa->answerQuestionStream($request->input('q'));

        $streamToIterator = function (StreamInterface $stream): \Generator {
            while (!$stream->eof()) {
                yield $stream->read(32); 
            }
        };
        $iteratorStream = $streamToIterator($stream);

        foreach ($iteratorStream as $chunk) {
            echo $chunk;
        }

Using OpenAI, partial reads work perfectly, though.

       $responseStream = OpenAI::chat()->createStreamed([
           'model' => 'gpt-3.5-turbo',
           'messages' => [
               ['role' => 'user', 'content' => $request->input('q')],
           ],
       ]);

       foreach ($responseStream as $delta) {
           // Stream the delta content back to the client
           echo $delta['choices'][0]['delta']['content'] ?? '';
           ob_flush(); 
           flush();    
       }

       ob_flush();
       flush();

I'm not sure what else I can test to make answerQuestionStream work.

[mortensi]

Solved. I needed ob_flush() and flush() after each chunk to force Laravel send the output.

        $streamToIterator = function (StreamInterface $stream): \Generator {
            while (!$stream->eof()) {
                yield $stream->read(32); 
                ob_flush();
                flush();
            }
        };

        $iteratorStream = $streamToIterator($stream);

        foreach ($iteratorStream as $chunk) {
            echo $chunk;
        }