How can I stop the request inside on_data.

[kristoph]

What is the cleanest way to halt/stop a request inside on_data. This is specifically to support streaming responses from most LLM endpoints like Open AI and Anthropic.

Thank you!

[iMacTia]

Hi @kristoph, that is a great question!
I'm no expert in streaming so I wouldn't be able to advise here, but I'd be happy to consider adding some sort of support in faraday based on how the different adapters implement it.

I had a quick look at Net:HTTP and I couldn't find a "nice" way to break the socket read cycle: https://ruby-doc.org/3.3.3/stdlibs/net/Net/HTTPResponse.html#method-i-read_body

So from what I can see, the only solution seem to raise an exception.
This could be supported in Faraday by introducing a custom exception (e.g. Faraday::InterruptStreamingResponse) that will be automatically rescued by Faraday if raised from inside the on_data block 🤔

However we first need to check the other clients/adapters as well

[kristoph]

Thank you for the quick response.

Looking at how this is handled in other languages I believe the most common technique is to simply call close on the connection/request and this then end the data stream. In Faraday, I see it calls app.close but I am sorry I don't know the code well enough to chase down what that does. Do you think that would work or should I throw an exception?

More generally I think ideally there should be a method on a request like quit so that in the request block I can quit/end/stop just that request without impacting every other request. ( In Net::Http there is a finish on Http but this throws an exception. )

[iMacTia]

Faraday does support a #close method, but that simply delegates to the adapter and is supposed to be used by adapters who open a persistent connection to know when that can be closed. It was not introduced or intended to close or interrupt (hey, I think I like this term most!) an ongoing request, which I think is a slightly different thing.

Since Net::HTTP doesn't have a persistent connection, calling finish and closing the TCP connection makes sense (which is what close should probably do as well). But in a "persistent" adapter (like Net::HTTP::Persistent) we'd probably want the request being interrupted, but the TCP connection being left intact if possible.

So from a Faraday perspective, it might make sense to introduce a new #interrupt method (name TBD and not final, but you get the idea). It would then be up to the different adapters to support this new method.
The Net::HTTP adapter can implement this by calling #finish.

Would you be willing to take a quick stab at this?
Implementing this would require changes to both faraday and faraday-net_http.
You can see how support for ciphers was recently added as an example: #1581

[kristoph]

Sorry to follow up on this but it still remains a pain point. In Net::HTTP the solution I can use is to call 'finish' on the http session. Is that a viable option for faraday?

[iMacTia]

Sorry this fell through the cracks, I replied in thread to your previous message 👍