#163: Evolving the resiliency & routing architecture to support chatStream

[roma-glushko]

Integrating the streaming chat workflow into the existing resiliency & routing mechanisms.

Changes

• Separated sync and streaming chat schemas
• Extracted assumptions on where to find latency from routing strategies to a separate LatencyGetters that can be different for different models/workflows
• Elaborated the client provider chatStream() interface. Clients now expose a response channel instead of being provided with by caller
• Connected the stream chat workflow to latency & health tracking
• Refined the chatStream() method of clients to return a stream struct
• Separated latency tracking of the streaming workflow from the sync chat workflow
• defined a new HealthTracker to incorporate all health tracking logic

[codecov]

Codecov Report

Attention: Patch coverage is 74.95108% with 128 lines in your changes are missing coverage. Please review.

Project coverage is 64.20%. Comparing base (1add6e3) to head (c3b7470).

Files	Patch %	Lines
pkg/providers/lang.go	75.75%	22 Missing and 2 partials ⚠️
pkg/routers/router.go	76.38%	14 Missing and 3 partials ⚠️
pkg/providers/openai/errors.go	60.00%	12 Missing and 2 partials ⚠️
pkg/providers/testing/lang.go	85.26%	12 Missing and 2 partials ⚠️
pkg/routers/config.go	76.27%	13 Missing and 1 partial ⚠️
pkg/api/http/handlers.go	0.00%	10 Missing ⚠️
pkg/providers/openai/chat_stream.go	80.76%	9 Missing and 1 partial ⚠️
pkg/providers/clients/stream.go	0.00%	9 Missing ⚠️
pkg/providers/anthropic/chat_stream.go	0.00%	2 Missing ⚠️
pkg/providers/azureopenai/chat_stream.go	0.00%	2 Missing ⚠️
... and 7 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #166      +/-   ##
===========================================
+ Coverage    61.39%   64.20%   +2.81%     
===========================================
  Files           65       70       +5     
  Lines         2839     3079     +240     
===========================================
+ Hits          1743     1977     +234     
- Misses         994     1000       +6     
  Partials       102      102              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.