Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement fallbacking for lang model pool #43

Closed
roma-glushko opened this issue Dec 26, 2023 · 0 comments · Fixed by #68
Closed

Implement fallbacking for lang model pool #43

roma-glushko opened this issue Dec 26, 2023 · 0 comments · Fixed by #68
Assignees
@roma-glushko
Labels
area:resiliency gep:required size:large
Milestone
Glide: Private Pr...

Comments

@roma-glushko
Copy link
Member

roma-glushko commented Dec 26, 2023 •
edited
Loading

Implement fallbacking on provider failures to another healthy provider.

Create a GEP to explain how will that work (GEP0005).
@roma-glushko roma-glushko self-assigned this Dec 30, 2023
@roma-glushko roma-glushko mentioned this issue Jan 1, 2024
Merged
roma-glushko added a commit that referenced this issue Jan 7, 2024
@roma-glushko
#43: Implemented token bucket based on atomic operations
fb40aab
@roma-glushko roma-glushko linked a pull request Jan 7, 2024 that will close this issue
#43: Fallback Mechanism for Lang Models #68
Merged
roma-glushko added a commit that referenced this issue Jan 7, 2024
@roma-glushko
#43 Covered token bucket by tests
4824c78
roma-glushko added a commit that referenced this issue Jan 7, 2024
@roma-glushko
#43 Refactored the time units
2c79303
roma-glushko added a commit that referenced this issue Jan 7, 2024
@roma-glushko
#43: Added health and rate limit trackers
6824752
roma-glushko added a commit that referenced this issue Jan 7, 2024
@roma-glushko
#43: Moved Config to Model builder to the config struct & covered rat…
20ef234 
…elimit tracker by tests
roma-glushko added a commit that referenced this issue Jan 7, 2024
@roma-glushko
#43: Moved LangRouter building to configs & used atomic counter in th…
06ebe93 
…e priority routing
roma-glushko added a commit that referenced this issue Jan 7, 2024
@roma-glushko
#43 Defined a LangModel struct and merged all health tracking code there
f64828c
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43 Added exp retry class and error budget field
014d0cd
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43 linting
3cd26b5
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43 Updated Swagger specs
9008a33
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43 Connected retry config to the router
de57035
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43 Fixed the linting issues
31bc73b
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43: Fixed issue with bucket replenish rate & covered router workflow…
3f7b4c5 
… by tests
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43: Covered router workflow by tests
020cf3a
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43: Covered the routing retry logic by tests
81b61bc
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43 fixed tests
a97e296
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43: Removed unneeded method
8abb389
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43: silenced unused params
c112d57
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43: updated openapi spec
c71432f
roma-glushko added a commit that referenced this issue Jan 8, 2024
@roma-glushko
#43: Corrected the command name
11d8441
roma-glushko added a commit that referenced this issue Jan 9, 2024
@roma-glushko
#43: Exposed errors from OpenAI client to control the fallback flow
a258942
roma-glushko added a commit that referenced this issue Jan 9, 2024
@roma-glushko
#43: Added ModelID to track what model in the pool has served the req…
dea3e0c 
…uest
roma-glushko added a commit that referenced this issue Jan 9, 2024
@roma-glushko
#43 Fixing the infinite loop on error
39c65f4
roma-glushko added a commit that referenced this issue Jan 9, 2024
@roma-glushko
#43 added model ID info
7407148
roma-glushko added a commit that referenced this issue Jan 9, 2024
@roma-glushko
#43: Fixed tests
e2df092
roma-glushko added a commit that referenced this issue Jan 9, 2024
@roma-glushko
#43: Using model name from the config & fixing the provider name
d4d400d
roma-glushko added a commit that referenced this issue Jan 11, 2024
@roma-glushko
#43 Covered error budget by tests
a5a6871
roma-glushko added a commit that referenced this issue Jan 11, 2024
@roma-glushko
🧘‍♀️ #43: Fallback Mechanism for Lang Models (#68)
7a58b35 
- Implemented lang router handling flow
- Implemented the priority routing strategy
- Implemented exponential retry when no model is healthy
- Implemented model health (based on the token bucket) and basic rate limit tracker
- Implemented a custom error budget field to accept budgets in form of human-friendly string e.g. "10/m".
- Moved build/factory methods on the config level
@roma-glushko roma-glushko mentioned this issue Jan 21, 2024
Merged
roma-glushko added a commit that referenced this issue Jan 21, 2024
@roma-glushko
📦 Release: v0.0.1-rc.1 (#93)
1c3e68a 
- ✨ [providers] Support for OpenAI Chat API #3 (@mkrueger12 )
- ✨ [API] #54 Unified Chat API  (@mkrueger12 )
- ✨ [providers] Support for Cohere Chat API #5 (@mkrueger12 )
- ✨ [providers] Support for Azure OpenAI Chat API #4  (@mkrueger12 )
- ✨ [providers] Support for OctoML Chat API #58 (@mkrueger12 )
- ✨ [routing] The Routing Mechanism, Adaptive Health Tracking, and Fallbacks #42 #43 #51 (@roma-glushko)
- ✨ [routing] Support for round robin routing strategy #44 (@roma-glushko)
- ✨ [routing] Support for the least latency routing strategy #46 (@roma-glushko)
- ✨ [routing] Support for weighted round robin routing strategy #45 (@roma-glushko)
- ✨ [providers] Support for Anthropic Chat API #60 (@mkrueger12 )
- ✨ [docs] OpenAPI specifications #22 (@roma-glushko )
- 🔧 [chores] Inited the project #6 (@roma-glushko)
- 🔊 [telemetry] Inited logging  #14 (@roma-glushko)
- 🔧 [chores] Inited Glide's CLI #12 (@roma-glushko)
- 👷 [chores] Setup CI workflows #8 (@roma-glushko)
- ⚙️ [config] Inited configs #11 (@roma-glushko)
-  🔧 [chores] Automatic coverage reports #39 (@roma-glushko)
- 👷 [build] Setup release workflows #9 (@roma-glushko)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@roma-glushko roma-glushko

Labels
area:resiliency gep:required size:large
Projects
None yet
Milestone
Glide: Private Preview (MVP)
Development

Successfully merging a pull request may close this issue.

#43: Fallback Mechanism for Lang Models EinStack/glide
1 participant
@roma-glushko