Add LiteLLM as an agent for model connections #53

alimosaed · 2024-10-21T19:36:16Z

What is the purpose of this change?

Easily connect to various LLMs using LiteLLM with minimal coding effort.

How is this accomplished?

Users can add a LiteLLM agent to a workflow (YAML file) and set the proper model (e.g., model name, api key and base url). Then, they can run the workflow to connect to an LLM.
User can configure the following features for the agent: streaming/non-streaming, embedding/inference, enable/disable history
User can enable load balancer to distribute requests among multiple LLM models.
Refactored chat history and reused methods for LiteLLM and OpenAI components.

Anything reviews should focus on/be aware of?
The model name typically combines the model provider and the specific model name (e.g., azure/chatgpt-v-2). Find the accurate names from the list of Proviers.

How to test?

Install the Solace AI Connector
Install the litellm python module using pip install litellm
Set the OpenAI keys and models in the 'example/llm/litellm_chat.yaml' file
Run the flow by cd src && python3 -m solace_ai_connector.main ../config.yaml
Subscribe to 'demo/question/response' and send a message to the 'demo/question' topic

Repeat steps 3 to 5 for 'litellm_embedding.yaml' and 'litellm_chat_with_history.yaml'

gitstream-cm · 2024-10-21T19:37:09Z

Please mark whether you used Copilot to assist coding in this PR

Copilot Assisted

cyrus2281

Added some comments

src/solace_ai_connector/components/general/litellm/litellm_base.py

src/solace_ai_connector/components/general/litellm/litellm_chat_model_with_history.py

efunneko

A few things to address

docs/components/litellm_chat_model.md

efunneko · 2024-10-24T20:50:30Z

examples/llm/litellm_chat_with_loadbalancing.yaml

+                    # add any other parameters here
+            - model_name: "claude-3-5-sonnet" # model alias
+              litellm_params:
+                    model: ${OPENAI_MODEL_NAME}


This example should use different model names and keys here

efunneko · 2024-10-24T20:54:26Z

src/solace_ai_connector/components/general/litellm/litellm_chat_model_with_history.py

+        return response
+
+    def prune_history(self, session_id, history):
+        current_time = time.time()


I think it may be time to refactor this code, since I expect it is identical to other components

examples/llm/litellm_chat_with_loadbalancing.yaml

examples/llm/litellm_embedding.yaml

src/solace_ai_connector/components/general/litellm/litellm_chat_model_with_history.py

src/solace_ai_connector/components/general/utils/chat_history_handler.py

…ncer

alimosaed

Addressed comments
Replied to some comments

cyrus2281

LGTM

src/solace_ai_connector/components/general/llm/litellm/litellm_base.py

cyrus2281 · 2024-11-01T16:18:07Z

examples/llm/litellm_embedding.yaml

+            source_expression: |
+              template:You are a helpful AI assistant. Please help with the user's request below:
+              <user-question>
+              {{text://input.payload:text}}
+              </user-question>
+            dest_expression: user_data.llm_input:messages.0.content
+          - type: copy
+            source_expression: static:user
+            dest_expression: user_data.llm_input:messages.0.role
+        input_selection:


Embedding example should not modify the user query. It should be used to get back the vector

src/solace_ai_connector/components/general/llm/litellm/litellm_embeddings.py

cyrus2281

Looks good, Thanks

sonarqube-solacecloud · 2024-11-04T13:37:18Z

SonarQube Quality Gate

40.35% Duplicated Lines (%) on New Code (is greater than 3%)

See analysis details on SonarQube

efunneko

Looks good to me. Thanks for all the refactoring. It is much better organized now.

efunneko · 2024-10-25T19:19:48Z

src/solace_ai_connector/components/general/utils/chat_history_handler.py

+from ...component_base import ComponentBase
+from ....common.log import log
+
+class ChatHistoryHandler(ComponentBase):


It feels strange that this inherits from ComponentBase since it isn't a component, however I see that it does use a bunch of the services of the parent (timer, kv_store, etc). I am not necessarily saying we shouldn't do it, but I would be interested in hearing other opinions on it @cyrus2281

alimosaed added 3 commits October 18, 2024 16:35

added litellm component

a3f3870

support chat history

8a7b8a7

trimmed comments

be89285

alimosaed requested a review from efunneko October 21, 2024 19:36

alimosaed self-assigned this Oct 21, 2024

cyrus2281 reviewed Oct 21, 2024

View reviewed changes

alimosaed added 4 commits October 22, 2024 17:49

dynamically get the model parameters

ab3032b

added llm load balancer

0cadcc0

added the AI PR reviewer workflow

2565990

fixed minor issues

9dc1a67

efunneko requested changes Oct 24, 2024

View reviewed changes

alimosaed added 5 commits October 25, 2024 10:12

controlled session id

26b07d3

refactored chat history and reused codes

c3d1dab

resolved conflicts

afcc8ff

fixed minor logging issue

9c87e41

reverted minor changes

60f7b9f

cyrus2281 requested changes Oct 25, 2024

View reviewed changes

alimosaed added 3 commits October 27, 2024 13:25

handle all LiteLLM inferences and embedding requests by the load bala…

f862a44

…ncer

updated documents

a4cf8fd

fix: remove useless import command

07e7f2c

alimosaed commented Oct 28, 2024

View reviewed changes

refactor: restructure the LLM components

ea59e7e

cyrus2281 approved these changes Oct 28, 2024

View reviewed changes

cyrus2281 requested changes Nov 1, 2024

View reviewed changes

src/solace_ai_connector/components/general/llm/litellm/litellm_base.py Outdated Show resolved Hide resolved

cyrus2281 reviewed Nov 1, 2024

View reviewed changes

refactor: divided litellm chat and embedding

de19706

cyrus2281 reviewed Nov 1, 2024

View reviewed changes

src/solace_ai_connector/components/general/llm/litellm/litellm_embeddings.py Outdated Show resolved Hide resolved

fix: update the embedding input format

0e1d545

cyrus2281 reviewed Nov 4, 2024

View reviewed changes

src/solace_ai_connector/components/general/llm/litellm/litellm_embeddings.py Outdated Show resolved Hide resolved

alimosaed added 2 commits November 4, 2024 08:30

fix: update naming

777fbba

fix: update naming

dcf250d

cyrus2281 approved these changes Nov 4, 2024

View reviewed changes

alimosaed requested a review from efunneko November 4, 2024 13:57

efunneko approved these changes Nov 4, 2024

View reviewed changes

alimosaed merged commit 948554f into main Nov 4, 2024
2 of 4 checks passed

efunneko deleted the ap/integrate_litellm branch November 5, 2024 16:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add LiteLLM as an agent for model connections #53

Add LiteLLM as an agent for model connections #53

alimosaed commented Oct 21, 2024 •

edited

Loading

gitstream-cm bot commented Oct 21, 2024

cyrus2281 left a comment

efunneko left a comment

efunneko Oct 24, 2024

efunneko Oct 24, 2024

alimosaed left a comment

cyrus2281 left a comment

cyrus2281 Nov 1, 2024 •

edited

Loading

cyrus2281 left a comment

sonarqube-solacecloud bot commented Nov 4, 2024

efunneko left a comment

efunneko Oct 25, 2024

Add LiteLLM as an agent for model connections #53

Add LiteLLM as an agent for model connections #53

Conversation

alimosaed commented Oct 21, 2024 • edited Loading

gitstream-cm bot commented Oct 21, 2024

cyrus2281 left a comment

Choose a reason for hiding this comment

efunneko left a comment

Choose a reason for hiding this comment

efunneko Oct 24, 2024

Choose a reason for hiding this comment

efunneko Oct 24, 2024

Choose a reason for hiding this comment

alimosaed left a comment

Choose a reason for hiding this comment

cyrus2281 left a comment

Choose a reason for hiding this comment

cyrus2281 Nov 1, 2024 • edited Loading

Choose a reason for hiding this comment

cyrus2281 left a comment

Choose a reason for hiding this comment

sonarqube-solacecloud bot commented Nov 4, 2024

efunneko left a comment

Choose a reason for hiding this comment

efunneko Oct 25, 2024

Choose a reason for hiding this comment

alimosaed commented Oct 21, 2024 •

edited

Loading

cyrus2281 Nov 1, 2024 •

edited

Loading