Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add litellm #450

Closed
wants to merge 4 commits into from
Closed

Add litellm #450

wants to merge 4 commits into from

Conversation

PCSwingle
Copy link
Member

TODO: Add documentation saying to check litellm to see how to use other models.

Pull Request Checklist

  • Documentation has been updated, or this change doesn't require that

@jakethekoenig
Copy link
Member

I'm having some trouble getting it to work with ollama. We need to set litellm.drop_params to true because the ollama api doesn't support response_format. And we also need to pass in an api_base to the litellm completions. After I did that though I got the following exception which may be a bug with litellm itself? Not sure.

Traceback (most recent call last):
  File "/Users/jakekoenig/mentat/.venv/lib/python3.11/site-packages/litellm/llms/ollama.py", line 260, in ollama_async_streaming
    status_code=response.status_code, message=response.text
                                              ^^^^^^^^^^^^^
  File "/Users/jakekoenig/mentat/.venv/lib/python3.11/site-packages/httpx/_models.py", line 574, in text
    content = self.content
              ^^^^^^^^^^^^
  File "/Users/jakekoenig/mentat/.venv/lib/python3.11/site-packages/httpx/_models.py", line 568, in content
    raise ResponseNotRead()
httpx.ResponseNotRead: Attempted to access streaming response content, without having called `read()`.

@PCSwingle
Copy link
Member Author

Replaced by #451.

@PCSwingle PCSwingle closed this Jan 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants