Ollama support issue. #635

JayQuimby · 2024-04-03T03:47:32Z

Describe the bug

When trying to configure OpenDevin to run with Ollama there are requests that are being sent to the ollama server like this:

The post request should look like this:
"POST /chat/completions HTTP/1.1"

Setup and configuration

Current version:

commit 5c640c99cafb3c718dad60f377f3a725a8bab1de (HEAD -> local-llm-flag, origin/main, origin/HEAD, main)

My config.toml and environment vars (be sure to redact API keys):

WORKSPACE_DIR="./workspace"
LLM_BASE_URL="http://localhost:8000"
LLM_MODEL="ollama/starcoder2:15b"
LLM_EMBEDDING_MODEL="ollama/starcoder2:15b"

My model and agent (you can see these settings in the UI):

Model: ollama/starcoder2
Agent: MonologueAgent

Commands I ran to install and run OpenDevin:

git clone ...
make build
make start-backend
make start-frontend

Steps to Reproduce:

In opendevin/llm/llm.py in __init__ replace self.model = model if model else DEFAULT_MODEL_NAME with self.model_name = DEFAULT_MODEL_NAME
Run your local model on litellm litellm --model ollama/starcoder2:15b --port 8000
Run make build then make start-backend and make start-frontend
Ask devin to do anything ex 'make a hello world script in python'
Observe 404 errors spammed in litellm server log

Logs, error messages, and screenshots:
This is a log from the backend server running from make start-backend steps 0-99 all look the same.

==============
STEP 99

PLAN:
please make a simple flask app that says hello world.
Traceback (most recent call last):
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1436, in function_with_retries
    response = original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 386, in _completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 334, in _completion
    deployment = self.get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 2313, in get_available_deployment
    raise ValueError(f"No healthy deployment available, passed model={model}")
ValueError: No healthy deployment available, passed model=ollama/starcoder2:15b

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/utils/monologue.py", line 31, in condense
    resp = llm.completion(messages=messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 328, in completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 325, in completion
    response = self.function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1419, in function_with_fallbacks
    raise original_exception
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1344, in function_with_fallbacks
    response = self.function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1496, in function_with_retries
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1462, in function_with_retries
    response = original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 386, in _completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 334, in _completion
    deployment = self.get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 2313, in get_available_deployment
    raise ValueError(f"No healthy deployment available, passed model={model}")
ValueError: No healthy deployment available, passed model=ollama/starcoder2:15b

ERROR:
Error condensing thoughts: No healthy deployment available, passed model=ollama/starcoder2:15b
Traceback (most recent call last):
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1436, in function_with_retries
    response = original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 386, in _completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 334, in _completion
    deployment = self.get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 2313, in get_available_deployment
    raise ValueError(f"No healthy deployment available, passed model={model}")
ValueError: No healthy deployment available, passed model=ollama/starcoder2:15b

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/utils/monologue.py", line 31, in condense
    resp = llm.completion(messages=messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 328, in completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 325, in completion
    response = self.function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1419, in function_with_fallbacks
    raise original_exception
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1344, in function_with_fallbacks
    response = self.function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1496, in function_with_retries
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1462, in function_with_retries
    response = original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 386, in _completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 334, in _completion
    deployment = self.get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 2313, in get_available_deployment
    raise ValueError(f"No healthy deployment available, passed model={model}")
ValueError: No healthy deployment available, passed model=ollama/starcoder2:15b

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/quimbo/OpenDevin/opendevin/controller/agent_controller.py", line 112, in step
    action = self.agent.step(self.state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/agent.py", line 153, in step
    self._add_event(prev_action.to_dict())
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/agent.py", line 96, in _add_event
    self.monologue.condense(self.llm)
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/utils/monologue.py", line 36, in condense
    raise RuntimeError(f"Error condensing thoughts: {e}")
RuntimeError: Error condensing thoughts: No healthy deployment available, passed model=ollama/starcoder2:15b

OBSERVATION:
Error condensing thoughts: No healthy deployment available, passed model=ollama/starcoder2:15b
Exited before finishing

Additional Context

Litellm for local models is expecting api calls in the following format:

From: http://localhost:8000/#/

I know that the problem is whatever is managing the api calls is set to call /api/generate/ because this is the convention, but for local server that is not supported. I do not know where to look to fix this, any ideas?

The server responds when I test it like this:

def query_local_llm(prompt, limit=TOKEN_LIMIT):
    # Replace with your actual server address and port
    url = "http://0.0.0.0:8000/chat/completions"
    payload = {
        "model": "ollama/mistral",
        "messages" : [{"content": prompt, "role": "user"}],
        "max_tokens": limit
    }
    response = requests.post(url, json=payload)

The text was updated successfully, but these errors were encountered:

stratte89 · 2024-04-03T04:15:49Z

EDIT: a guide for ollama was added
08a2dfb

this config was working for me, before the devin update today, its not right now with the latest update, idk if its about the config tho

LLM_API_KEY="ollama"
LLM_BASE_URL="http://0.0.0.0:11434"
LLM_MODEL="ollama/mistral"
LLM_EMBEDDING_MODEL="local"
WORKSPACE_DIR="./workspace"

Also make sure to start the ollama serve after loading the mode and that you are using the correct ollama server port. If the server is already running load the model and kill the server process. I am using sudo fuser -k -n tcp 11434 to kill it but im on ubuntu. Btw I tried it on windows using wsl and i wasn't able to get it to work, since wsl is using a virtual network. There is a workaround by creating a wsl config file to mirror you host network, it didn't work for me tho for someone else it did so you have you try it yourself.

"Open wsl config file C:\Users%username%.wslconfig (create one if it doesnt exist), and add this:

[wsl2]
networkingMode=mirrored"

if your ollama server is listening on 0.0.0.0:port then change the Makefile adding --host 0.0.0.0 and --host

Start backend

start-backend:
@echo "Starting backend..."
@python -m pipenv run uvicorn opendevin.server.listen:app --port $(BACKEND_PORT) --host 0.0.0.0

Start frontend

start-frontend:
@echo "Starting frontend..."
@cd frontend && npm run start -- --port $(FRONTEND_PORT) --host

Also what i just mentioned is, that you're using the cmd, you should use a wsl terminal like anaconda command prompt in windows

* doc: Guide for using local LLM with Ollama

imtpalmer · 2024-04-04T14:17:06Z

I tested this yesterday and again this morning using Ollama local with OpenDevin patch-11 and it works. However, it fails when I switch to patch-12 and the main branch.

JayQuimby · 2024-04-04T19:32:55Z

@imtpalmer An updated guide got merged this morning here

JayQuimby added the bug Something isn't working label Apr 3, 2024

JayQuimby referenced this issue Apr 3, 2024

Guide for Ollama local LLM (#615)

08a2dfb

* doc: Guide for using local LLM with Ollama

JayQuimby closed this as completed Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ollama support issue. #635

Ollama support issue. #635

JayQuimby commented Apr 3, 2024 •

edited

Loading

stratte89 commented Apr 3, 2024 •

edited

Loading

imtpalmer commented Apr 4, 2024

JayQuimby commented Apr 4, 2024

Ollama support issue. #635

Ollama support issue. #635

Comments

JayQuimby commented Apr 3, 2024 • edited Loading

Describe the bug

Setup and configuration

Additional Context

stratte89 commented Apr 3, 2024 • edited Loading

Start backend

Start frontend

imtpalmer commented Apr 4, 2024

JayQuimby commented Apr 4, 2024

JayQuimby commented Apr 3, 2024 •

edited

Loading

stratte89 commented Apr 3, 2024 •

edited

Loading