2.2.1 Backend: Ollama

Ollama

Handle: ollama URL: http://localhost:33821

Ergonomic wrapper around llama.cpp with plenty of QoL features.

Ollama is connected directly to the Open WebUI as the main LLM backend.

Starting

Ollama is one of the default services, so you don't need to specify anything special to start it.

harbor up

See harbor defaults on managing default services.

Models

You can discover new models via Ollama's model library.

Management of the models is possible right from the Open WebUI Admin Settings. The models are stored in the global ollama cache on your local machine.

Alternatively, you can use ollama CLI itself.

# Show the list of available models
harbor ollama list

# Pull a new model
harbor ollama pull phi3

More generally, you can use a full ollama CLI, when the corresponding service is running.

# Ollama service should be running to access the cli
harbor ollama --help
# See the envrionment variables
# supported by ollama service
harbor ollama serve --help

# Access Ollama CLI commands
harbor ollama version

Configuration

You can specify Ollama's environment variables (run harbor ollama serve --help for reference) in the .env and docker-compose.ollama.yml files.

# Configure ollama version, accepts a docker tag
harbor config set ollama.version 0.3.7-rc5-rocm

API

Retreive the endpoint for ollama service with:

harbor url ollama

Additionally, you can find a small HTTP playbook in the http-catalog folder.