1. Harbor User Guide
Firstly, welcome to Harbor! This guide will help you get started with Harbor CLI and manage the main services. It assumes that you already have the Harbor CLI installed and running. See the Harbor CLI Installation guide for more information.
Start Harbor with default services:
harbor upThis will start Ollama and Open WebUI by default. These two are the pre-configured as the default Frontend and Backend respectively. Being a "default" service means that they will be started automatically when you run harbor up (and many other commands) without any additional arguments.
Additionally, these two services are the ones that are tested the most and targeted to be integrated with the rest of the Harbor in the most seamless way.
Tip
You can configure default services using harbor defaults
Here are some sample logs from a successful start:
user@os:~$ ▼ h u
[+] Running 3/3
✔ Network harbor_harbor-network Created 0.1s
✔ Container harbor.ollama Healthy 0.7s
✔ Container harbor.webui Healthy 5.7sh and u are both aliases. First is for the CLI itself (see harbor link --short), the second is for the up command. You can typically see available aliases in the CLI Help output as well as on the pages of CLI Reference.
When the harbor is running, you are ready to access its default UI. You can open it in your default browser using a built-in command:
harbor openharbor open uses harbor url under the hood. You can specify a service name to open a specific service:
# See the URL
harbor url <service handle>
harbor open <service handle>When you are running harbor up / harbor open / harbor url commands without specifying a service handle, it will default to the service that is configured as the default UI. You can adjust that using harbor config.
Now, let's add some more services to your Harbor setup. To start a new service, that is not a default one, you can use the harbor up command with the service handle as an argument:
harbor up searxngTip
Every service in Harbor will have its own dedicated "handle" that can be used with most compatible commands. You'll find all the handles in the Services section. For example, Open WebUI has the handle webui.
Harbor allows you to specify the set of services that will be started by default. You can manage the default services using the harbor defaults command. It includes frontends, backends and satellite services that may be included in your Harbor setup.
# Show the list of default services
harbor defaults
# Add a new default service
harbor defaults add searxng
# Swap a default backend
harbor defaults rm ollama
harbor defaults add vllmWhen running Harbor, most of the value comes from running everything at once with a few simple commands. This includes running LLMs, which are the main focus of Harbor. In order to run them, you'll want to run one of the supported backend services. You will find detailed guides on configuration and usage of the services in their own respective documentation pages, but here are some quick examples for the most common backends:
ollama is the default LLM backend due to its convenience for the end-user, it's main benefit is that it determines system resources and features very well automatically.
Harbor not only runs ollama as a service, but also gives you full access to its CLI via dockerized harbor ollama command.
List available models:
harbor ollama listPull a new model:
harbor ollama pull <model_name>Tip
You can run any ollama commands via harbor ollama ...
Llama.cpp own cache:
harbor llamacpp model <full hugging face gguf URL>
# Example
harbor llamacpp model https://huggingface.co/cognitivecomputations/dolphin-2.9.4-llama3.1-8b-gguf/blob/main/dolphin-2.9.4-llama3.1-8b-Q4_K_S.ggufThe model will be downloaded and cached on the next harbor up with llamacpp service.
Tip
You can quickly jump to HuggingFace model lookup with harbor hf find command:
harbor hf find dolphin ggufUse shared HuggingFace hub cache:
# Download
harbor hf download <user/repo> <file.gguf>
# Locate the file
harbor find file.gguf
# Set the path to the model
harbor config set llamacpp.model.specifier -m /app/models/<path to file.gguf>Set model:
harbor vllm model <user/repo>
# Example
harbor vllm model google/gemma-2-2b-itTip
You can set HuggingFace Hub token to access gated/private models:
harbor hf token <your HF token>This token will be pre-configured for services that might need it for such purpose.
Please refer to the Backends section in the Services documentation for even more backends and their configuration.
There are three layers of configuration in Harbor:
For the most common/frequent configs, Harbor provides CLI aliases, one such example is configuration of service models:
# Setting service model via CLI alias
harbor llamacpp model https://huggingface.co/lm-kit/gemma-2-2b-gguf/blob/main/gemma-2-2B-Q8_0.gguf
harbor tgi model google/gemma-2-2b-it
harbor vllm model google/gemma-2-2b-it
harbor aphrodite model google/gemma-2-2b-it
harbor tabbyapi model google/gemma-2-2b-it-exl2
harbor mistralrs model google/gemma-2-2b-it
harbor opint model google/gemma-2-2b-it
harbor cmdh model google/gemma-2-2b-it
harbor fabric model google/gemma-2-2b-it
harbor parler model parler-tts/parler-tts-large-v1
harbor airllm model meta-llama/Meta-Llama-3.1-8B-Instruct
harbor txtai rag model llama3.1:8b-instruct-q6_K
harbor aider model llama3.1:8b-instruct-q6_K
harbor chatui model llama3.1:8b-instruct-q6_K
harbor aichat model llama3.1:8b-instruct-q6_KThese aliases are all linked to the harbor config under the hood. It means that you can use either the alias or the config option directly - it'll work the same way.
Aliases are typically setup for the "hot" configuration options, like setting the model, version, or other frequently used ones.
Note
As you can see above, services accept different values for the model specifier, depending on their internal configuration. Refer to Services docs for detailed instructions on how to configure each service.
Versions:
# Setting service version via CLI alias
harbor vllm version 0.5.3
harbor webui version main
harbor mistralrs version 0.3
harbor chatui version latest
harbor comfyui version latest-cudaTip
Most Service CLIs will also have dedicated help entries showing all available options:
harbor <service> --helpOne example of the option that doesn't typically have an alias are the service ports on the host. But you can still adjust them, see the section below!
Service CLIs are aliases for the underlying Harbor Config. It can be accessed directly via harbor config command:
# Usage help
harbor config --helpharbor config works with the .env file as a key/value store, allowing you to set and get configuration values for services.
# See all configuration options
harbor config ls
# WEBUI_NAME Harbor
# VLLM_VERSION v0.5.3
# ... Many more options
# Get a specific value, accepts various aliases
harbor config get VLLM_VERSION
harbor config get vllm_version
harbor config get vllm.version
# Set a value, accepts same aliases as above
harbor config set webui.name v0.5.3See a more detailed overview in the harbor config CLI reference
Most of the services can be configured via either .env or a specific yaml, json, toml configuration (refer to service docs for specifics). These files are all stored in the Harbor workspace on your machine and can be edited directly.
# Show the path to the Harbor workspace
harbor home
# Open workspace in the file manager
open $(harbor home)
# Shortcut for VS Code users
harbor vscodeTip
You can add arbitrary environment variables to the .env file in the workspace and they will be available to all services. Alternatively, you can lookup if there's an override.env in the service folder and use that for variables that'll only be visible to that service.
# Open the .env file with default editor
open $(harbor home)/.envYou can find all the external tracked configuration folders by listing related configurations:
harbor config ls | grep CONFIG_PATHCertain services will also have a dedicated cache folder, which can be found in the same way:
harbor config ls | grep CACHEWhen you have a specific config key, you can open it in the file manager using:
open $(eval echo "$(harbor config get hf.cache)")When you have multiple configurations for different use-cases, you can save them as profiles for easy switching. Profiles include everything that can be configured via harbor config (most of the settings configured via CLI) and are stored in the Harbor workspace. They are just a way to swap between .env files using command line.
# Use a profile
$ ▼ h profile use default
21:49:20 [INFO] Profile 'default' loaded.
# Check settings specific to the vLLM service
$ ▼ h config ls | grep VLLM
VLLM_CACHE ~/.cache/vllm
VLLM_HOST_PORT 33911
VLLM_VERSION v0.6.0
VLLM_MODEL microsoft/Phi-3.5-mini-instruct
VLLM_EXTRA_ARGS
VLLM_ATTENTION_BACKEND FLASH_ATTN
VLLM_MODEL_SPECIFIER --model microsoft/Phi-3.5-mini-instruct
# Switch to another profile
$ ▼ h profile use phimoe
21:49:42 [INFO] Profile 'phimoe' loaded.
# vLLM settings are different now
$ ▼ h config ls | grep VLLM
VLLM_CACHE ~/.cache/vllm
VLLM_HOST_PORT 33911
VLLM_VERSION v0.6.1.post2
VLLM_MODEL microsoft/Phi-3.5-MoE-instruct
VLLM_EXTRA_ARGS --max-model-len 1024 --trust-remote-code --cpu-offload-gb 56 --enforce-eager --gpu-memory-utilization 0 --device cpu
VLLM_ATTENTION_BACKEND FLASH_ATTN
VLLM_MODEL_SPECIFIER --model microsoft/Phi-3.5-MoE-instructThere are a few considerations when using profiles:
- When the profile is loaded, modifications are not saved by default and will be lost when switching to another profile (or reloading the current one). Use
harbor profile save <name>to persist the changes after making them - Profiles are stored in the Harbor workspace and can be shared between different Harbor instances
- Profiles are not versioned and are not guaranteed to work between different Harbor versions
- You can also edit profiles as
.envfiles in the workspace, it's not necessary to use the CLI
Harbor comes with a variety of satellite services intended to be used with LLMs. These services can be started, stopped and configured in the same way as the Frontends and Backends.
SearXNG is a great example of a satellite service that is exceptionally useful for LLMs and on its own.
harbor up searxngOpen WebUI (and many other built-in services) will automatically use it for Web RAG functionality.
Start TTS service:
harbor up ttsConfigure voices in tts/config/voice_to_speaker.yaml
Please refer to the Satellites section in the Services documentation for even more additional services and their configuration in Harbor.
Get URL for any service:
harbor url <service handle>See the harbor url CLI Reference for more information.
# Show logs for all running services
harbor logs
# Show logs for a specific service
harbor logs <service handle>
# Start tailing logs immediately
# after the service is started
harbor up <service handle> --tailSee the harbor logs CLI Reference for more information.
Stop all services:
harbor down- Services - Detailed documentation on services available in Harbor
- Harbor CLI Reference - Detailed documentation for the Harbor CLI