Elia + LocalAI + Kubernetes for chatting to a self hosted LLM

[richiejp]

Hi @darrenburns , great project! I have been looking for ways to quickly test LLMs I have deployed on Kubernetes and was told about Elia on the LocalAI Discord. I have created an Elia container and K8s config which automatically configures Elia to connect to a model deployed on LocalAI (OpenAI API compatible). This is a bit hacky because Elia doesn't allow arbitrary models to be specified, but works.

https://github.com/premAI-io/prem-operator/blob/main/docs/guides/elia.md

[darrenburns]

Hey - thanks for this. Sorry I didn't reply earlier, it slipped by me.

I just recently released 1.0 which greatly changes how things work - not sure if that's going to make things harder or easier for you. Elia does now support local LLMs, but it offers that support via ollama.

[richiejp]

No problem. It seems like Ollama has a different API from OpenAI which is what LocalAI targets. I guess ideally you would support this #35

When configuring a model if we can select which API to use, the model name and the API base then this would cover a lot of scenarios. Including hosting a model on vLLM, Huggingface TGI, KServe etc.

In fact you don't need to specify the model name, this can be fetched from the service using the v1/models endpoint. Although some services will return 1000s of models.

[darrenburns]

Elia uses LiteLLM internally which I believe is OpenAI compatible - I just advertsie Ollama because it seems the easiest to set up for most people.

It seems like if you set litellm.api_base to your local server URL, then specify the model, it might "just work" ™️ ?
https://docs.litellm.ai/docs/providers/openai#using-helicone-proxy-with-litellm

I don't have time to test this out at the moment, but it might be a simple update to Elia to add api_base to EliaChatModel, and then when the model changes, set litellm.api_base to the new value.

[darrenburns]

I've implemented this on a branch - would you like to test it out?

pipx install git+https://github.com/darrenburns/elia.git@more-control-over-models

Hopefully adding something like this to the config with that branch installed will be enough:

# example of a model running on a local server, e.g. LocalAI
[[models]]
name = "some-model"
api_base = "http://localhost:8080/v1/chat/completions"
api_key = "api-key-if-required"

[richiejp]

Great works perfectly with a config like

default_model="openai/phi-2-chat"
    
[[models]]
name="openai/phi-2-chat"
api_base="http://phi-2-chat:8080/v1"

The model name is prefixed with 'openai/' because LiteLLM uses it to decide which API to use, the actual LocalAI model name is just phi-2-chat (incase someone reading this is wondering).

[ristomatti]

Is it possible to use an Ollama endpoint through the api_base setting? I tried with http://host:11434, http://host:11434/api and http://host:11434/api/chat but always get "There was a problem using this model".