Elastic AI Assistant with Local LLM

• 1. Description
• 2. Requirements
  • 2.1. Ollama
  • 2.2. Kibana Playground Specific
  • 2.3. Elastic Stack

1. Description

Leveraging local LLMs within Elastic environments ensures robust data confidentiality, as sensitive information never leaves the secure premises. Additionally, local LLMs provide faster processing times and greater control over model customization, catering to specific business needs without reliance on external servers. This setup enhances compliance with privacy regulations and minimizes the risk of data breaches.

2. Requirements

In the following setup we will have the following scenario running a local developpement environement :

stateDiagram-v2
    User --> Kibana
    state Host {
        ollama --> lama3
        ollama --> mistral
        Kibana --> ollama
    }
    state Docker {
        Kibana --> elasticsearch
        Logstash --> elasticsearch
    }

Loading

You could adapt your architecture by running ollama directly in the docker-compose environment.

2.1. Ollama

1. Download & Install Ollama for your platform
2. Once started download the model you want to run.

ollama pull mistral
ollama pull llama3

3. Test the two models

For Llam3:

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama3",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'

For Mistral:

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "mistral",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'

2.2. Kibana Playground Specific

If you are planning to use Kibana Playground, you will need to fake the model to one of the following one:

1. gpt-3.5-turbo
2. gpt-4o
3. gpt-4-turbo

For this, we are going to use Ollama Modelfile file to import the model. You'll find Modelfile-Mistral example in the repository, you can adapt it per your need. Once done, you can execute the following command to expose the model as the name you've configured, ex. gpt-4o :

ollama create gpt-4o -f Modelfile-Mistral

NOTE: If your Ollama server is running on a separated server, you will have to change the listen port to listen on all interface. for example on a Mac :

    launchctl setenv OLLAMA_HOST "0.0.0.0"

and restart Ollama application.

For more information please refer to the Ollama documentation

2.3. Elastic Stack

1. Install Elastic stack

For the purpose of this setup we will use docker-elk as a base.

simply clone the repository and start the stack

git clone [email protected]:deviantony/docker-elk.git
docker compose up setup
docker compose up -d

2. Configure the stack

In order to be able to use Kibana connectors, you will need to generate encryption keys from kibana docker and add them to kibana.yml

docker exec -ti docker-elk-kibana-1 bash
kibana@1d5263e8d004:~$ bin/kibana-encryption-keys generate 
Settings:
xpack.encryptedSavedObjects.encryptionKey: <GENERATED_ENCRYPTEDSAVEDOBJECTS.ENCRYPTIONKEY_VALUE>
xpack.reporting.encryptionKey: <GENERATED_REPORTING.ENCRYPTIONKEY_VALUE>
xpack.security.encryptionKey: <GENERATED_SECURITY.ENCRYPTIONKEY_VALUE>

and add the values to kibana/config/kibana.yml of the docker compose file :

xpack.encryptedSavedObjects.encryptionKey: <GENERATED_ENCRYPTEDSAVEDOBJECTS.ENCRYPTIONKEY_VALUE>
xpack.reporting.encryptionKey: <GENERATED_REPORTING.ENCRYPTIONKEY_VALUE>
xpack.security.encryptionKey: <GENERATED_SECURITY.ENCRYPTIONKEY_VALUE>

Finally restart the stack

docker compose restart

For more information about please refer to the Elastic documentation

3. Create the connectors

Finally you can create the connectors within Kibana UI. For this go to Stack Management -> Connectors -> Create connector -> OpenAI

Connectors are handled by Kibana, which in our case, is executed instide a docker and need to interact with Ollama, executed on the host. For Kibana to reach out to Ollama, we will need to use the following host : host.docker.internal

NOTE: The connector is expecting to have an API Key configured to work. Ollama doesn't provide this feature so you can a random string and save the connector.

For Llama3 this will look like this :

For Mistral this will look like this :

NOTE: For more information on the host.docker.internal please refer to the docker documentation

You can now enjoy your local LLM instance within the Elastic AI Assistant