Image and video analyzer for Home Assistant using multimodal LLMs

🌟 Features · 📖 Resources · ⬇️ Installation · 🚧 Roadmap · 🪲 How to report Bugs · ☕ Support

LLM Vision is a Home Assistant integration to analyze images, videos, live camera feeds and frigate events using the vision capabilities of multimodal LLMs. Supported providers are OpenAI, Anthropic, Google Gemini, Groq, LocalAI, Ollama and any OpenAI compatible API.

Features

• Compatible with OpenAI, Anthropic Claude, Google Gemini, Groq, LocalAI, Ollama and custom OpenAI compatible APIs
• Analyzes images and video files, live camera feeds and Frigate events
• Remembers Frigate events and camera motion events so you can ask about them later
• Seamlessly updates sensors based on image input

Blueprint

With the easy to use blueprint, you'll get important notifications intelligently summarized by AI from either Frigate or cameras in Home Assistant. LLM Vision can also remember events, so you can ask about them later. LLM Vision needs to be installed to use the blueprint.

Learn how to install the blueprint

Resources

Check the docs for detailed instructions on how to set up LLM Vision and each of the supported providers, get inspiration from examples or join the discussion on the Home Assistant Community.

Installation

1. Search for LLM Vision in Home Assistant Settings/Devices & services
2. Select your provider
3. Follow the instructions to add your AI providers.

Detailed instruction on how to set up LLM Vision and each of the supported providers are available here: https://llm-vision.gitbook.io/getting-started/

Debugging

To enable debugging, add the following to your configuration.yaml:

logger:
  logs:
    custom_components.llmvision: debug

Roadmap

Note

These are planned features and ideas. They are subject to change and may not be implemented in the order listed or at all.

1. HACS: Include in HACS default repository
2. Providers: Support for Azure

For features added in previous versions, check the changelogs in the release notes.

How to report a bug or request a feature

Important

Bugs: If you encounter any bugs and have followed the instructions carefully, feel free to file a bug report.
Feature Requests: If you have an idea for a feature, create a feature request.

 
Create new Issue 
 

Support

You can support this project by starring this GitHub repository. If you want, you can also buy me a coffee here: