Skip to content

Commit

Permalink
Added 'frigate event summary' blueprint
Browse files Browse the repository at this point in the history
  • Loading branch information
valentinfrlch committed Oct 18, 2024
1 parent dd06b59 commit 5d81fe5
Show file tree
Hide file tree
Showing 2 changed files with 153 additions and 14 deletions.
30 changes: 16 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,26 @@
<a href="#roadmap">🚧 Roadmap</a>
·
<a href="#how-to-report-a-bug-or-request-a-feature">🪲 How to report Bugs</a>
·
<a href="#support">☕ Support</a>
</p>


<br>
<br>
<br>

**LLM Vision** is a Home Assistant integration to analyze images, videos and camera feeds using the vision capabilities of multimodal LLMs.
Supported providers are OpenAI, Anthropic, Google Gemini, [LocalAI](https://github.com/mudler/LocalAI), [Ollama](https://ollama.com/) and any OpenAI compatible API.
<p align="center">
<strong>LLM Vision</strong> is a Home Assistant integration to analyze images, videos,
live camera feeds and frigate events using the vision capabilities of multimodal LLMs.
Supported providers are OpenAI, Anthropic, Google Gemini, Groq,
<a href="https://github.com/mudler/LocalAI">LocalAI</a>,
<a href="https://ollama.com/">Ollama</a> and any OpenAI compatible API.
</p>

## Features
- Compatible with OpenAI, Anthropic Claude, Google Gemini, [LocalAI](https://github.com/mudler/LocalAI), [Ollama](https://ollama.com/) and custom OpenAI compatible APIs
- Takes images and video from camera entities as input
- Takes local image and video files as input
- Compatible with OpenAI, Anthropic Claude, Google Gemini, Groq, [LocalAI](https://github.com/mudler/LocalAI), [Ollama](https://ollama.com/) and custom OpenAI compatible APIs
- Analyzes images and video files
- Captures and analyzes live camera feeds
- Analyzes frigate events
- Images can be downscaled for faster processing

## Resources
Expand Down Expand Up @@ -67,13 +73,9 @@ logger:
> [!NOTE]
> These are planned features and ideas. They are subject to change and may not be implemented in the order listed or at all.
1. **New Provider**: NVIDIA ChatRTX
2. **HACS**: Include in HACS default
3. [x] ~~**Animation Support**: Support for animated GIFs~~
4. [x] ~~**New Provider**: Custom (OpenAI API compatible) Providers~~
5. [x] ~~**Feature**: HTTPS support for LocalAI and Ollama~~
6. [x] ~~**Feature**: Support for video files~~
7. [x] ~~**Feature**: Analyze Frigate Recordings using frigate's `event_id`~~
1. **HACS**: Include in HACS default
For features added in previous versions, check the changelogs in the release notes.
## How to report a bug or request a feature
Expand Down
137 changes: 137 additions & 0 deletions blueprints/frigate_analyzer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
blueprint:
name: Frigate Event Summary
description: Summarizes frigate events and sends a notification with a preview to your phone
domain: automation
input:
notify_device:
name: Notify Device
description: The device to send the notification to
selector:
device:
integration: mobile_app
frigate_url:
name: Frigate URL
description: Frigate's base url to fetch preview (e.g. http://localhost:5000)
selector:
text:
multiline: false
cooldown:
name: Cooldown
description: Time in minutes to wait before running again
default: 10
selector:
number:
min: 0
max: 60
provider:
name: Provider
description: Configuration to use for the video_analyzer service. See docs for additional information.
selector:
config_entry:
integration: llmvision
model:
name: Model
description: Model to use for the video_analyzer service
default: "gpt-4o-mini"
selector:
text:
multiline: false
message:
name: Prompt
description: Model prompt for the video_analyzer service
default: "Summarize briefly what's happening in the camera feed (one sentence max). Don't describe the scene. If there is a person, describe what they're doing."
selector:
text:
multiline: true
interval:
name: Interval
description: Analyze frames every <interval> seconds
default: 3
selector:
number:
min: 1
max: 60
target_width:
name: Target Width
description: Width in pixels to downscale (uses less tokens)
default: 1280
selector:
number:
min: 512
max: 1920
detail:
name: Detail
description: Detail parameter (OpenAI only)
default: 'high'
selector:
select:
options:
- 'high'
- 'low'
max_tokens:
name: Maximum Tokens
description: Maximum number of tokens to generate
default: 50
selector:
number:
min: 1
max: 100
temperature:
name: Temperature
description: Randomness. Lower is more accurate, higher is more creative
default: 0.1
selector:
number:
min: 0.1
max: 1.0
step: 0.1

variables:
base_url: !input frigate_url
cooldown: !input cooldown

trigger:
platform: mqtt
topic: "frigate/events"

condition:
- condition: template
value_template: '{{ trigger.payload_json["type"] == "end" }}'
- condition: template
value_template: "{{ (now() - state_attr(this.entity_id, 'last_triggered')).total_seconds() / 60 > cooldown }}"

action:
- service: llmvision.video_analyzer
data:
event_id: '{{trigger.payload_json["after"]["id"]}}'
provider: !input provider
model: !input model
message: !input message
interval: !input interval
include_filename: false
target_width: !input target_width
detail: !input detail
max_tokens: !input max_tokens
temperature: !input temperature
response_variable: response

- choose:
- conditions:
- condition: template
value_template: "{{ base_url is not none and base_url != '' }}"
sequence:
- alias: "Send notification with preview"
domain: mobile_app
type: notify
device_id: !input notify_device
title: "{{ trigger.payload_json.after.label|capitalize }} seen"
message: "{{ response.response_text }}"
data:
video: "{{base_url}}/api/events/{{trigger.payload_json['after']['id']}}/clip.mp4"
default:
- alias: "Send notification"
domain: mobile_app
type: notify
device_id: !input notify_device
title: "{{ trigger.payload_json.after.label|capitalize }} seen"
message: "{{ response.response_text }}"

0 comments on commit 5d81fe5

Please sign in to comment.