Commit Graph

70 Commits

Author SHA1 Message Date
Nicolas Mowen
13e90fc6e0 Face recognition backend (#14495)
* Add basic config and face recognition table

* Reconfigure updates processing to handle face

* Crop frame to face box

* Implement face embedding calculation

* Get matching face embeddings

* Add support face recognition based on existing faces

* Use arcface face embeddings instead of generic embeddings model

* Add apis for managing faces

* Implement face uploading API

* Build out more APIs

* Add min area config

* Handle larger images

* Add more debug logs

* fix calculation

* Reduce timeout

* Small tweaks

* Use webp images

* Use facenet model
2024-11-24 08:33:08 -07:00
Nicolas Mowen
66277fbb6c
Fix embeddings (#15072)
* Fix embeddings reading frames

* Fix event update reading

* Formatting

* Pin AIO http to fix build failure

* Pin starlette
2024-11-19 12:20:04 -06:00
Josh Hawkins
9ae839ad72
Tracked object metadata changes (#15055)
* add enum and change topic name

* frontend renaming

* docs

* only display sublabel score if it it exists

* remove debug print
2024-11-18 11:26:44 -07:00
Evan Jarrett
e4a6b29279
fix string comparison on mqtt error message for Server unavailable (#14675) 2024-10-30 05:05:58 -06:00
Nicolas Mowen
eca504cb07
More bug fixes (#14593)
* Adjust mqtt logging behavior

* Set disconnect

* Only consider intel gpu stats error if None is returned
2024-10-25 09:45:11 -05:00
Nicolas Mowen
94f7528885
Bug fixes (#14588)
* Get intel stats manually if parsing fails

* Fix assignment

* Clean up mqtt

* Formatting

* Fix logic
2024-10-25 06:47:56 -06:00
Josh Hawkins
4ca267ea17
Search UI tweaks and bugfixes (#14328)
* Publish model state and embeddings reindex in dispatcher onConnect

* remove unneeded from explore

* add embeddings reindex progress to statusbar

* don't allow right click or show similar button if semantic search is disabled

* fix status bar
2024-10-13 19:36:49 -06:00
Nicolas Mowen
6df541e1fd
Openvino models (#14283)
* Enable model conversion cache for openvino

* Use openvino directly for onnx embeddings if available

* Don't fail if zmq is busy
2024-10-11 10:47:23 -06:00
Josh Hawkins
ee8091ba91
Correctly handle camera command in dispatcher (#14273) 2024-10-10 18:48:56 -06:00
Nicolas Mowen
dd6276e706
Embeddings fixes (#14269)
* Add debugging logs for more info

* Improve timeout handling

* Fix event cleanup

* Handle zmq error and empty data

* Don't run download

* Remove unneeded embeddings creations

* Update timouts

* Init models immediately

* Fix order of init

* Cleanup
2024-10-10 16:37:43 -05:00
Josh Hawkins
f67ec241d4
Add embeddings reindex progress to the UI (#14268)
* refactor dispatcher

* add reindex to dictionary

* add circular progress bar component

* Add progress to UI when embeddings are reindexing

* readd comments to dispatcher for clarity

* Only report progress every 10 events so we don't spam the logs and websocket

* clean up
2024-10-10 13:28:43 -06:00
Nicolas Mowen
8ade85edec
Restructure embeddings (#14266)
* Restructure embeddings

* Use ZMQ to proxy embeddings requests

* Handle serialization

* Formatting

* Remove unused
2024-10-10 09:42:24 -06:00
Josh Hawkins
24ac9f3e5a
Use sqlite-vec extension instead of chromadb for embeddings (#14163)
* swap sqlite_vec for chroma in requirements

* load sqlite_vec in embeddings manager

* remove chroma and revamp Embeddings class for sqlite_vec

* manual minilm onnx inference

* remove chroma in clip model

* migrate api from chroma to sqlite_vec

* migrate event cleanup from chroma to sqlite_vec

* migrate embedding maintainer from chroma to sqlite_vec

* genai description for sqlite_vec

* load sqlite_vec in main thread db

* extend the SqliteQueueDatabase class and use peewee db.execute_sql

* search with Event type for similarity

* fix similarity search

* install and add comment about transformers

* fix normalization

* add id filter

* clean up

* clean up

* fully remove chroma and add transformers env var

* readd uvicorn for fastapi

* readd tokenizer parallelism env var

* remove chroma from docs

* remove chroma from UI

* try removing custom pysqlite3 build

* hard code limit

* optimize queries

* revert explore query

* fix query

* keep building pysqlite3

* single pass fetch and process

* remove unnecessary re-embed

* update deps

* move SqliteVecQueueDatabase to db directory

* make search thumbnail take up full size of results box

* improve typing

* improve model downloading and add status screen

* daemon downloading thread

* catch case when semantic search is disabled

* fix typing

* build sqlite_vec from source

* resolve conflict

* file permissions

* try build deps

* remove sources

* sources

* fix thread start

* include git in build

* reorder embeddings after detectors are started

* build with sqlite amalgamation

* non-platform specific

* use wget instead of curl

* remove unzip -d

* remove sqlite_vec from requirements and load the compiled version

* fix build

* avoid race in db connection

* add scale_factor and bias to description zscore normalization
2024-10-07 14:30:45 -06:00
gtsiam
54900ae318
Properly call super() in subclasses (#14124) 2024-10-02 19:35:46 -06:00
Josh Hawkins
95d6da3111
Add ability to configure genai to use snapshot instead of thumbnails (#14077)
* Allow embedding of snapshot for description via config option

* docs

* frontend button

* Backend

* crop snapshot to region

* only show dropdown when event has snapshot

* fix cursor on dropdown

* crop on initial generation as well

* use enum for type

* fix type
2024-09-30 15:54:53 -06:00
gtsiam
c0bd3b362c
Custom classes for Process and Metrics (#13950)
* Subclass Process for audio_process

* Introduce custom mp.Process subclass

In preparation to switch the multiprocessing startup method away from
"fork", we cannot rely on os.fork cloning the log state at fork time.
Instead, we have to set up logging before we run the business logic of
each process.

* Make camera_metrics into a class

* Make ptz_metrics into a class

* Fixed PtzMotionEstimator.ptz_metrics type annotation

* Removed pointless variables

* Do not start audio processor when no audio cameras are configured
2024-09-27 07:53:23 -05:00
Nicolas Mowen
35a4460334
Tweaks (#13983)
* Add onConnect to mqtt and doucment

* Add play pause via mouse click
2024-09-26 09:12:15 -05:00
Josh Hawkins
ecbf0410eb
Regenerate genai tracked object descriptions (#13930)
* add genai to frigateconfig

* add regenerate button if genai is enabled

* add endpoint and new zmq pub/sub model

* move publisher to app

* dont override

* logging

* debug timeouts

* clean up

* clean up

* allow saving of empty description

* ensure descriptions can be empty

* update search detail when results change

* revalidate explore page on focus

* global mutate hook

* description websocket hook and dispatcher

* revalidation and mutation

* fix merge conflicts

* update tests

* fix merge conflicts

* fix response message

* fix response message

* fix fastapi

* fix test

* remove log

* json content

* fix content response

* more json content fixes

* another one
2024-09-24 08:14:51 -06:00
gtsiam
edababa88e
Work through most of the cspell warnings in python (#13794) 2024-09-17 10:41:46 -05:00
Nicolas Mowen
07d1692f2b
Make notifications toggleable via MQTT (#13657)
* Add ability to toggle mqtt state from MQTT / ws

* Listen to notification config updates

* Add docs for notifications
2024-09-10 11:24:44 -06:00
Nicolas Mowen
01fa1777ac Fix ZMQ race condition with events (#13198) 2024-08-29 20:19:50 -06:00
Nicolas Mowen
04fd05bc7d Notification action (#12742) 2024-08-29 20:19:50 -06:00
Nicolas Mowen
690ee3dc15 Implement support for notifications (#12523)
* Setup basic notification page

* Add basic notification implementation

* Register for push notifications

* Implement dispatching

* Add fields

* Handle image and link

* Add notification config

* Add field for users notification tokens

* Implement saving of notification tokens

* Implement VAPID key generation

* Implement public key encoding

* Implement webpush from server

* Implement push notification handling

* Make notifications config only

* Add maskable icon

* Use zod form to control notification settings in the UI

* Use js

* Always open notification

* Support multiple endpoints

* Handle cleaning up expired notification registrations

* Correctly unsubscribe notifications

* Change ttl dynamically

* Add note about notification latency and features

* Cleanup docs

* Fix firefox pushes

* Add links to docs and improve formatting

* Improve wording

* Fix docstring

Co-authored-by: Blake Blackshear <blake@frigate.video>

* Handle case where native auth is not enabled

* Show errors in UI

---------

Co-authored-by: Blake Blackshear <blake@frigate.video>
2024-08-29 20:19:50 -06:00
Jason Hunter
36cbffcc5e Semantic Search for Detections (#11899)
* Initial re-implementation of semantic search

* put docker-compose back and make reindex match docs

* remove debug code and fix import

* fix docs

* manually build pysqlite3 as binaries are only available for x86-64

* update comment in build_pysqlite3.sh

* only embed objects

* better error handling when genai fails

* ask ollama to pull requested model at startup

* update ollama docs

* address some PR review comments

* fix lint

* use IPC to write description, update docs for reindex

* remove gemini-pro-vision from docs as it will be unavailable soon

* fix OpenAI doc available models

* fix api error in gemini and metadata for embeddings
2024-08-29 20:19:50 -06:00
Nicolas Mowen
8212b66ee0 Use camera status to get state of camera config (#12787)
* Use camera status to get state of camera config

* Fix spelling
2024-08-08 07:54:13 -06:00
Nicolas Mowen
1bd3285679
Use pickle for config objects (#12594) 2024-07-24 10:37:29 -05:00
Nicolas Mowen
d28ad0f0c8
Use JSON instead of pickle for serialization (#12590) 2024-07-24 08:58:23 -06:00
Jason Hunter
7917bf55ff
Fix unclean shutdown of ZMQ (#11740) 2024-06-04 06:39:34 -06:00
Nicolas Mowen
90bdb07463
Simplify on connect (#11176) 2024-04-30 07:27:39 -06:00
Nicolas Mowen
461442b399
Save initial camera state to update when websocket connects (#11174)
* Send camera state to dispatcher

* Fix logic

* Cleanup

* Send camera activitiy in on connect

* Support reading initial camera state

* Fix key

* Formatting

* Sorting
2024-04-30 07:09:50 -06:00
Nicolas Mowen
c035241b39
Clear ongoing reviews on startup (#11097)
* Add topic to clear ongoing review segments

* Clear ongoing review items on startup
2024-04-24 08:44:28 -05:00
Nicolas Mowen
8163c036ef
Use zmq for event end queue and update python deps (#10886)
* Use zmq for events ended

* Cleanup

* Update deps

* formatting
2024-04-08 18:19:45 -05:00
Nicolas Mowen
d1082ec305
Support manual detections in review items (#10784)
* Only update frame time if it is older

* Support manual detections as review items

* Don't handle api detections in recordings

* Store recordings for manual events
2024-04-02 07:07:50 -06:00
Josh Hawkins
3a9607e59b
Add relative movement by clicking on camera image for supported ptzs (#10629) 2024-03-23 10:53:33 -06:00
Nicolas Mowen
76a114a3cd
Rewrite events communication to use zmq instead of mp.Queue (#10627)
* Move to using zmq for events updating

* Use event updater in manual events handler

* Formatting
2024-03-23 11:11:32 -05:00
Nicolas Mowen
ea9316e770
Update and migrate mqtt to v2 (#10160) 2024-03-01 00:59:47 +00:00
Nicolas Mowen
940be5dc6f
Create ReviewSegment table in DB for organizing detections to be reviewed (#9918)
* Add review to database

* Create main manager for review segments

* Upsert and maintain review segments

* Update logic for adding new segments

* Add api

* Support deleting review segments on recording cleanup

* Add field for alert labels

* Formatting

* Logic fixes

* Save 16:9 thumbnail for review segment

* Ensure that crop is 16:9

* Fix non detected objects being added

* Only include true positives

* Add sub labels to data
2024-02-20 16:26:09 -07:00
Nicolas Mowen
dba21b606d
Convert config updating and video/audio queues to use zmq (#9893)
* Add config pub / sub pattern

* remove recording from feature metrics

* remove audio and feature metrics

* Check for updates from all cameras

* remove birdseye from camera metrics

* remove motion and detection camera metrics

* Ensure that all processes are stopped

* Stop communicators

* Detections

* Cleanup video output queue

* Use select for time sensitive polls

* Use ipc instead of tcp
2024-02-19 07:26:59 -06:00
Nicolas Mowen
dd3dc7949a
Use zmq for inter process communication (#9309)
* Use zmq for inter process communication

* Use localhost for reply and request

* Use pyobj instead of json and Need to use separate requestors for each audio listener

* Cleanup port defining
2024-02-14 18:24:36 -06:00
Blake Blackshear
198dbbdff1 Merge remote-tracking branch 'origin/master' into dev 2024-02-14 18:20:55 -06:00
Nicolas Mowen
00804a0f81
Don't fail if message is received before websocket start (#9634) 2024-02-05 17:53:16 -06:00
Nicolas Mowen
cfda531f5a Write a low resolution low fps stream from decoded frames (#8673)
* Generate low res low fps previews for recordings viewer

* Make sure previews end on the hour

* Fix durations and decrase keyframe interval to ensure smooth scrubbing

* Ensure minimized resolution is compatible with yuv

* Add ability to configure preview quality

* Fix

* Clean up previews more efficiently

* Use iterator
2024-01-31 12:56:11 +00:00
Josh Hawkins
7b520e8a9d
use enabled_in_config for switch in HA integration (#8630) 2023-11-16 01:25:48 +00:00
Nicolas Mowen
2b2c831253
Set the min region size as the model size (#8486) 2023-11-06 06:45:37 -06:00
Josh Hawkins
fc36be4f88
suppress error by overriding class func (#8431) 2023-11-02 23:24:14 +00:00
Shaun Berryman
36c1e00a6b
MQTT: Birdseye enabled/disabled and mode change support (#8291)
* support enabled and mode change for birdseye via mqtt

* resolve feedback from PR review
https://github.com/blakeblackshear/frigate/pull/8291#discussion_r1370083613

* change birdseye mode topic to set

* type in the docs

* these commented out lines should have never been in here
2023-10-26 06:20:55 -05:00
Josh Hawkins
9df5927ac5
Autotracking bugfixes and zooming updates (#8103)
* zoom in/out in search for lost objects

* predicted box should not be empty

* clean up and update zoom logic

* only zoom if enabled

* more cleanup

* check for valid velocity when zooming

* only try absolute zoom in if obj area has changed

* zoom logic

* don't enqueue lost object zoom if already at limit

* don't disable motion boxes during ptz moves

* velocity threshold based on move coefficients

* fix area zoom logic

* disable debug zoom

* don't process objects if ptz moving

* recalc with exponent

* change exponent

* remove lost object zooming

* increase distance threshold for stationary object

* increase distance threshold constant

* only zoom out if nonzero

* camera name in all debug logging

* add camera name to debug logging

* camera variable name consistency

* update calibration behavior and docs

* docs and better zooming

* more sensible target values

* docs wording

* fix velocity threshold variable

* zooming tweaks and remove iou for current objects

* debug and docs

* get valid velocity

* include zero

* additional debug statements

* add zoom hysteresis

* zoom on initial move if relative

* only update target box if we actually zoom

* merge dev

* use getattr instead of get

* increase distance threshold

* reverse logic

* get_camera_status after preset move to store zoom

* final tweaks and docs

* use constants and catch possible debug exception

* adjust zoom factor exponent

* don't run motion estimation when calling preset

* adjust dimension threshold

* use numpy for velocity estimate calcs

* more numpy conversion

* fix numpy shapes

* numpy zeros dimension

* more zoom out conditions

* fix velocity bug

* ensure init has been called in debug view

* ensure onvif init if enabling by mqtt

* change default hysteresis values

* recalc relative zoom value

* zoom out value

* try to zoom when object isn't moving

* try zoom when tracked object is not moving

* don't try to zoom every time

* negate zoom out condition when needed

* hysteresis constants for absolute zooming

* update zoom conditions

* don't recalc target box on zoom only

* zoom out if above area threshold

* don't print zooming debug for stationary obj

* revamp zooming to use area moving average

* zooming tweaks and expose property

* limit zoom with max target box

* use calibration to determine zoom levels

* zoom logic fix

* docs

* add tapo c200 camera

* fix initial absolute zoom

* small zoom logic fix

* better invalid velocity checks

* fix test

* really fix test this time
2023-10-22 12:59:13 -04:00
Nicolas Mowen
cb3990a0ac
Catch ws reset error (#8266)
* Catch ws reset error

* Formatting
2023-10-22 06:23:31 -04:00
Nicolas Mowen
91f7d67c5e
Smarter Regions (#8194)
* Smarter Regions

* Formatting

* Cleanup

* Fix motion region checking logic

* Add database table and migration for regions

* Update region grid on startup

* Revert init delay change

* Fix mypy

* Move object related functions to util

* Remove unused

* Fix tests

* Remove log

* Update the region daily at 2

* Fix logic

* Formatting

* Initialize grid before starting processing frames

* Move back to creating grid in main process

* Formatting

* Fixes

* Formating

* Fix region check

* Accept all but true

* Use regions grid for startup scan

* Add clarifying comment

* Fix new grid requests

* Add tests

* Delete stale region grids from DB
2023-10-18 18:21:52 -05:00
Nicolas Mowen
bd0d16a75b
Set commands before subscribing to messages (#7550) 2023-09-01 07:06:59 -05:00