Commit Graph

45 Commits

Author SHA1 Message Date
Nicolas Mowen
66277fbb6c
Fix embeddings (#15072)
* Fix embeddings reading frames

* Fix event update reading

* Formatting

* Pin AIO http to fix build failure

* Pin starlette
2024-11-19 12:20:04 -06:00
Josh Hawkins
9ae839ad72
Tracked object metadata changes (#15055)
* add enum and change topic name

* frontend renaming

* docs

* only display sublabel score if it it exists

* remove debug print
2024-11-18 11:26:44 -07:00
Josh Hawkins
143bab87f1
Genai bugfix (#14880)
* Fix genai init when disabled at global level

* use genai config for class init
2024-11-09 06:48:53 -07:00
Nicolas Mowen
40c6fda19d
Various fixes and improvements (#14492)
* Refactor preprocessing of images

* Cleanup preprocessing

* Improve naming and handling of embeddings

* Handle invalid intel json

* remove unused

* Use enum for model types

* Formatting
2024-10-21 16:19:34 -06:00
Josh Hawkins
6c70e56059
Misc bugfixes and improvements (#14460)
* only save a fixed number of thumbnails if genai is enabled

* disable cpu_mem_arena to save on memory until its actually needed

* fix search settings pane so it actually saves to the config
2024-10-20 14:14:51 -06:00
Nicolas Mowen
3f1ab66899
Embeddings UI updates (#14378)
* Handle Frigate+ submitted case

* Add search settings and rename general to ui settings

* Add platform aware sheet component

* use two columns on mobile view

* Add cameras page to more filters

* clean up search settings view

* Add time range to side filter

* better match with ui settings

* fix icon size

* use two columns on mobile view

* clean up search settings view

* Add zones and saving logic

* Add all filters to side panel

* better match with ui settings

* fix icon size

* Fix mobile fitler page

* Fix embeddings access

* Cleanup

* Fix scroll

* fix double scrollbars and add separators on mobile too

* two columns on mobile

* italics for emphasis

---------

Co-authored-by: Josh Hawkins <32435876+hawkeye217@users.noreply.github.com>
2024-10-15 19:25:59 -05:00
Nicolas Mowen
b75efcbca2
UI tweaks (#14369)
* Adjust text size

* Make cursor consistent

* Fix lint
2024-10-15 09:37:04 -06:00
Nicolas Mowen
25043278ab
Always run embedding descs one by one (#14365) 2024-10-15 07:40:45 -06:00
Nicolas Mowen
0eccb6a610
Db fixes (#14364)
* Handle case where embeddings overflow token limit

* Set notification tokens

* Fix sort
2024-10-15 07:17:54 -06:00
Josh Hawkins
1ec459ea3a
Batch embeddings fixes (#14325)
* fixes

* more readable loops

* more robust key check and warning message

* ensure we get reindex progress on mount

* use correct var for length
2024-10-13 15:25:13 -06:00
Nicolas Mowen
e8b2fde753
Support batch embeddings when reindexing (#14320)
* Refactor onnx embeddings to handle multiple inputs by default

* Process items in batches when reindexing
2024-10-13 12:33:27 -05:00
Josh Hawkins
0fc7999780
Improve reindex completion flag (#14308) 2024-10-12 14:44:01 -05:00
Nicolas Mowen
3a403392e7
Fixes for model downloading (#14305)
* Use different requestor for downloaders

* Handle case where lock is left over from failed partial download

* close requestor

* Formatting
2024-10-12 13:36:10 -05:00
Josh Hawkins
48c60621b6
Fix substitution on genai prompts (#14298) 2024-10-12 06:19:24 -06:00
Nicolas Mowen
6e332bbdf8
Remove device config and use model size to configure device used (#14290)
* Remove device config and use model size to configure device used

* Don't show Frigate+ submission when in progress

* Add docs link for bounding box colors
2024-10-11 17:08:14 -05:00
Josh Hawkins
8a8a0c7dec
Embeddings normalization fixes (#14284)
* Use cosine distance metric for vec tables

* Only apply normalization to multi modal searches

* Catch possible edge case in stddev calc

* Use sigmoid function for normalization for multi modal searches only

* Ensure we get model state on initial page load

* Only save stats for multi modal searches and only use cosine similarity for image -> image search
2024-10-11 13:11:11 -05:00
Nicolas Mowen
6df541e1fd
Openvino models (#14283)
* Enable model conversion cache for openvino

* Use openvino directly for onnx embeddings if available

* Don't fail if zmq is busy
2024-10-11 10:47:23 -06:00
Josh Hawkins
ae91fa6a39
Add time remaining to embedding reindex pane (#14279)
* Add function to convert seconds to human readable duration

* Add estimated time remaining to reindexing pane
2024-10-11 07:04:25 -06:00
Josh Hawkins
2897afce41
Reset saved search stats on reindex (#14280) 2024-10-11 06:59:29 -06:00
Josh Hawkins
54eb03d2a1
Add config option to select fp16 or quantized jina vision model (#14270)
* Add config option to select fp16 or quantized jina vision model

* requires_fp16 for text and large models only

* fix model type check

* fix cpu

* pass model size
2024-10-10 16:46:21 -06:00
Nicolas Mowen
dd6276e706
Embeddings fixes (#14269)
* Add debugging logs for more info

* Improve timeout handling

* Fix event cleanup

* Handle zmq error and empty data

* Don't run download

* Remove unneeded embeddings creations

* Update timouts

* Init models immediately

* Fix order of init

* Cleanup
2024-10-10 16:37:43 -05:00
Josh Hawkins
f67ec241d4
Add embeddings reindex progress to the UI (#14268)
* refactor dispatcher

* add reindex to dictionary

* add circular progress bar component

* Add progress to UI when embeddings are reindexing

* readd comments to dispatcher for clarity

* Only report progress every 10 events so we don't spam the logs and websocket

* clean up
2024-10-10 13:28:43 -06:00
Nicolas Mowen
8ade85edec
Restructure embeddings (#14266)
* Restructure embeddings

* Use ZMQ to proxy embeddings requests

* Handle serialization

* Formatting

* Remove unused
2024-10-10 09:42:24 -06:00
Nicolas Mowen
a2ca18a714
Bug fixes (#14263)
* Simplify loitering logic

* Fix divide by zero

* Add device config for semantic search

* Add docs
2024-10-10 07:09:12 -06:00
Nicolas Mowen
bc3a06178b
Embedding gpu (#14253) 2024-10-09 19:46:31 -06:00
Josh Hawkins
d4925622f9
Use JinaAI models for embeddings (#14252)
* add generic onnx model class and use jina ai clip models for all embeddings

* fix merge confligt

* add generic onnx model class and use jina ai clip models for all embeddings

* fix merge confligt

* preferred providers

* fix paths

* disable download progress bar

* remove logging of path

* drop and recreate tables on reindex

* use cache paths

* fix model name

* use trust remote code per transformers docs

* ensure tokenizer and feature extractor are correctly loaded

* revert

* manually download and cache feature extractor config

* remove unneeded

* remove old clip and minilm code

* docs update
2024-10-09 15:31:54 -06:00
Josh Hawkins
23ce1e930d
Adjust zscore bias for descriptions (#14205) 2024-10-07 18:51:37 -06:00
Josh Hawkins
24ac9f3e5a
Use sqlite-vec extension instead of chromadb for embeddings (#14163)
* swap sqlite_vec for chroma in requirements

* load sqlite_vec in embeddings manager

* remove chroma and revamp Embeddings class for sqlite_vec

* manual minilm onnx inference

* remove chroma in clip model

* migrate api from chroma to sqlite_vec

* migrate event cleanup from chroma to sqlite_vec

* migrate embedding maintainer from chroma to sqlite_vec

* genai description for sqlite_vec

* load sqlite_vec in main thread db

* extend the SqliteQueueDatabase class and use peewee db.execute_sql

* search with Event type for similarity

* fix similarity search

* install and add comment about transformers

* fix normalization

* add id filter

* clean up

* clean up

* fully remove chroma and add transformers env var

* readd uvicorn for fastapi

* readd tokenizer parallelism env var

* remove chroma from docs

* remove chroma from UI

* try removing custom pysqlite3 build

* hard code limit

* optimize queries

* revert explore query

* fix query

* keep building pysqlite3

* single pass fetch and process

* remove unnecessary re-embed

* update deps

* move SqliteVecQueueDatabase to db directory

* make search thumbnail take up full size of results box

* improve typing

* improve model downloading and add status screen

* daemon downloading thread

* catch case when semantic search is disabled

* fix typing

* build sqlite_vec from source

* resolve conflict

* file permissions

* try build deps

* remove sources

* sources

* fix thread start

* include git in build

* reorder embeddings after detectors are started

* build with sqlite amalgamation

* non-platform specific

* use wget instead of curl

* remove unzip -d

* remove sqlite_vec from requirements and load the compiled version

* fix build

* avoid race in db connection

* add scale_factor and bias to description zscore normalization
2024-10-07 14:30:45 -06:00
Josh Hawkins
fafe5623d1
UI tweaks (#14088)
* fix squashed alert thumbnails in filmstrip

* add genai debug logs

* consistent themed image loading indicator background color

* improve image loading skeleton in object lifecycle pane

* less rounding when screen is smaller

* use browser back button to dismiss review pane

* initial state
2024-10-01 07:01:45 -06:00
Josh Hawkins
95d6da3111
Add ability to configure genai to use snapshot instead of thumbnails (#14077)
* Allow embedding of snapshot for description via config option

* docs

* frontend button

* Backend

* crop snapshot to region

* only show dropdown when event has snapshot

* fix cursor on dropdown

* crop on initial generation as well

* use enum for type

* fix type
2024-09-30 15:54:53 -06:00
Josh Hawkins
40fe3b4358
Search functionality and UI tweaks (#13978)
* Portal tooltips

* Add ability to time_range filter chroma searches

* centering and padding consistency

* add event id back to chroma metadata

* query sqlite first and pass those ids to chroma for embeddings search

* ensure we pass timezone to the api call

* remove object lifecycle from search details for non-object events

* simplify hour calculation

* fix query without filters

* bump chroma version

* chroma 0.5.7

* fix selecting camera group in cameras filter button
2024-09-26 15:30:56 -05:00
Josh Hawkins
ae547d27e4
Use not instead of is None (#13962) 2024-09-25 11:53:25 -06:00
Josh Hawkins
4c12673fbb
Add ability to restrict genai to objects and zones at the camera level (#13958)
* Add ability to restrict genai to labels and zones at the camera level

* fix comment

* clarify docs

* use objects instead of labels

* docs

* object list
2024-09-25 09:42:39 -06:00
Josh Hawkins
a7f1f8d327
Add 1s timeout to metadata subscriber in embeddings maintainer (#13947) 2024-09-24 19:08:12 -05:00
Josh Hawkins
ecbf0410eb
Regenerate genai tracked object descriptions (#13930)
* add genai to frigateconfig

* add regenerate button if genai is enabled

* add endpoint and new zmq pub/sub model

* move publisher to app

* dont override

* logging

* debug timeouts

* clean up

* clean up

* allow saving of empty description

* ensure descriptions can be empty

* update search detail when results change

* revalidate explore page on focus

* global mutate hook

* description websocket hook and dispatcher

* revalidation and mutation

* fix merge conflicts

* update tests

* fix merge conflicts

* fix response message

* fix response message

* fix fastapi

* fix test

* remove log

* json content

* fix content response

* more json content fixes

* another one
2024-09-24 08:14:51 -06:00
Josh Hawkins
0f813962be
Chroma fixes (#13902)
* Ensure descriptions saved in chroma are non-empty

* delete only existing ids in event cleanup

* add debug logging
2024-09-23 06:53:19 -06:00
Nicolas Mowen
e44a9e8921
Use cpu provider for embeddings models (#13806) 2024-09-17 17:20:18 -05:00
Nicolas Mowen
1ed8642010
Refactor onnx providers (#13804)
* Ensure dirs exist for model caches

* Formatting

* Don't use tensorrt for embeddings
2024-09-17 15:54:44 -05:00
Nicolas Mowen
2362d0e838
Override onnx clip loading (#13800)
* Set caching options for hardware providers

* Always use CPU for searching

* Use new install strategy to remove onnxruntime and then install post wheels
2024-09-17 13:24:35 -06:00
gtsiam
edababa88e
Work through most of the cspell warnings in python (#13794) 2024-09-17 10:41:46 -05:00
Josh Hawkins
e3edcf057c
write prompts for genai at the camera level (#13767) 2024-09-16 08:46:11 -06:00
Nicolas Mowen
9afa1354da
Limit shm frame count (#12363)
* Limited shm frame count (#12346)

* Only keep 2x detect fps frames in SHM

* Don't delete previous shm frames in output

* Catch case where images do not exist

* Ensure files are closed

* Clear out all frames when shutting down

* Correct the number of frames saved

* Simplify empty shm error handling

* Improve frame safety

* Add handler logs when frame is None

* Don't fail on cleanup

* Cleanup logging

* Update docs

* Update calculation

* Restore condition

* Fix case where thumbnail is saved without frame

* Adjust debug logs

* Calculate best shm frame count

* Fix shm count calculation

* Catch missing frame

* Formatting

* Clarify docs

* Catch none frame in autotracking
2024-09-03 10:22:30 -06:00
Jason Hunter
0d7a148897 reindex events in batches to reduce memory and cpu load (#12124) 2024-08-29 20:19:50 -06:00
Jason Hunter
9e825811f2 Semantic Search API (#12105)
* initial event search api implementation

* fix lint

* fix tests

* move chromadb imports and pysqlite hotswap to fix tests

* remove unused import

* switch default limit to 50

* fix events accidently pulling inside chroma results loop
2024-08-29 20:19:50 -06:00
Jason Hunter
36cbffcc5e Semantic Search for Detections (#11899)
* Initial re-implementation of semantic search

* put docker-compose back and make reindex match docs

* remove debug code and fix import

* fix docs

* manually build pysqlite3 as binaries are only available for x86-64

* update comment in build_pysqlite3.sh

* only embed objects

* better error handling when genai fails

* ask ollama to pull requested model at startup

* update ollama docs

* address some PR review comments

* fix lint

* use IPC to write description, update docs for reindex

* remove gemini-pro-vision from docs as it will be unavailable soon

* fix OpenAI doc available models

* fix api error in gemini and metadata for embeddings
2024-08-29 20:19:50 -06:00