From a995872d1c5948e326beb03ac2fcb1c90aa786cc Mon Sep 17 00:00:00 2001 From: Nicolas Mowen Date: Fri, 14 Mar 2025 10:23:37 -0600 Subject: [PATCH] Add face recognition and license plate recognition to settings UI (#17152) * Refactor explore settings to classification settings * Cleanup * Add face config section * Add license plate recognition to settings * Update face recognition docs * Fix variable usage * Fix typo * Update docs/docs/configuration/face_recognition.md Co-authored-by: Josh Hawkins <32435876+hawkeye217@users.noreply.github.com> * Improve spacing and add face library to mobile * Clarify docs --------- Co-authored-by: Josh Hawkins <32435876+hawkeye217@users.noreply.github.com> --- docs/docs/configuration/face_recognition.md | 40 +- .../license_plate_recognition.md | 2 +- .../common/license_plate/mixin.py | 6 - web/src/components/menu/GeneralSettings.tsx | 13 + web/src/pages/FaceLibrary.tsx | 2 +- web/src/pages/Settings.tsx | 8 +- web/src/types/frigateConfig.ts | 3 +- .../settings/ClassificationSettingsView.tsx | 449 ++++++++++++++++++ web/src/views/settings/SearchSettingsView.tsx | 295 ------------ 9 files changed, 508 insertions(+), 310 deletions(-) create mode 100644 web/src/views/settings/ClassificationSettingsView.tsx delete mode 100644 web/src/views/settings/SearchSettingsView.tsx diff --git a/docs/docs/configuration/face_recognition.md b/docs/docs/configuration/face_recognition.md index 4d934afce..aac1be9b5 100644 --- a/docs/docs/configuration/face_recognition.md +++ b/docs/docs/configuration/face_recognition.md @@ -3,19 +3,55 @@ id: face_recognition title: Face Recognition --- -Face recognition allows people to be assigned names and when their face is recognized Frigate will assign the person's name as a sub label. This information is included in the UI, filters, as well as in notifications. +Face recognition identifies known individuals by matching detected faces with previously learned facial data. When a known person is recognized, their name will be added as a `sub_label`. This information is included in the UI, filters, as well as in notifications. + +## Model Requirements Frigate has support for CV2 Local Binary Pattern Face Recognizer to recognize faces, which runs locally. A lightweight face landmark detection model is also used to align faces before running them through the face recognizer. +Users running a Frigate+ model (or any custom model that natively detects faces) should ensure that `face` is added to the [list of objects to track](../plus/#available-label-types) either globally or for a specific camera. This will allow face detection to run at the same time as object detection and be more efficient. + +Users without a model that detects faces can still run face recognition. Frigate uses a lightweight DNN face detection model that runs on the CPU. In this case, you should _not_ define `face` in your list of objects to track. + +:::note + +Frigate needs to first detect a `face` before it can recognize a face. + +::: + +## Minimum System Requirements + +Face recognition is lightweight and runs on the CPU, there are no significantly different system requirements than running Frigate itself. + ## Configuration -Face recognition is disabled by default, face recognition must be enabled in your config file before it can be used. Face recognition is a global configuration setting. +Face recognition is disabled by default, face recognition must be enabled in the UI or in your config file before it can be used. Face recognition is a global configuration setting. ```yaml face_recognition: enabled: true ``` +## Advanced Configuration + +Fine-tune face recognition with these optional parameters: + +### Detection + +- `detection_threshold`: Face detection confidence score required before recognition runs: + - Default: `0.7` + - Note: This is field only applies to the standalone face detection model, `min_score` should be used to filter for models that have face detection built in. +- `min_area`: Defines the minimum size (in pixels) a face must be before recognition runs. + - Default: `500` pixels. + - Depending on the resolution of your camera's `detect` stream, you can increase this value to ignore small or distant faces. + +### Recognition + +- `recognition_threshold`: Recognition confidence score required to add the face to the object as a sub label. + - Default: `0.9`. +- `blur_confidence_filter`: Enables a filter that calculates how blurry the face is and adjusts the confidence based on this. + - Default: `True`. + ## Dataset The number of images needed for a sufficient training set for face recognition varies depending on several factors: diff --git a/docs/docs/configuration/license_plate_recognition.md b/docs/docs/configuration/license_plate_recognition.md index 776f30cf9..b7fbcdea1 100644 --- a/docs/docs/configuration/license_plate_recognition.md +++ b/docs/docs/configuration/license_plate_recognition.md @@ -51,7 +51,7 @@ Fine-tune the LPR feature using these optional parameters: - **`detection_threshold`**: License plate object detection confidence score required before recognition runs. - Default: `0.7` - - Note: If you are using a Frigate+ model and you set the `threshold` in your objects config for `license_plate` higher than this value, recognition will never run. It's best to ensure these values match, or this `detection_threshold` is lower than your object config `threshold`. + - Note: This is field only applies to the standalone license plate detection model, `min_score` should be used to filter for models that have license plate detection built in. - **`min_area`**: Defines the minimum size (in pixels) a license plate must be before recognition runs. - Default: `1000` pixels. - Depending on the resolution of your camera's `detect` stream, you can increase this value to ignore small or distant plates. diff --git a/frigate/data_processing/common/license_plate/mixin.py b/frigate/data_processing/common/license_plate/mixin.py index 751a674f5..c07163819 100644 --- a/frigate/data_processing/common/license_plate/mixin.py +++ b/frigate/data_processing/common/license_plate/mixin.py @@ -937,12 +937,6 @@ class LicensePlateProcessingMixin: if not license_plate: return - if license_plate.get("score") < self.lpr_config.detection_threshold: - logger.debug( - f"Plate detection score is less than the threshold ({license_plate['score']:0.2f} < {self.lpr_config.detection_threshold})" - ) - return - license_plate_box = license_plate.get("box") # check that license plate is valid diff --git a/web/src/components/menu/GeneralSettings.tsx b/web/src/components/menu/GeneralSettings.tsx index b07ace2a3..3d65cf8f0 100644 --- a/web/src/components/menu/GeneralSettings.tsx +++ b/web/src/components/menu/GeneralSettings.tsx @@ -255,6 +255,19 @@ export default function GeneralSettings({ className }: GeneralSettingsProps) { )} + {isAdmin && isMobile && ( + <> + + + + Configuration editor + + + + )} Appearance diff --git a/web/src/pages/FaceLibrary.tsx b/web/src/pages/FaceLibrary.tsx index fbb75c681..9099c4977 100644 --- a/web/src/pages/FaceLibrary.tsx +++ b/web/src/pages/FaceLibrary.tsx @@ -344,7 +344,7 @@ function TrainingGrid({ key={image} image={image} faceNames={faceNames} - threshold={config.face_recognition.threshold} + threshold={config.face_recognition.recognition_threshold} selected={selectedFaces.includes(image)} onClick={() => onClickFace(image)} onRefresh={onRefresh} diff --git a/web/src/pages/Settings.tsx b/web/src/pages/Settings.tsx index bfc3f6f8e..8e0ead43a 100644 --- a/web/src/pages/Settings.tsx +++ b/web/src/pages/Settings.tsx @@ -35,7 +35,7 @@ import MotionTunerView from "@/views/settings/MotionTunerView"; import MasksAndZonesView from "@/views/settings/MasksAndZonesView"; import AuthenticationView from "@/views/settings/AuthenticationView"; import NotificationView from "@/views/settings/NotificationsSettingsView"; -import SearchSettingsView from "@/views/settings/SearchSettingsView"; +import ClassificationSettingsView from "@/views/settings/ClassificationSettingsView"; import UiSettingsView from "@/views/settings/UiSettingsView"; import { useSearchEffect } from "@/hooks/use-overlay-state"; import { useSearchParams } from "react-router-dom"; @@ -46,7 +46,7 @@ import { useIsAdmin } from "@/hooks/use-is-admin"; const allSettingsViews = [ "UI settings", - "explore settings", + "classification settings", "camera settings", "masks / zones", "motion tuner", @@ -245,8 +245,8 @@ export default function Settings() {
{page == "UI settings" && } - {page == "explore settings" && ( - + {page == "classification settings" && ( + )} {page == "debug" && ( diff --git a/web/src/types/frigateConfig.ts b/web/src/types/frigateConfig.ts index 9b3d60606..2910118f4 100644 --- a/web/src/types/frigateConfig.ts +++ b/web/src/types/frigateConfig.ts @@ -333,7 +333,8 @@ export interface FrigateConfig { face_recognition: { enabled: boolean; - threshold: number; + detection_threshold: number; + recognition_threshold: number; }; ffmpeg: { diff --git a/web/src/views/settings/ClassificationSettingsView.tsx b/web/src/views/settings/ClassificationSettingsView.tsx new file mode 100644 index 000000000..f6ce3c37d --- /dev/null +++ b/web/src/views/settings/ClassificationSettingsView.tsx @@ -0,0 +1,449 @@ +import Heading from "@/components/ui/heading"; +import { FrigateConfig, SearchModelSize } from "@/types/frigateConfig"; +import useSWR from "swr"; +import axios from "axios"; +import ActivityIndicator from "@/components/indicators/activity-indicator"; +import { useCallback, useContext, useEffect, useState } from "react"; +import { Label } from "@/components/ui/label"; +import { Button } from "@/components/ui/button"; +import { Switch } from "@/components/ui/switch"; +import { Toaster } from "@/components/ui/sonner"; +import { toast } from "sonner"; +import { Separator } from "@/components/ui/separator"; +import { Link } from "react-router-dom"; +import { LuExternalLink } from "react-icons/lu"; +import { StatusBarMessagesContext } from "@/context/statusbar-provider"; +import { + Select, + SelectContent, + SelectGroup, + SelectItem, + SelectTrigger, +} from "@/components/ui/select"; + +type ClassificationSettings = { + search: { + enabled?: boolean; + reindex?: boolean; + model_size?: SearchModelSize; + }; + face: { + enabled?: boolean; + }; + lpr: { + enabled?: boolean; + }; +}; + +type ClassificationSettingsViewProps = { + setUnsavedChanges: React.Dispatch>; +}; +export default function ClassificationSettingsView({ + setUnsavedChanges, +}: ClassificationSettingsViewProps) { + const { data: config, mutate: updateConfig } = + useSWR("config"); + const [changedValue, setChangedValue] = useState(false); + const [isLoading, setIsLoading] = useState(false); + + const { addMessage, removeMessage } = useContext(StatusBarMessagesContext)!; + + const [classificationSettings, setClassificationSettings] = + useState({ + search: { + enabled: undefined, + reindex: undefined, + model_size: undefined, + }, + face: { + enabled: undefined, + }, + lpr: { + enabled: undefined, + }, + }); + + const [origSearchSettings, setOrigSearchSettings] = + useState({ + search: { + enabled: undefined, + reindex: undefined, + model_size: undefined, + }, + face: { + enabled: undefined, + }, + lpr: { + enabled: undefined, + }, + }); + + useEffect(() => { + if (config) { + if (classificationSettings?.search.enabled == undefined) { + setClassificationSettings({ + search: { + enabled: config.semantic_search.enabled, + reindex: config.semantic_search.reindex, + model_size: config.semantic_search.model_size, + }, + face: { + enabled: config.face_recognition.enabled, + }, + lpr: { + enabled: config.lpr.enabled, + }, + }); + } + + setOrigSearchSettings({ + search: { + enabled: config.semantic_search.enabled, + reindex: config.semantic_search.reindex, + model_size: config.semantic_search.model_size, + }, + face: { + enabled: config.face_recognition.enabled, + }, + lpr: { + enabled: config.lpr.enabled, + }, + }); + } + // we know that these deps are correct + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [config]); + + const handleClassificationConfigChange = ( + newConfig: Partial, + ) => { + setClassificationSettings((prevConfig) => ({ + search: { + ...prevConfig.search, + ...newConfig.search, + }, + face: { ...prevConfig.face, ...newConfig.face }, + lpr: { ...prevConfig.lpr, ...newConfig.lpr }, + })); + setUnsavedChanges(true); + setChangedValue(true); + }; + + const saveToConfig = useCallback(async () => { + setIsLoading(true); + + axios + .put( + `config/set?semantic_search.enabled=${classificationSettings.search.enabled ? "True" : "False"}&semantic_search.reindex=${classificationSettings.search.reindex ? "True" : "False"}&semantic_search.model_size=${classificationSettings.search.model_size}`, + { + requires_restart: 0, + }, + ) + .then((res) => { + if (res.status === 200) { + toast.success("Classification settings have been saved.", { + position: "top-center", + }); + setChangedValue(false); + updateConfig(); + } else { + toast.error(`Failed to save config changes: ${res.statusText}`, { + position: "top-center", + }); + } + }) + .catch((error) => { + const errorMessage = + error.response?.data?.message || + error.response?.data?.detail || + "Unknown error"; + toast.error(`Failed to save config changes: ${errorMessage}`, { + position: "top-center", + }); + }) + .finally(() => { + setIsLoading(false); + }); + }, [updateConfig, classificationSettings.search]); + + const onCancel = useCallback(() => { + setClassificationSettings(origSearchSettings); + setChangedValue(false); + removeMessage("search_settings", "search_settings"); + }, [origSearchSettings, removeMessage]); + + useEffect(() => { + if (changedValue) { + addMessage( + "search_settings", + `Unsaved Classification settings changes`, + undefined, + "search_settings", + ); + } else { + removeMessage("search_settings", "search_settings"); + } + // we know that these deps are correct + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [changedValue]); + + useEffect(() => { + document.title = "Classification Settings - Frigate"; + }, []); + + if (!config) { + return ; + } + + return ( +
+ +
+ + Classification Settings + + + + Semantic Search + +
+
+

+ Semantic Search in Frigate allows you to find tracked objects + within your review items using either the image itself, a + user-defined text description, or an automatically generated one. +

+ +
+ + Read the Documentation + + +
+
+
+ +
+
+ { + handleClassificationConfigChange({ + search: { enabled: isChecked }, + }); + }} + /> +
+ +
+
+
+
+ { + handleClassificationConfigChange({ + search: { reindex: isChecked }, + }); + }} + /> +
+ +
+
+
+ Re-indexing will reprocess all thumbnails and descriptions (if + enabled) and apply the embeddings on each startup.{" "} + Don't forget to disable the option after restarting! +
+
+
+
+
Model Size
+
+

+ The size of the model used for Semantic Search embeddings. +

+
    +
  • + Using small employs a quantized version of the + model that uses less RAM and runs faster on CPU with a very + negligible difference in embedding quality. +
    • +
  • + Using large employs the full Jina model and will + automatically run on the GPU if applicable. +
    • +
+
+
+ +
+
+ +
+ + + + Face Recognition + +
+
+

+ Face recognition allows people to be assigned names and when + their face is recognized Frigate will assign the person's name + as a sub label. This information is included in the UI, filters, + as well as in notifications. +

+ +
+ + Read the Documentation + + +
+
+
+ +
+
+ { + handleClassificationConfigChange({ + face: { enabled: isChecked }, + }); + }} + /> +
+ +
+
+
+ + + + + License Plate Recognition + +
+
+

+ Frigate can recognize license plates on vehicles and + automatically add the detected characters to the + recognized_license_plate field or a known name as a sub_label to + objects that are of type car. A common use case may be to read + the license plates of cars pulling into a driveway or cars + passing by on a street. +

+ +
+ + Read the Documentation + + +
+
+
+ +
+
+ { + handleClassificationConfigChange({ + lpr: { enabled: isChecked }, + }); + }} + /> +
+ +
+
+
+ + + +
+ + +
+
+
+
+ ); +} diff --git a/web/src/views/settings/SearchSettingsView.tsx b/web/src/views/settings/SearchSettingsView.tsx deleted file mode 100644 index b3f35bde7..000000000 --- a/web/src/views/settings/SearchSettingsView.tsx +++ /dev/null @@ -1,295 +0,0 @@ -import Heading from "@/components/ui/heading"; -import { FrigateConfig, SearchModelSize } from "@/types/frigateConfig"; -import useSWR from "swr"; -import axios from "axios"; -import ActivityIndicator from "@/components/indicators/activity-indicator"; -import { useCallback, useContext, useEffect, useState } from "react"; -import { Label } from "@/components/ui/label"; -import { Button } from "@/components/ui/button"; -import { Switch } from "@/components/ui/switch"; -import { Toaster } from "@/components/ui/sonner"; -import { toast } from "sonner"; -import { Separator } from "@/components/ui/separator"; -import { Link } from "react-router-dom"; -import { LuExternalLink } from "react-icons/lu"; -import { StatusBarMessagesContext } from "@/context/statusbar-provider"; -import { - Select, - SelectContent, - SelectGroup, - SelectItem, - SelectTrigger, -} from "@/components/ui/select"; - -type SearchSettingsViewProps = { - setUnsavedChanges: React.Dispatch>; -}; - -type SearchSettings = { - enabled?: boolean; - reindex?: boolean; - model_size?: SearchModelSize; -}; - -export default function SearchSettingsView({ - setUnsavedChanges, -}: SearchSettingsViewProps) { - const { data: config, mutate: updateConfig } = - useSWR("config"); - const [changedValue, setChangedValue] = useState(false); - const [isLoading, setIsLoading] = useState(false); - - const { addMessage, removeMessage } = useContext(StatusBarMessagesContext)!; - - const [searchSettings, setSearchSettings] = useState({ - enabled: undefined, - reindex: undefined, - model_size: undefined, - }); - - const [origSearchSettings, setOrigSearchSettings] = useState({ - enabled: undefined, - reindex: undefined, - model_size: undefined, - }); - - useEffect(() => { - if (config) { - if (searchSettings?.enabled == undefined) { - setSearchSettings({ - enabled: config.semantic_search.enabled, - reindex: config.semantic_search.reindex, - model_size: config.semantic_search.model_size, - }); - } - - setOrigSearchSettings({ - enabled: config.semantic_search.enabled, - reindex: config.semantic_search.reindex, - model_size: config.semantic_search.model_size, - }); - } - // we know that these deps are correct - // eslint-disable-next-line react-hooks/exhaustive-deps - }, [config]); - - const handleSearchConfigChange = (newConfig: Partial) => { - setSearchSettings((prevConfig) => ({ ...prevConfig, ...newConfig })); - setUnsavedChanges(true); - setChangedValue(true); - }; - - const saveToConfig = useCallback(async () => { - setIsLoading(true); - - axios - .put( - `config/set?semantic_search.enabled=${searchSettings.enabled ? "True" : "False"}&semantic_search.reindex=${searchSettings.reindex ? "True" : "False"}&semantic_search.model_size=${searchSettings.model_size}`, - { - requires_restart: 0, - }, - ) - .then((res) => { - if (res.status === 200) { - toast.success("Explore settings have been saved.", { - position: "top-center", - }); - setChangedValue(false); - updateConfig(); - } else { - toast.error(`Failed to save config changes: ${res.statusText}`, { - position: "top-center", - }); - } - }) - .catch((error) => { - const errorMessage = - error.response?.data?.message || - error.response?.data?.detail || - "Unknown error"; - toast.error(`Failed to save config changes: ${errorMessage}`, { - position: "top-center", - }); - }) - .finally(() => { - setIsLoading(false); - }); - }, [ - updateConfig, - searchSettings.enabled, - searchSettings.reindex, - searchSettings.model_size, - ]); - - const onCancel = useCallback(() => { - setSearchSettings(origSearchSettings); - setChangedValue(false); - removeMessage("search_settings", "search_settings"); - }, [origSearchSettings, removeMessage]); - - useEffect(() => { - if (changedValue) { - addMessage( - "search_settings", - `Unsaved Explore settings changes`, - undefined, - "search_settings", - ); - } else { - removeMessage("search_settings", "search_settings"); - } - // we know that these deps are correct - // eslint-disable-next-line react-hooks/exhaustive-deps - }, [changedValue]); - - useEffect(() => { - document.title = "Explore Settings - Frigate"; - }, []); - - if (!config) { - return ; - } - - return ( -
- -
- - Explore Settings - - - - Semantic Search - -
-
-

- Semantic Search in Frigate allows you to find tracked objects - within your review items using either the image itself, a - user-defined text description, or an automatically generated one. -

- -
- - Read the Documentation - - -
-
-
- -
-
- { - handleSearchConfigChange({ enabled: isChecked }); - }} - /> -
- -
-
-
-
- { - handleSearchConfigChange({ reindex: isChecked }); - }} - /> -
- -
-
-
- Re-indexing will reprocess all thumbnails and descriptions (if - enabled) and apply the embeddings on each startup.{" "} - Don't forget to disable the option after restarting! -
-
-
-
-
Model Size
-
-

- The size of the model used for Semantic Search embeddings. -

-
    -
  • - Using small employs a quantized version of the - model that uses less RAM and runs faster on CPU with a very - negligible difference in embedding quality. -
    • -
  • - Using large employs the full Jina model and will - automatically run on the GPU if applicable. -
    • -
-
-
- -
-
- - -
- - -
-
-
- ); -}