Mirrors/blakeblackshear.frigate
1
0
Fork 0
mirror of https://github.com/blakeblackshear/frigate.git synced 2025-07-30 13:48:07 +02:00
Code Issues Projects Releases Wiki Activity

Adding fixes

Browse Source
This commit is contained in:
OmriAx 2025-02-26 14:04:27 +02:00
parent 11177292de
commit ee45f50e09
4 changed files with 612 additions and 180 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
docs/docs/configuration/object_detectors.md
View File

@ -11,7 +11,7 @@ Frigate supports multiple different detectors that work on different types of ha
**Most Hardware** **Most Hardware**
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB and m.2 format allowing for a wide range of compatibility with devices. - [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB and m.2 format allowing for a wide range of compatibility with devices.
- [Hailo](#hailo-8l): The Hailo8 AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices. - [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
**AMD** **AMD**
- [ROCm](#amdrocm-gpu-detector): ROCm can run on AMD Discrete GPUs to provide efficient object detection. - [ROCm](#amdrocm-gpu-detector): ROCm can run on AMD Discrete GPUs to provide efficient object detection.
@ -124,14 +124,36 @@ detectors:
device: pci device: pci
``` ```
## Hailo-8l ## Hailo-8
This detector is available for use with Hailo-8 AI Acceleration Module. This detector is available for use with Hailo-8 and Hailo-8L AI Acceleration Module.
See the [installation docs](../frigate/installation.md#hailo-8l) for information on configuring the hailo8. See the [installation docs](../frigate/installation.md#hailo-8l) for information on configuring the hailo8.
### Configuration ### Configuration
#### YOLO (Recommended)
```yaml
detectors:
hailo8l:
type: hailo8l
device: PCIe
model:
width: 640
height: 640
input_tensor: nhwc
input_pixel_format: rgb
input_dtype: int
model_type: hailoyolo
# The detector will automatically use the appropriate model:
# - YOLOv8s for Hailo-8L hardware
# - YOLOv8m for Hailo-8 hardware
```
#### SSD
```yaml ```yaml
detectors: detectors:
hailo8l: hailo8l:
@ -148,6 +170,33 @@ model:
``` ```
### Custom Models
The Hailo-8l detector supports all YOLO models that have been compiled for the Hailo hardware and include post-processing. The detector automatically detects your hardware type (Hailo-8 or Hailo-8L) and uses the appropriate model.
#### Using a Custom URL
You can specify a custom URL to download a model directly:
```yaml
detectors:
hailo8l:
type: hailo8l
device: PCIe
url: https://custom-model-url.com/path/to/model.hef
model:
width: 640
height: 640
input_tensor: nhwc
input_pixel_format: rgb
input_dtype: int
model_type: hailoyolo
```
The detector will automatically handle different output formats from all supported YOLO variants. It's important to match the `model_type` with the actual model architecture for proper processing.
* Tsted custom models : yolov5 , yolov8 , yolov9 , yolov11
## OpenVINO Detector ## OpenVINO Detector
The OpenVINO detector type runs an OpenVINO IR model on AMD and Intel CPUs, Intel GPUs and Intel VPU hardware. To configure an OpenVINO detector, set the `"type"` attribute to `"openvino"`. The OpenVINO detector type runs an OpenVINO IR model on AMD and Intel CPUs, Intel GPUs and Intel VPU hardware. To configure an OpenVINO detector, set the `"type"` attribute to `"openvino"`.

32
docs/docs/frigate/hardware.md
View File

@ -92,11 +92,37 @@ Inference speeds will vary greatly depending on the GPU and the model used.
With the [rocm](../configuration/object_detectors.md#amdrocm-gpu-detector) detector Frigate can take advantage of many discrete AMD GPUs. With the [rocm](../configuration/object_detectors.md#amdrocm-gpu-detector) detector Frigate can take advantage of many discrete AMD GPUs.
### Hailo-8l PCIe ### Hailo-8
Frigate supports the Hailo-8l M.2 card on any hardware but currently it is only tested on the Raspberry Pi5 PCIe hat from the AI kit. Frigate supports the Hailo8 and Hailo-8L AI Acceleration Module on compatible hardware platforms, including the Raspberry Pi 5 with the PCIe hat from the AI kit. The Hailo accelerator provides dedicated hardware for efficiently running neural network inference.
The inference time for the Hailo-8L chip at time of writing is around 17-21 ms for the SSD MobileNet Version 1 model. The inference time for the Hailo-8L chip is around 17-21 ms for the SSD MobileNet Version 1 model and 15-18 ms for YOLOv8s models. For the more powerful Hailo-8 chip, the YOLOv8m model has an inference time of approximately 12-15 ms.
In real-world testing with 8 cameras running simultaneously, each camera maintained a detection rate of approximately 20-25 FPS, demonstrating the Hailo accelerator's capability to handle multiple video streams efficiently.
Testing on x86 platforms has also been conducted with excellent results. The x86 implementation benefits from having two PCIe lanes available instead of one, resulting in improved FPS , throughput and lower latency compared to the Raspberry Pi setup.
#### Supported Models
The Hailo-8L detector supports all YOLO variants that have been compiled for Hailo hardware with post-processing, including:
- YOLOv5
- YOLOv8
- any Yolo variant with HailoRT Post-process
- SSD mobilnet v1
| Model Type | Hardware | Inference Time | Resolution |
|------------|----------|----------------|------------|
| SSD MobileNet V1 | Hailo-8L (RPi) | 17-21 ms | 300×300 |
| SSD MobileNet V1 | Hailo-8L (x86) | 12-15 ms | 300×300 |
| SSD MobileNet V1 | Hailo-8 (rpi) | 13-16 ms | 300×300 |
| YOLOv8s | Hailo-8L (RPi) | 15-18 ms | 640×640 |
| YOLOv8s | Hailo-8L (x86) | 10-13 ms | 640×640 |
| YOLOv8m | Hailo-8 (RPi) | 12-15 ms | 640×640 |
| YOLOv8m | Hailo-8 (x86) | 8-11 ms | 640×640 |
The detector automatically detects your hardware type (Hailo-8 or Hailo-8L) and downloads the appropriate model. The Hailo detector optimizes inference by maintaining a persistent pipeline between detection calls, reducing overhead and providing fast, consistent performance with multiple cameras.
## Community Supported Detectors ## Community Supported Detectors

4
docs/docs/frigate/installation.md
View File

@ -100,9 +100,9 @@ By default, the Raspberry Pi limits the amount of memory available to the GPU. I
Additionally, the USB Coral draws a considerable amount of power. If using any other USB devices such as an SSD, you will experience instability due to the Pi not providing enough power to USB devices. You will need to purchase an external USB hub with it's own power supply. Some have reported success with <a href="https://amzn.to/3a2mH0P" target="_blank" rel="nofollow noopener sponsored">this</a> (affiliate link). Additionally, the USB Coral draws a considerable amount of power. If using any other USB devices such as an SSD, you will experience instability due to the Pi not providing enough power to USB devices. You will need to purchase an external USB hub with it's own power supply. Some have reported success with <a href="https://amzn.to/3a2mH0P" target="_blank" rel="nofollow noopener sponsored">this</a> (affiliate link).
### Hailo-8L ### Hailo-8
The Hailo-8L is an M.2 card typically connected to a carrier board for PCIe, which then connects to the Raspberry Pi 5 as part of the AI Kit. However, it can also be used on other boards equipped with an M.2 M key edge connector. The Hailo-8 and Hailo-8L AI accelerators are available in both M.2 and HAT form factors for the Raspberry Pi. The M.2 version typically connects to a carrier board for PCIe, which then interfaces with the Raspberry Pi 5 as part of the AI Kit. The HAT version can be mounted directly onto compatible Raspberry Pi models. Both form factors have been successfully tested on x86 platforms as well, making them versatile options for various computing environments.
#### Installation #### Installation

669
frigate/detectors/plugins/hailo8l.py
View File

@ -1,204 +1,561 @@
import logging import logging
import os import os
import queue
import threading
import subprocess import subprocess
import urllib.request import urllib.request
import numpy as np import numpy as np
try:
from hailo_platform import ( from hailo_platform import (
HEF, HEF,
ConfigureParams, ConfigureParams,
FormatType, FormatType,
HailoRTException, HailoRTException,
HailoStreamInterface, HailoStreamInterface,
VDevice,
HailoSchedulingAlgorithm,
InferVStreams, InferVStreams,
InputVStreamParams, InputVStreamParams,
OutputVStreamParams OutputVStreamParams,
VDevice,
) )
from frigate.detectors.detection_api import DetectionApi except ModuleNotFoundError:
from frigate.detectors.detector_config import BaseDetectorConfig pass
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
from typing_extensions import Literal from typing_extensions import Literal
from typing import Optional from typing import Dict, Optional, List
from functools import partial
from frigate.detectors.detection_api import DetectionApi
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum, InputTensorEnum, PixelFormatEnum, InputDTypeEnum
# Setup logging
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
file_handler = logging.FileHandler('hailo_detector_debug.log')
file_handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
# Define the detector key for Hailo
DETECTOR_KEY = "hailo8l" DETECTOR_KEY = "hailo8l"
ARCH = None
class ModelConfig(BaseModel): def detect_hailo_arch():
path: str = Field(default=None, title="Model Path") try:
# Run the hailortcli command to get device information
result = subprocess.run(['hailortcli', 'fw-control', 'identify'], capture_output=True, text=True)
# Check if the command was successful
if result.returncode != 0:
print(f"Error running hailortcli: {result.stderr}")
return None
# Search for the "Device Architecture" line in the output
for line in result.stdout.split('\n'):
if "Device Architecture" in line:
if "HAILO8L" in line:
return "hailo8l"
elif "HAILO8" in line:
return "hailo8"
print("Could not determine Hailo architecture from device information.")
return None
except Exception as e:
print(f"An error occurred while detecting Hailo architecture: {e}")
return None
# Configuration class for Hailo detector
class HailoDetectorConfig(BaseDetectorConfig): class HailoDetectorConfig(BaseDetectorConfig):
type: Literal[DETECTOR_KEY] type: Literal[DETECTOR_KEY] # Type of the detector
device: str = Field(default="PCIe", title="Device Type") device: str = Field(default="PCIe", title="Device Type") # Device type (e.g., PCIe)
url: Optional[str] = Field(default=None, title="Model URL") url: Optional[str] = Field(default=None, title="Custom Model URL")
dir: Optional[str] = Field(default=None, title="Model Directory")
model: ModelConfig
class HailoAsyncInference:
def __init__(self, config: HailoDetectorConfig):
self.config = config
self.input_queue = queue.Queue()
self.output_queue = queue.Queue()
params = VDevice.create_params()
params.scheduling_algorithm = HailoSchedulingAlgorithm.ROUND_ROBIN
self.target = VDevice(params)
# Initialize HEF
self.hef = HEF(self.config.model.path)
self.infer_model = self.target.create_infer_model(self.config.model.path)
self.infer_model.set_batch_size(1)
def infer(self):
while True:
batch_data = self.input_queue.get()
if batch_data is None:
break
with self.infer_model.configure() as configured_model:
bindings_list = []
for frame in batch_data:
# Create empty output buffers
output_buffers = {
output_info.name: np.empty(
self.infer_model.output(output_info.name).shape,
dtype=np.float32
)
for output_info in self.hef.get_output_vstream_infos()
}
# Create bindings using the configured model
binding = configured_model.create_bindings(output_buffers=output_buffers)
binding.input().set_buffer(frame)
bindings_list.append(binding)
# Run async inference on the configured model
configured_model.run_async(bindings_list, partial(self._callback, batch_data=batch_data))
def _callback(self, completion_info, bindings_list, batch_data):
if completion_info.exception:
logger.error(f"Inference error: {completion_info.exception}")
else:
results = [binding.output().get_buffer() for binding in bindings_list]
self.output_queue.put((batch_data, results))
def stop(self):
self.input_queue.put(None)
# Hailo detector class implementation
class HailoDetector(DetectionApi): class HailoDetector(DetectionApi):
type_key = DETECTOR_KEY type_key = DETECTOR_KEY # Set the type key to the Hailo detector key
DEFAULT_CACHE_DIR = "/config/model_cache/"
def __init__(self, detector_config: HailoDetectorConfig): def __init__(self, detector_config: HailoDetectorConfig):
super().__init__(detector_config) print(f"[INIT] Starting HailoDetector initialization with config: {detector_config}")
self.config = detector_config logger.info(f"[INIT] Starting HailoDetector initialization with config: {detector_config}")
# Get the model path # Set global ARCH variable
model_path = self.check_and_prepare_model() global ARCH
self.config.model.path = model_path ARCH = detect_hailo_arch()
print(self.config.model.path) logger.info(f"[INIT] Detected Hailo architecture: {ARCH}")
# Initialize async inference with the correct model path supported_models = [
self.async_inference = HailoAsyncInference(detector_config) ModelTypeEnum.ssd,
self.worker_thread = threading.Thread(target=self.async_inference.infer) ModelTypeEnum.yolov9,
self.worker_thread.start() ModelTypeEnum.hailoyolo,
]
# Initialize device type and model path from the configuration
self.h8l_device_type = detector_config.device
self.h8l_model_path = detector_config.model.path
self.h8l_model_type = detector_config.model.model_type
def check_and_prepare_model(self) -> str: # Set configuration based on model type
""" self.set_correct_config(self.h8l_model_type)
Check if model exists at specified path, download from URL if needed.
Returns the final model path to use.
"""
# Ensure cache directory exists
if not os.path.exists(self.DEFAULT_CACHE_DIR):
os.makedirs(self.DEFAULT_CACHE_DIR)
model_path = self.config.dir # the directory path of the model # Override with custom URL if provided
model_url = self.config.url # the url of the model if hasattr(detector_config, "url") and detector_config.url:
self.model_url = detector_config.url
self.expected_model_filename = self.model_url.split('/')[-1]
if (model_path and os.path.isfile(model_path)): self.check_and_prepare_model()
return model_path
if (model_url):
model_filename = os.path.basename(model_url)
model_file_path = os.path.join(self.DEFAULT_CACHE_DIR, model_filename)
if os.path.isfile(model_file_path):
return model_file_path
else:
logger.info(f"Downloading model from URL: {model_url}")
try: try:
urllib.request.urlretrieve(model_url, model_file_path) # Validate device type
logger.info(f"Model downloaded successfully to: {model_file_path}") if self.h8l_device_type not in ["PCIe", "M.2"]:
return model_file_path raise ValueError(f"Unsupported device type: {self.h8l_device_type}")
# Initialize the Hailo device
logger.info("[INIT] Creating VDevice instance")
self.target = VDevice()
# Load the HEF (Hailo's binary format for neural networks)
logger.info(f"[INIT] Loading HEF from {self.h8l_model_path}")
self.hef = HEF(self.h8l_model_path)
# Create configuration parameters from the HEF
logger.info("[INIT] Creating configuration parameters")
self.configure_params = ConfigureParams.create_from_hef(
hef=self.hef, interface=HailoStreamInterface.PCIe
)
# Configure the device with the HEF
logger.info("[INIT] Configuring device with HEF")
self.network_groups = self.target.configure(self.hef, self.configure_params)
self.network_group = self.network_groups[0]
self.network_group_params = self.network_group.create_params()
# Create input and output virtual stream parameters
logger.info("[INIT] Creating input/output stream parameters")
self.input_vstream_params = InputVStreamParams.make(
self.network_group,
format_type=self.hef.get_input_vstream_infos()[0].format.type,
)
self.output_vstream_params = OutputVStreamParams.make(
self.network_group, format_type=getattr(FormatType, self.output_type)
)
# Get input and output stream information from the HEF
self.input_vstream_info = self.hef.get_input_vstream_infos()
self.output_vstream_info = self.hef.get_output_vstream_infos()
for i, info in enumerate(self.input_vstream_info):
logger.info(f"[INIT] Input Stream {i}: Name={info.name}, Format={info.format}, Shape={info.shape}")
for i, info in enumerate(self.output_vstream_info):
logger.info(f"[INIT] Output Stream {i}: Name={info.name}, Format={info.format}, Shape={info.shape}")
logger.info("Hailo device initialized successfully")
except HailoRTException as e:
logger.error(f"HailoRTException during initialization: {e}")
raise
except Exception as e: except Exception as e:
logger.error(f"Failed to download model: {str(e)}") logger.error(f"Failed to initialize Hailo device: {e}")
raise RuntimeError(f"Failed to download model from {model_url}") raise
raise RuntimeError("No valid model path or URL provided")
def set_correct_config(self, modelname):
if modelname == ModelTypeEnum.ssd:
self.h8l_model_height = 300
self.h8l_model_width = 300
self.h8l_tensor_format = InputTensorEnum.nhwc
self.h8l_pixel_format = PixelFormatEnum.rgb
self.h8l_input_dtype = InputDTypeEnum.float
self.cache_dir = "/config/model_cache/h8l_cache"
self.expected_model_filename = "ssd_mobilenet_v1.hef"
self.output_type = "FLOAT32"
if ARCH == "hailo8":
self.model_url = "https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ModelZoo/Compiled/v2.14.0/hailo8/ssd_mobilenet_v1.hef"
else:
self.model_url = "https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ModelZoo/Compiled/v2.14.0/hailo8l/ssd_mobilenet_v1.hef"
else:
self.h8l_model_height = 640
self.h8l_model_width = 640
self.h8l_tensor_format = InputTensorEnum.nhwc
self.h8l_pixel_format = PixelFormatEnum.rgb # Default to RGB
self.h8l_input_dtype = InputDTypeEnum.int
self.cache_dir = "/config/model_cache/h8l_cache"
self.output_type = "FLOAT32"
if ARCH == "hailo8":
self.model_url = "https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ModelZoo/Compiled/v2.14.0/hailo8/yolov8m.hef"
self.expected_model_filename = "yolov8m.hef"
else:
self.model_url = "https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ModelZoo/Compiled/v2.14.0/hailo8l/yolov8s.hef"
self.expected_model_filename = "yolov8s.hef"
def check_and_prepare_model(self):
logger.info(f"[CHECK_MODEL] Checking for model at {self.cache_dir}/{self.expected_model_filename}")
# Ensure cache directory exists
if not os.path.exists(self.cache_dir):
logger.info(f"[CHECK_MODEL] Creating cache directory: {self.cache_dir}")
os.makedirs(self.cache_dir)
# Check for the expected model file
model_file_path = os.path.join(self.cache_dir, self.expected_model_filename)
if not os.path.isfile(model_file_path):
logger.info(f"[CHECK_MODEL] Model not found at {model_file_path}, downloading from {self.model_url}")
urllib.request.urlretrieve(self.model_url, model_file_path)
logger.info(f"[CHECK_MODEL] Model downloaded to {model_file_path}")
else:
logger.info(f"[CHECK_MODEL] Model already exists at {model_file_path}")
self.h8l_model_path = model_file_path
def detect_raw(self, tensor_input): def detect_raw(self, tensor_input):
""" logger.info("[DETECT_RAW] Starting detection")
Perform inference and return raw detection results.
""" if tensor_input is None:
preprocessed_input = self.preprocess(tensor_input) error_msg = "[DETECT_RAW] The 'tensor_input' argument must be provided"
self.async_inference.input_queue.put([preprocessed_input]) logger.error(error_msg)
raise ValueError(error_msg)
# Log input tensor information
logger.info(f"[DETECT_RAW] Input tensor type: {type(tensor_input)}")
if isinstance(tensor_input, np.ndarray):
logger.info(f"[DETECT_RAW] Input tensor shape: {tensor_input.shape}")
logger.info(f"[DETECT_RAW] Input tensor dtype: {tensor_input.dtype}")
logger.info(f"[DETECT_RAW] Input tensor min value: {np.min(tensor_input)}")
logger.info(f"[DETECT_RAW] Input tensor max value: {np.max(tensor_input)}")
logger.info(f"[DETECT_RAW] Input tensor mean value: {np.mean(tensor_input)}")
# Print sample of the tensor (first few elements)
flat_sample = tensor_input.flatten()[:10]
logger.info(f"[DETECT_RAW] Input tensor sample: {flat_sample}")
elif isinstance(tensor_input, list):
logger.info(f"[DETECT_RAW] Input is a list with length: {len(tensor_input)}")
tensor_input = np.array(tensor_input)
logger.info(f"[DETECT_RAW] Converted to array with shape: {tensor_input.shape}, dtype: {tensor_input.dtype}")
elif isinstance(tensor_input, dict):
logger.info(f"[DETECT_RAW] Input is a dictionary with keys: {tensor_input.keys()}")
input_data = tensor_input
logger.debug("[DETECT_RAW] Input data prepared for inference")
try: try:
batch_data, raw_results = self.async_inference.output_queue.get(timeout=5) logger.info("[DETECT_RAW] Creating inference pipeline")
return self.postprocess(raw_results) with InferVStreams(
except queue.Empty: self.network_group,
logger.warning("Inference timed out") self.input_vstream_params,
return np.zeros((20, 6), np.float32) self.output_vstream_params,
) as infer_pipeline:
def preprocess(self, frame): input_dict = {}
input_shape = (self.async_inference.hef.get_input_vstream_infos()[0].shape) if isinstance(input_data, dict):
resized_frame = np.resize(frame, input_shape) logger.info("[DETECT_RAW] Input is already a dictionary, using as-is")
return resized_frame / 255.0 input_dict = input_data
elif isinstance(input_data, (list, tuple)):
def postprocess(self, raw_output): logger.info("[DETECT_RAW] Converting list/tuple to dictionary for inference")
model_type = self.async_inference.config.model.type for idx, layer_info in enumerate(self.input_vstream_info):
if model_type == "ssd_mobilenet_v1": input_dict[layer_info.name] = input_data[idx]
return self._process_ssd(raw_output) logger.info(f"[DETECT_RAW] Assigned data to input layer '{layer_info.name}'")
elif model_type in ["yolov8s", "yolov8m", "yolov6n"]:
return self._process_yolo(raw_output, version=model_type[-1])
else: else:
logger.error(f"Unsupported model type: {model_type}") if len(input_data.shape) == 3:
return [] logger.info(f"[DETECT_RAW] Adding batch dimension to input with shape {input_data.shape}")
input_data = np.expand_dims(input_data, axis=0)
logger.info(f"[DETECT_RAW] New input shape after adding batch dimension: {input_data.shape}")
input_dict[self.input_vstream_info[0].name] = input_data
logger.info(f"[DETECT_RAW] Assigned data to input layer '{self.input_vstream_info[0].name}'")
def _process_ssd(self, raw_output): logger.info(f"[DETECT_RAW] Final input dictionary keys: {list(input_dict.keys())}")
detections = []
for detection in raw_output[1]: # Log details about each input layer
score = detection[4] for key, value in input_dict.items():
if score >= self.async_inference.config.model.score_threshold: if isinstance(value, np.ndarray):
ymin, xmin, ymax, xmax = detection[:4] logger.info(f"[DETECT_RAW] Layer '{key}' has shape: {value.shape}, dtype: {value.dtype}")
detections.append({
"bounding_box": [xmin, ymin, xmax, ymax], logger.info("[DETECT_RAW] Activating network group")
"score": score, with self.network_group.activate(self.network_group_params):
"class": int(detection[5]) logger.info("[DETECT_RAW] Running inference")
}) raw_output = infer_pipeline.infer(input_dict)
logger.info(f"[DETECT_RAW] Inference complete, output keys: {list(raw_output.keys())}")
# Log details about output structure for debugging
for key, value in raw_output.items():
logger.info(f"[DETECT_RAW] Output layer '{key}' details:")
debug_output_structure(value, prefix=" ")
# Process outputs based on model type
if self.h8l_model_type in [ModelTypeEnum.hailoyolo, ModelTypeEnum.yolov9, ModelTypeEnum.yolox, ModelTypeEnum.yolonas]:
logger.info(f"[DETECT_RAW] Processing YOLO-type output for model type: {self.h8l_model_type}")
detections = self.process_yolo_output(raw_output)
else:
# Default to SSD processing
logger.info(f"[DETECT_RAW] Processing SSD output for model type: {self.h8l_model_type}")
expected_output_name = self.output_vstream_info[0].name
if expected_output_name not in raw_output:
error_msg = f"[DETECT_RAW] Missing output stream {expected_output_name} in inference results"
logger.error(error_msg)
return np.zeros((20, 6), np.float32)
detections = self.process_ssd_output(raw_output[expected_output_name])
logger.info(f"[DETECT_RAW] Processed detections shape: {detections.shape}")
return detections return detections
def _process_yolo(self, raw_output, version): except HailoRTException as e:
detections = [] logger.error(f"[DETECT_RAW] HailoRTException during inference: {e}")
for detection in raw_output[1]: return np.zeros((20, 6), np.float32)
confidence = detection[4] except Exception as e:
if confidence >= self.async_inference.config.model.score_threshold: logger.error(f"[DETECT_RAW] Exception during inference: {e}")
x, y, w, h = detection[:4] return np.zeros((20, 6), np.float32)
ymin, xmin, ymax, xmax = y - h / 2, x - w / 2, y + h / 2, x + w / 2 finally:
class_id = np.argmax(detection[5:]) logger.debug("[DETECT_RAW] Exiting function")
detections.append({
"bounding_box": [xmin, ymin, xmax, ymax], def process_yolo_output(self, raw_output):
"score": confidence, """
"class": class_id Process YOLO outputs to match the expected Frigate detection format.
}) Returns detections in the format [class_id, score, ymin, xmin, ymax, xmax]
"""
logger.info("[PROCESS_YOLO] Processing YOLO output")
# Initialize empty array for our results - match TFLite format
detections = np.zeros((20, 6), np.float32)
try:
# Identify output layers for boxes, classes, and scores
boxes_layer = None
classes_layer = None
scores_layer = None
count_layer = None
# Try to identify layers by name pattern
for key in raw_output.keys():
key_lower = key.lower()
if any(box_term in key_lower for box_term in ['box', 'bbox', 'location']):
boxes_layer = key
elif any(class_term in key_lower for class_term in ['class', 'category', 'label']):
classes_layer = key
elif any(score_term in key_lower for score_term in ['score', 'confidence', 'prob']):
scores_layer = key
elif any(count_term in key_lower for count_term in ['count', 'num', 'detection_count']):
count_layer = key
logger.info(f"[PROCESS_YOLO] Identified layers - Boxes: {boxes_layer}, Classes: {classes_layer}, "
f"Scores: {scores_layer}, Count: {count_layer}")
# If we found all necessary layers
if boxes_layer and classes_layer and scores_layer:
# Extract data from the identified layers
boxes = raw_output[boxes_layer]
class_ids = raw_output[classes_layer]
scores = raw_output[scores_layer]
# If these are lists, extract the first element (batch)
if isinstance(boxes, list) and len(boxes) > 0:
boxes = boxes[0]
if isinstance(class_ids, list) and len(class_ids) > 0:
class_ids = class_ids[0]
if isinstance(scores, list) and len(scores) > 0:
scores = scores[0]
# Get detection count (if available)
count = 0
if count_layer:
count_val = raw_output[count_layer]
if isinstance(count_val, list) and len(count_val) > 0:
count_val = count_val[0]
count = int(count_val[0] if isinstance(count_val, np.ndarray) else count_val)
else:
# Use the length of scores as count
count = len(scores) if hasattr(scores, '__len__') else 0
# Process detections like in the example
for i in range(count):
if i >= 20: # Limit to 20 detections
break
if scores[i] < 0.4: # Use 0.4 threshold as in the example
continue
# Add detection in the format [class_id, score, ymin, xmin, ymax, xmax]
detections[i] = [
float(class_ids[i]),
float(scores[i]),
float(boxes[i][0]), # ymin
float(boxes[i][1]), # xmin
float(boxes[i][2]), # ymax
float(boxes[i][3]), # xmax
]
else:
# Fallback: Try to process output as a combined detection array
logger.info("[PROCESS_YOLO] Couldn't identify separate output layers, trying unified format")
# Look for a detection array in the output
detection_layer = None
for key, value in raw_output.items():
if isinstance(value, list) and len(value) > 0:
if isinstance(value[0], np.ndarray) and value[0].ndim >= 2:
detection_layer = key
break
if detection_layer:
# Get the detection array
detection_array = raw_output[detection_layer]
if isinstance(detection_array, list):
detection_array = detection_array[0] # First batch
# Process each detection
detection_count = 0
for i, detection in enumerate(detection_array):
if detection_count >= 20:
break
# Format depends on YOLO variant but typically includes:
# class_id, score, box coordinates (could be [x,y,w,h] or [xmin,ymin,xmax,ymax])
# Extract elements based on shape
if len(detection) >= 6: # Likely [class_id, score, xmin, ymin, xmax, ymax]
class_id = detection[0]
score = detection[1]
# Check if this is actually [x, y, w, h, conf, class_id]
if score > 1.0: # Score shouldn't be > 1, might be a coordinate
# Reorganize assuming [x, y, w, h, conf, class_id] format
x, y, w, h, confidence, *class_probs = detection
# Get class with highest probability
if len(class_probs) > 1:
class_id = np.argmax(class_probs)
score = confidence * class_probs[class_id]
else:
class_id = class_probs[0]
score = confidence
# Convert [x,y,w,h] to [ymin,xmin,ymax,xmax]
xmin = x - w/2
ymin = y - h/2
xmax = x + w/2
ymax = y + h/2
else:
# Use as is, but verify we have box coordinates
xmin, ymin, xmax, ymax = detection[2:6]
elif len(detection) >= 4: # Might be [class_id, score, xmin, ymin]
class_id = detection[0]
score = detection[1]
# For incomplete boxes, assume zeros
xmin, ymin = detection[2:4]
xmax = xmin + 0.1 # Small default size
ymax = ymin + 0.1
else:
# Skip invalid detections
continue
# Skip low confidence detections
if score < 0.4:
continue
# Add to detection array
detections[detection_count] = [
float(class_id),
float(score),
float(ymin),
float(xmin),
float(ymax),
float(xmax)
]
detection_count += 1
logger.info(f"[PROCESS_YOLO] Processed {np.count_nonzero(detections[:, 1] > 0)} valid detections")
except Exception as e:
logger.error(f"[PROCESS_YOLO] Error processing YOLO output: {e}")
# detections already initialized as zeros
return detections return detections
def stop(self): def process_ssd_output(self, raw_output):
self.async_inference.stop() """
self.worker_thread.join() Process SSD MobileNet v1 output with special handling for jagged arrays
"""
logger.info("[PROCESS_SSD] Processing SSD output")
# Initialize empty lists for our results
all_detections = []
try:
if isinstance(raw_output, list) and len(raw_output) > 0:
# Handle first level of nesting
raw_detections = raw_output[0]
logger.debug(f"[PROCESS_SSD] First level output type: {type(raw_detections)}")
# Process all valid detections
for i, detection_group in enumerate(raw_detections):
# Skip empty arrays or invalid data
if not isinstance(detection_group, np.ndarray):
continue
# Skip empty arrays
if detection_group.size == 0:
continue
# For the arrays with actual detections
if detection_group.shape[0] > 0:
# Extract the detection data - typical format is (ymin, xmin, ymax, xmax, score)
for j in range(detection_group.shape[0]):
detection = detection_group[j]
# Check if we have 5 values (expected format)
if len(detection) == 5:
ymin, xmin, ymax, xmax, score = detection
class_id = i # Use index as class ID
# Add detection if score is reasonable
if 0 <= score <= 1.0 and score > 0.1: # Basic threshold
all_detections.append([float(class_id), float(score),
float(ymin), float(xmin),
float(ymax), float(xmax)])
# Convert to numpy array if we have valid detections
if all_detections:
detections_array = np.array(all_detections, dtype=np.float32)
# Sort by score (descending)
sorted_idx = np.argsort(detections_array[:, 1])[::-1]
detections_array = detections_array[sorted_idx]
# Take top 20 (or fewer if less available)
detections_array = detections_array[:20]
else:
detections_array = np.zeros((0, 6), dtype=np.float32)
except Exception as e:
logger.error(f"[PROCESS_SSD] Error processing SSD output: {e}")
detections_array = np.zeros((0, 6), dtype=np.float32)
# Pad to 20 detections if needed
if len(detections_array) < 20:
padding = np.zeros((20 - len(detections_array), 6), dtype=np.float32)
detections_array = np.vstack((detections_array, padding))
logger.info(f"[PROCESS_SSD] Final output shape: {detections_array.shape}")
return detections_array
def process_detections(self, raw_detections, threshold=0.5):
"""
Legacy detection processing method, kept for compatibility.
Now redirects to the more robust process_ssd_output method.
"""
logger.info("[PROCESS] Starting to process detections")
logger.info(f"[PROCESS] Using threshold: {threshold}")
# Wrap the raw_detections in a list to match expected format for process_ssd_output
if not isinstance(raw_detections, list):
raw_detections = [raw_detections]
# Process using the more robust method
return self.process_ssd_output(raw_detections)
def close(self):
logger.info("[CLOSE] Closing Hailo device")
try:
self.target.close()
logger.info("Hailo device closed successfully")
except Exception as e:
logger.error(f"Failed to close Hailo device: {e}")
raise