Frigate - Realtime Object Detection for IP Cameras

Note: This version requires the use of a Google Coral USB Accelerator

Uses OpenCV and Tensorflow to perform realtime object detection locally for IP cameras. Designed for integration with HomeAssistant or others via MQTT.

Leverages multiprocessing and threads heavily with an emphasis on realtime over processing every frame
Allows you to define specific regions (squares) in the image to look for objects
No motion detection (for now)
Object detection with Tensorflow runs in a separate thread
Object info is published over MQTT for integration into HomeAssistant as a binary sensor
An endpoint is available to view an MJPEG stream for debugging

Example video (from older version)

You see multiple bounding boxes because it draws bounding boxes from all frames in the past 1 second where a person was detected. Not all of the bounding boxes were from the current frame.

Getting Started

Build the container with

docker build -t frigate .

The mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite model is included and used by default. You can use your own model and labels by mounting files in the container at /frozen_inference_graph.pb and /label_map.pbtext. Models must be compatible with the Coral according to this.

Run the container with

docker run --rm \
--privileged \
-v /dev/bus/usb:/dev/bus/usb \
-v <path_to_config_dir>:/config:ro \
-v /etc/localtime:/etc/localtime:ro \
-p 5000:5000 \
-e FRIGATE_RTSP_PASSWORD='password' \
frigate:latest

Example docker-compose:

  frigate:
    container_name: frigate
    restart: unless-stopped
    privileged: true
    image: frigate:latest
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /etc/localtime:/etc/localtime:ro
      - <path_to_config>:/config
    ports:
      - "5000:5000"
    environment:
      FRIGATE_RTSP_PASSWORD: "password"

A config.yml file must exist in the config directory. See example here.

Access the mjpeg stream at http://localhost:5000/<camera_name> and the best person snapshot at http://localhost:5000/<camera_name>/best_person.jpg

Integration with HomeAssistant

camera:
  - name: Camera Last Person
    platform: mqtt
    topic: frigate/<camera_name>/snapshot

binary_sensor:
  - name: Camera Person
    platform: mqtt
    state_topic: "frigate/<camera_name>/objects"
    value_template: '{{ value_json.person }}'
    device_class: motion
    availability_topic: "frigate/available"

automation:
  - alias: Alert me if a person is detected while armed away
    trigger: 
      platform: state
      entity_id: binary_sensor.camera_person
      from: 'off'
      to: 'on'
    condition:
      - condition: state
        entity_id: alarm_control_panel.home_alarm
        state: armed_away
    action:
      - service: notify.user_telegram
        data:
          message: "A person was detected."
          data:
            photo:
              - url: http://<ip>:5000/<camera_name>/best_person.jpg
                caption: A person was detected.

Tips

Lower the framerate of the video feed on the camera to reduce the CPU usage for capturing the feed

Future improvements

Remove motion detection for now
Try running object detection in a thread rather than a process
Implement min person size again
Switch to a config file
Handle multiple cameras in the same container
Attempt to figure out coral symlinking
Add object list to config with min scores for mqtt
Move mjpeg encoding to a separate process
Simplify motion detection (check entire image against mask, resize instead of gaussian blur)
See if motion detection is even worth running
Scan for people across entire image rather than specfic regions
Dynamically resize detection area and follow people
Add ability to turn detection on and off via MQTT
Output movie clips of people for notifications, etc.
Integrate with homeassistant push camera
Merge bounding boxes that span multiple regions
Implement mode to save labeled objects for training
Try and reduce CPU usage by simplifying the tensorflow model to just include the objects we care about
Look into GPU accelerated decoding of RTSP stream
Send video over a socket and use JSMPEG
Look into neural compute stick

4.7 KiB Raw Blame History