* Move log level initialization to log
* Use logger config
* Formatting
* Fix config order
* Set process names
---------
Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
* Set runtime
* Use count correctly
* Don't assume camera sizes
* Use separate zmq proxy for object detection
* Correct order
* Use forkserver
* Only store PID instead of entire process reference
* Cleanup
* Catch correct errors
* Fix typing
* Remove before_run from process util
The before_run never actually ran because:
You're right to suspect an issue with before_run not being called and a potential deadlock. The way you've implemented the run_wrapper using __getattribute__ for the run method of BaseProcess is a common pitfall in Python's multiprocessing, especially when combined with how multiprocessing.Process works internally.
Here's a breakdown of why before_run isn't being called and why you might be experiencing a deadlock:
The Problem: __getattribute__ and Process Serialization
When you create a multiprocessing.Process object and call start(), the multiprocessing module needs to serialize the process object (or at least enough of it to re-create the process in the new interpreter). It then pickles this serialized object and sends it to the newly spawned process.
The issue with your __getattribute__ implementation for run is that:
run is retrieved during serialization: When multiprocessing tries to pickle your Process object to send to the new process, it will likely access the run attribute. This triggers your __getattribute__ wrapper, which then tries to bind run_wrapper to self.
run_wrapper is bound to the parent process's self: The run_wrapper closure, when created in the parent process, captures the self (the Process instance) from the parent's memory space.
Deserialization creates a new object: In the child process, a new Process object is created by deserializing the pickled data. However, the run_wrapper method that was pickled still holds a reference to the self from the parent process. This is a subtle but critical distinction.
The child's run is not your wrapped run: When the child process starts, it internally calls its own run method. Because of the serialization and deserialization process, the run method that's ultimately executed in the child process is the original multiprocessing.Process.run or the Process.run if you had directly overridden it. Your __getattribute__ magic, which wraps run, isn't correctly applied to the Process object within the child's context.
* Cleanup
* Logging bugfix (#18465)
* use mp Manager to handle logging queues
A Python bug (https://github.com/python/cpython/issues/91555) was preventing logs from the embeddings maintainer process from printing. The bug is fixed in Python 3.14, but a viable workaround is to use the multiprocessing Manager, which better manages mp queues and causes the logging to work correctly.
* consolidate
* fix typing
* Fix typing
* Use global log queue
* Move to using process for logging
* Convert camera tracking to process
* Add more processes
* Finalize process
* Cleanup
* Cleanup typing
* Formatting
* Remove daemon
---------
Co-authored-by: Josh Hawkins <32435876+hawkeye217@users.noreply.github.com>
* Moved FrigateApp.init_config() into FrigateConfig.load()
* Move frigate config loading into main
* Store PlusApi in FrigateConfig
* Register SIGTERM handler in main
* Ensure logging is setup during config parsing
* Removed pointless try
* Moved config initialization out of FrigateApp
* Made FrigateApp.shm_frame_count into a function
* Removed log calls from signal handlers
python's logging calls are not re-entrant, which caused at least one of
these to deadlock randomly.
* Reopen stdout/err on process fork
This helps avoid deadlocks (https://github.com/python/cpython/issues/91776).
* Make mypy happy
* Whoops. I might have forgotten to save.
Truly an amateur mistake.
* Always call FrigateApp.stop()
* Refactor recordings config to be based off of review items
* Update object processing logic for when an event is created
* Migrate to deciding recording retention based on review items
* Refactor recording expiration to be based off of review items
* Remove remainder of recording events access
* Handle migration automatically
* Update version and cleanup
* Update docs
* Clarify docs
* Cleanup
* Target camera config
* Safely access all fields
* intentionally handle queues during shutdown and carefully manage shutdown order
* more carefully manage shutdown to avoid threadlocks
* use debug for signal logging
* ensure disabled cameras dont break shutdown
* typo
* Add config pub / sub pattern
* remove recording from feature metrics
* remove audio and feature metrics
* Check for updates from all cameras
* remove birdseye from camera metrics
* remove motion and detection camera metrics
* Ensure that all processes are stopped
* Stop communicators
* Detections
* Cleanup video output queue
* Use select for time sensitive polls
* Use ipc instead of tcp
* Use zmq for inter process communication
* Use localhost for reply and request
* Use pyobj instead of json and Need to use separate requestors for each audio listener
* Cleanup port defining
* Run ffmpeg sub process & video_properties as async
* Run recording cleanup in the main process
* More cleanup
* Use inter process communication to write recordings into the DB
* Formatting
* Store camera labels in dict and other optimizations
* Add max on timeout so it is at least 60
* Ensure db timeout is at least 60
* Update list once a day to ensure new labels are cleaned up
* Formatting
* Insert recordings as bulk instead of individually.
* Fix
* Refactor event and timeline cleanup
* Remove unused
* Refactored queues to use faster_fifo instead of mp.Queue
* Refactored LimitedQueue to include a counter for the number of items in the queue and updated put and get methods to use the counter
* Refactor app.py and util.py to use a custom Queue implementation called LQueue instead of the existing Queue
* Refactor put and get methods in LimitedQueue to handle queue size and blocking behavior more efficiently
* code format
* remove code from other branch (merging fuckup)
* Initial audio classification model implementation
* fix mypy
* Keep audio labelmap local
* Cleanup
* Start adding config for audio
* Add the detector
* Add audio detection process keypoints
* Build out base config
* Load labelmap correctly
* Fix config bugs
* Start audio process
* Fix startup issues
* Try to cleanup restarting
* Add ffmpeg input args
* Get audio detection working
* Save event to db
* End events if not heard for 30 seconds
* Use not heard config
* Stop ffmpeg when shutting down
* Fixes
* End events correctly
* Use api instead of event queue to save audio events
* Get events working
* Close threads when stop event is sent
* remove unused
* Only start audio process if at least one camera is enabled
* Add const for float
* Cleanup labelmap
* Add audio icon in frontend
* Add ability to toggle audio with mqtt
* Set initial audio value
* Fix audio enabling
* Close logpipe
* Isort
* Formatting
* Fix web tests
* Fix web tests
* Handle cases where args are a string
* Remove log
* Cleanup process close
* Use correct field
* Simplify if statement
* Use var for localhost
* Add audio detectors docs
* Add restream docs to mention audio detection
* Add full config docs
* Fix links to other docs
---------
Co-authored-by: Jason Hunter <hunterjm@gmail.com>
* Update pull_request.yml
* Add temporary table for deletion and use pagination to process recordings in chunks for deletion of recordings with missing files
* move RecordingsToDelete class to models.py
* recording cleanup: bugfixes
* Update cleanup.py
* improve log message in cleanup.py
Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
---------
Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>
* Enable auto vacuums
* Enable auto vacuum
* Fix separator
* Fix separator and remove incorrect log
* Limit to 1 row since that is all that is used
* Add index on camera + segment_size
* Formatting
* Increase timeout and cache_size
* Set DB mode to NORMAL synchronous level
* Formatting
* Vacuum every 2 weeks
* Remove fstring
* Use string
* Use consts
* Add isort and ruff linter
Both linters are pretty common among modern python code bases.
The isort tool provides stable sorting and grouping, as well as pruning
of unused imports.
Ruff is a modern linter, that is very fast due to being written in rust.
It can detect many common issues in a python codebase.
Removes the pylint dev requirement, since ruff replaces it.
* treewide: fix issues detected by ruff
* treewide: fix bare except clauses
* .devcontainer: Set up isort
* treewide: optimize imports
* treewide: apply black
* treewide: make regex patterns raw strings
This is necessary for escape sequences to be properly recognized.
* Move recordings management to own process and ensure db multiprocess access
* remove reference to old threads
* Cleanup directory remover
* Mypy fixes
* Fix mypy
* Add support back for setting record via MQTT and WS
* Formatting
* Fix rebase issue
