Commit Graph

69 Commits

Author SHA1 Message Date
Nicolas Mowen
b4d4adb75b
Limit recording retention to available storage (#3942)
* Add field and migration for segment size

* Store the segment size in db

* Add comment

* Add default

* Fix size parsing

* Include segment size in recordings endpoint

* Start adding storage maintainer

* Add storage maintainer and calculate average sizes

* Update comment

* Store segment and hour avg sizes per camera

* Formatting

* Keep track of total segment and hour averages

* Remove unused files

* Cleanup 2 hours of recordings at a time

* Formatting

* Fix bug

* Round segment size

* Cleanup some comments

* Handle case where segments are not deleted on initial run or is only retained segments

* Improve cleanup log

* Formatting

* Fix typo and improve logging

* Catch case where no recordings exist for camera

* Specifically define sort

* Handle edge case for cameras that only record part time

* Increase definition of part time recorder

* Remove warning about not supported storage based retention

* Add note about storage based retention to recording docs

* Add tests for storage maintenance calculation and cleanup

* Format tests

* Don't run for a camera with no recording segments

* Get size of file from cache

* Rework camera stats to be more efficient

* Remove total and other inefficencies

* Rewrite storage cleanup logic to be much more efficient

* Fix existing tests

* Fix bugs from tests

* Add another test

* Improve logging

* Formatting

* Set back correct loop time

* Update name

* Update comment

* Only include segments that have a nonzero size

* Catch case where camera has 0 nonzero segment durations

* Add test to cover zero bandwidth migration case

* Fix test

* Incorrect boolean logic

* Formatting

* Explicity re-define iterator
2022-10-09 06:28:26 -05:00
Nicolas Mowen
65c211bb6d
Catch ffprobe empty stdout (#4016) 2022-10-04 19:56:04 -05:00
Blake Blackshear
8669c29e3d
don't overwrite segments that already exist (#3995) 2022-10-01 18:11:29 -05:00
Blake Blackshear
3c46a33992 revert false warning messages 2022-07-20 06:55:06 -05:00
deviant77
ed1897db71
Add log message when discarding recording segments in cache (#3439)
* Add log message when discarding recording segments in cache

Currently Frigate silently discards recording segments in cache if there's more than "keep_count" for a camera, which can happen on high load/busy/slow systems.
This results in recording segments being lost with no apparent cause in the logs (even when set to debug).
This PR adds a warning log entry when discarding segments, in the same way as discarding corrupted segments

* Add explanatory warning and properly format cache_path warning

* lint fixes

Co-authored-by: Blake Blackshear <blakeb@blakeshome.com>
2022-07-19 07:24:44 -05:00
Blake Blackshear
bc078fcc88 remove deprecated max_seconds config option 2022-06-01 06:50:09 -05:00
Blake Blackshear
0d5ae9a399 cleanup 2022-05-12 06:29:43 -05:00
Blake Blackshear
d343a658ec batch deletes for massive cleanups 2022-05-12 06:29:43 -05:00
Blake Blackshear
0ff4acd59c remove invalid warning 2022-02-18 21:18:26 -06:00
Blake Blackshear
ac30091258 note for future 2022-02-18 21:18:26 -06:00
Blake Blackshear
329e5f8f91 invert active_count logic 2022-02-18 21:18:26 -06:00
Blake Blackshear
f512af2563 make expire interval configurable for users wanting to minimize i/o 2022-02-18 21:18:26 -06:00
Blake Blackshear
675f21e23a if recording not on disk, delete from db and return 2022-02-18 21:18:26 -06:00
Blake Blackshear
e6292c719d disable disk sync on startup 2022-02-18 21:18:26 -06:00
Blake Blackshear
ec3c15e4a7 expire overlapping segments based on mode 2022-02-18 21:18:26 -06:00
Blake Blackshear
188b202836 store objects and motion counts in the db 2022-02-18 21:18:26 -06:00
Blake Blackshear
5b164b72dc refactor segment stats logic 2022-02-18 21:18:26 -06:00
Blake Blackshear
dcf65febba switch to retain config instead of retain_days 2022-02-18 21:18:26 -06:00
Blake Blackshear
ef214fb80a retain frame data for recording maintenance 2022-02-18 21:18:26 -06:00
Blake Blackshear
689af4ff87 sync recordings with disk once on startup 2022-02-18 21:18:26 -06:00
Blake Blackshear
4ab0927de8 no need to expire recordings every minute 2022-02-18 21:18:26 -06:00
Blake Blackshear
014e6fc909 ensure cache copies when events have ended 2022-02-18 21:18:26 -06:00
Blake Blackshear
6832575643 cleanup missing files from database once per hour 2022-02-18 21:18:26 -06:00
Blake Blackshear
07ad2d97b1 handle missing file edge case 2022-02-18 21:18:26 -06:00
Blake Blackshear
173eaabddf add duration to cache 2022-02-18 21:18:26 -06:00
Blake Blackshear
a748b70da1 avoid running ffprobe for each segment multiple times 2022-02-18 21:18:26 -06:00
Blake Blackshear
8eabe5dd41 warn if no wait time 2022-02-18 21:18:26 -06:00
Blake Blackshear
114415b5e1 keep 5 segments in cache 2022-02-18 21:18:26 -06:00
Blake Blackshear
ba55b5a6db better cache handling 2022-02-18 21:18:26 -06:00
Blake Blackshear
7533f2a8ab avoid proactive messages with retain_days 0 and handle first pass 2022-02-18 21:18:26 -06:00
Blake Blackshear
a0235b7da4 more robust cache management 2022-02-18 21:18:26 -06:00
Blake Blackshear
34bc6a6457 error handling for the recording maintainer 2022-02-18 21:18:26 -06:00
Blake Blackshear
2f2329ba44 only save recordings when an event is in progress 2022-02-18 21:18:26 -06:00
Blake Blackshear
4cf55ad8e2 Revert switch to mpegts format and audio default 2021-10-23 08:21:15 -05:00
Blake Blackshear
b0e0abe385 improve performance of cache loop 2021-10-23 08:21:15 -05:00
Blake Blackshear
4916e1cd1d hide banner for ffmpeg conversion 2021-10-23 08:21:15 -05:00
Blake Blackshear
cd87f3e6f4 fix old style recording cleanup 2021-10-23 08:21:15 -05:00
Blake Blackshear
b2e05afff2 prevent oldest recording from being deleted 2021-10-15 21:56:03 -05:00
Blake Blackshear
c809494c98 switch to mpegts format for cache and create mp4 with faststart 2021-10-15 21:08:43 -05:00
Blake Blackshear
ef82c5c691 fix expiration when event spans the exire date 2021-10-15 07:30:55 -05:00
Blake Blackshear
7bf7365f6c better log message when corrupt segment detected 2021-10-02 06:58:29 -05:00
Blake Blackshear
5a8016de87 simplify logic and fix wrong segments expiring (fixes #1779) 2021-09-17 17:15:16 -05:00
Blake Blackshear
bc350644bd make expiration of deleted camera footage faster 2021-09-17 17:12:03 -05:00
Blake Blackshear
6c28613def moar speed 2021-09-03 06:31:06 -05:00
Blake Blackshear
56480dc1ef bulk delete recordings 2021-09-02 20:40:38 -05:00
Blake Blackshear
8e1c15291d optimize checking recordings for events
sorts events and recordings so you can avoid a cartesian product of checking all events against all recordings
2021-09-02 08:24:53 -05:00
Blake Blackshear
a1e52c51b1 dont expire events in two places 2021-09-01 07:06:52 -05:00
Blake Blackshear
8cc834633e reduce db queries for recording cleanup 2021-09-01 06:44:05 -05:00
Blake Blackshear
7d65c05994 properly handle scenario with no recordings 2021-08-30 06:58:50 -05:00
Blake Blackshear
6ccff71408 handle missing camera names 2021-08-28 07:43:51 -05:00