Commit Graph

73 Commits

Author SHA1 Message Date
Nicolas Mowen
cf2466c8c1
Fix recordings storage (#5031)
* Fix recordings storage

* Update record.py

* Update record.py

* Formatting
2023-01-12 05:53:38 -06:00
Nicolas Mowen
369299315f
Optimize nginx & recordings (#4688)
* Add segment duration metadata

* Use faststart only for kept segments

* Add more options for performance

* Build nginx locally

* Build nginx in dockerfile and enable threaded vod handling

* Use DASH instead of hls

* Allow player to continue on error

* Undo DASH change

* Fix typo

* Correct log

* Fix bad comments

* Fix indentation

* Preload stream

* remove unused

* Fix spacing

* Fix tabs / sspaces

* Retab

* More cleanup
2022-12-17 17:53:34 -06:00
Nicolas Mowen
037f3761e7
Use UTC for recordings (#4656)
* Write files in UTC and update folder structure to not conflict

* Add timezone arg for events summary

* Fixes for timezone in calls

* Use timezone for recording and recordings summary endpoints

* Fix sqlite parsing

* Fix sqlite parsing

* Fix recordings summary with timezone

* Fix

* Formatting

* Add pytz

* Fix default timezone

* Add note about times being displayed in localtime

* Improve timezone wording and show actual timezone

* Add alternate endpoint to cover existing usecase to avoid breaking change

* Formatting
2022-12-11 07:45:32 -06:00
Nicolas Mowen
2a36a1b980
Catch case where segment is bad length (#4228)
* Catch case where segment is bad length

* Log ffprobe error code and error
2022-11-02 06:37:27 -05:00
Nicolas Mowen
b4d4adb75b
Limit recording retention to available storage (#3942)
* Add field and migration for segment size

* Store the segment size in db

* Add comment

* Add default

* Fix size parsing

* Include segment size in recordings endpoint

* Start adding storage maintainer

* Add storage maintainer and calculate average sizes

* Update comment

* Store segment and hour avg sizes per camera

* Formatting

* Keep track of total segment and hour averages

* Remove unused files

* Cleanup 2 hours of recordings at a time

* Formatting

* Fix bug

* Round segment size

* Cleanup some comments

* Handle case where segments are not deleted on initial run or is only retained segments

* Improve cleanup log

* Formatting

* Fix typo and improve logging

* Catch case where no recordings exist for camera

* Specifically define sort

* Handle edge case for cameras that only record part time

* Increase definition of part time recorder

* Remove warning about not supported storage based retention

* Add note about storage based retention to recording docs

* Add tests for storage maintenance calculation and cleanup

* Format tests

* Don't run for a camera with no recording segments

* Get size of file from cache

* Rework camera stats to be more efficient

* Remove total and other inefficencies

* Rewrite storage cleanup logic to be much more efficient

* Fix existing tests

* Fix bugs from tests

* Add another test

* Improve logging

* Formatting

* Set back correct loop time

* Update name

* Update comment

* Only include segments that have a nonzero size

* Catch case where camera has 0 nonzero segment durations

* Add test to cover zero bandwidth migration case

* Fix test

* Incorrect boolean logic

* Formatting

* Explicity re-define iterator
2022-10-09 06:28:26 -05:00
Nicolas Mowen
65c211bb6d
Catch ffprobe empty stdout (#4016) 2022-10-04 19:56:04 -05:00
Blake Blackshear
8669c29e3d
don't overwrite segments that already exist (#3995) 2022-10-01 18:11:29 -05:00
Blake Blackshear
3c46a33992 revert false warning messages 2022-07-20 06:55:06 -05:00
deviant77
ed1897db71
Add log message when discarding recording segments in cache (#3439)
* Add log message when discarding recording segments in cache

Currently Frigate silently discards recording segments in cache if there's more than "keep_count" for a camera, which can happen on high load/busy/slow systems.
This results in recording segments being lost with no apparent cause in the logs (even when set to debug).
This PR adds a warning log entry when discarding segments, in the same way as discarding corrupted segments

* Add explanatory warning and properly format cache_path warning

* lint fixes

Co-authored-by: Blake Blackshear <blakeb@blakeshome.com>
2022-07-19 07:24:44 -05:00
Blake Blackshear
bc078fcc88 remove deprecated max_seconds config option 2022-06-01 06:50:09 -05:00
Blake Blackshear
0d5ae9a399 cleanup 2022-05-12 06:29:43 -05:00
Blake Blackshear
d343a658ec batch deletes for massive cleanups 2022-05-12 06:29:43 -05:00
Blake Blackshear
0ff4acd59c remove invalid warning 2022-02-18 21:18:26 -06:00
Blake Blackshear
ac30091258 note for future 2022-02-18 21:18:26 -06:00
Blake Blackshear
329e5f8f91 invert active_count logic 2022-02-18 21:18:26 -06:00
Blake Blackshear
f512af2563 make expire interval configurable for users wanting to minimize i/o 2022-02-18 21:18:26 -06:00
Blake Blackshear
675f21e23a if recording not on disk, delete from db and return 2022-02-18 21:18:26 -06:00
Blake Blackshear
e6292c719d disable disk sync on startup 2022-02-18 21:18:26 -06:00
Blake Blackshear
ec3c15e4a7 expire overlapping segments based on mode 2022-02-18 21:18:26 -06:00
Blake Blackshear
188b202836 store objects and motion counts in the db 2022-02-18 21:18:26 -06:00
Blake Blackshear
5b164b72dc refactor segment stats logic 2022-02-18 21:18:26 -06:00
Blake Blackshear
dcf65febba switch to retain config instead of retain_days 2022-02-18 21:18:26 -06:00
Blake Blackshear
ef214fb80a retain frame data for recording maintenance 2022-02-18 21:18:26 -06:00
Blake Blackshear
689af4ff87 sync recordings with disk once on startup 2022-02-18 21:18:26 -06:00
Blake Blackshear
4ab0927de8 no need to expire recordings every minute 2022-02-18 21:18:26 -06:00
Blake Blackshear
014e6fc909 ensure cache copies when events have ended 2022-02-18 21:18:26 -06:00
Blake Blackshear
6832575643 cleanup missing files from database once per hour 2022-02-18 21:18:26 -06:00
Blake Blackshear
07ad2d97b1 handle missing file edge case 2022-02-18 21:18:26 -06:00
Blake Blackshear
173eaabddf add duration to cache 2022-02-18 21:18:26 -06:00
Blake Blackshear
a748b70da1 avoid running ffprobe for each segment multiple times 2022-02-18 21:18:26 -06:00
Blake Blackshear
8eabe5dd41 warn if no wait time 2022-02-18 21:18:26 -06:00
Blake Blackshear
114415b5e1 keep 5 segments in cache 2022-02-18 21:18:26 -06:00
Blake Blackshear
ba55b5a6db better cache handling 2022-02-18 21:18:26 -06:00
Blake Blackshear
7533f2a8ab avoid proactive messages with retain_days 0 and handle first pass 2022-02-18 21:18:26 -06:00
Blake Blackshear
a0235b7da4 more robust cache management 2022-02-18 21:18:26 -06:00
Blake Blackshear
34bc6a6457 error handling for the recording maintainer 2022-02-18 21:18:26 -06:00
Blake Blackshear
2f2329ba44 only save recordings when an event is in progress 2022-02-18 21:18:26 -06:00
Blake Blackshear
4cf55ad8e2 Revert switch to mpegts format and audio default 2021-10-23 08:21:15 -05:00
Blake Blackshear
b0e0abe385 improve performance of cache loop 2021-10-23 08:21:15 -05:00
Blake Blackshear
4916e1cd1d hide banner for ffmpeg conversion 2021-10-23 08:21:15 -05:00
Blake Blackshear
cd87f3e6f4 fix old style recording cleanup 2021-10-23 08:21:15 -05:00
Blake Blackshear
b2e05afff2 prevent oldest recording from being deleted 2021-10-15 21:56:03 -05:00
Blake Blackshear
c809494c98 switch to mpegts format for cache and create mp4 with faststart 2021-10-15 21:08:43 -05:00
Blake Blackshear
ef82c5c691 fix expiration when event spans the exire date 2021-10-15 07:30:55 -05:00
Blake Blackshear
7bf7365f6c better log message when corrupt segment detected 2021-10-02 06:58:29 -05:00
Blake Blackshear
5a8016de87 simplify logic and fix wrong segments expiring (fixes #1779) 2021-09-17 17:15:16 -05:00
Blake Blackshear
bc350644bd make expiration of deleted camera footage faster 2021-09-17 17:12:03 -05:00
Blake Blackshear
6c28613def moar speed 2021-09-03 06:31:06 -05:00
Blake Blackshear
56480dc1ef bulk delete recordings 2021-09-02 20:40:38 -05:00
Blake Blackshear
8e1c15291d optimize checking recordings for events
sorts events and recordings so you can avoid a cartesian product of checking all events against all recordings
2021-09-02 08:24:53 -05:00