diff --git a/docs/docs/troubleshooting/recordings.md b/docs/docs/troubleshooting/recordings.md index d1d2dc7fe..611ba45e2 100644 --- a/docs/docs/troubleshooting/recordings.md +++ b/docs/docs/troubleshooting/recordings.md @@ -5,7 +5,7 @@ title: Troubleshooting Recordings ### WARNING : Unable to keep up with recording segments in cache for camera. Keeping the 5 most recent segments out of 6 and discarding the rest... -This error can be caused by a number of different issues. The first step in troubleshooting is to enable debug logging for recording, this will enable logging showing how long it takes for recordings to be moved from RAM cache to the disk. +This error can be caused by a number of different issues. The first step in troubleshooting is to enable debug logging for recording. This will enable logging showing how long it takes for recordings to be moved from RAM cache to the disk. ```yaml logger: @@ -25,6 +25,41 @@ It is important to let this run until the errors begin to happen, to confirm tha If the storage is too slow to keep up with the recordings then the maintainer will fall behind and purge the oldest recordings to ensure the cache does not fill up causing a crash. In this case it is important to diagnose why the copy times are slow. +##### Check RAM, swap, cache utilization, and disk utilization + +If CPU, RAM, disk throughput, or bus I/O is insufficient, nothing inside frigate will help. It is important to review each aspect of available system resources. + +On linux, some helpful tools/commands in diagnosing would be: + +- docker stats +- htop +- iotop -o +- iostat -sxy --human 1 1 +- vmstat 1 + +On modern linux kernels, the system will utilize some swap if enabled. Setting vm.swappiness=1 no longer means that the kernel will only swap in order to avoid OOM. To prevent any swapping inside a container, set allocations memory and memory+swap to be the same and disable swapping by setting the following docker/podman run parameters: + +**Compose example** +```yaml +version: "3.9" +services: + frigate: + ... + mem_swappiness: 0 + memswap_limit: + deploy: + resources: + limits: + memory: +``` + +**Run command example** +``` +--memory= --memory-swap= --memory-swappiness=0 +``` + +NOTE: These are hard-limits for the container, be sure there is enough headroom above what is shown by `docker stats` for your container. It will immediately halt if it hits ``. In general, running all cache and tmp filespace in RAM is preferable to disk I/O where possible. + ##### Check Storage Type Mounting a network share is a popular option for storing Recordings, but this can lead to reduced copy times and cause problems. Some users have found that using `NFS` instead of `SMB` considerably decreased the copy times and fixed the issue. It is also important to ensure that the network connection between the device running Frigate and the network share is stable and fast.