This patch implements [X-Accel](https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/)
redirect headers as an optional way for offloading static file delivery
from Express to Nginx, which is far better optimized for static file
delivery.
This provides a really easy to configure way for getting a huge
performance boost over delivering all files through Audiobookshelf.
How it works
------------
The way this works is basically that Audiobookshelf gets an HTTP request
for delivering a static file (let's say an audiobook). It will first
check the user is authorized and then convert the API path to a local
file path.
Now, instead of reading and delivering the file, Audiobookshelf will
return just the HTTP header with an additional `X-Accel-Redirect`
pointing to the file location on the file syste.
This header is picked up by Nginx which will then deliver the file.
Configuration
-------------
The configuration for this is very simple. You need to run Nginx as
reverse proxy and it must have access to your Audiobookshelf data
folder.
You then configure Audiobookshelf to use X-Accel by setting
`USE_X_ACCEL=/protected`. The path is the internal redirect path used by
Nginx.
In the Nginx configuration you then configure this location and map it
to the storage area to serve like this:
```
location /protected/ {
internal;
alias /;
}
```
That's all.
Impact
------
I just did a very simple performance test, downloading a 1170620819
bytes large audiobook file from another machine on the same network
like this, using `time -p` to measure how log the process took:
```sh
URL='https://url to audiobook…'
for i in `seq 1 50`
do
echo "$i"
curl -s -o /dev/null "${URL}"
done
```
This sequential test with 50 iterations and without x-accel resulted in:
```
real 413.42
user 197.11
sys 82.04
```
That is an average download speed of about 1080 MBit/s.
With X-Accel enabled, serving the files through Nginx, the same test
yielded the following results:
```
real 200.37
user 86.95
sys 29.79
```
That is an average download speed of about 2229 MBit/s, more than
doubling the previous speed.
I have also run the same test with 4 parallel processes and 25 downloads
each. Without x-accel, that test resulted in:
```
real 364.89
user 273.09
sys 112.75
```
That is an average speed of about 2448 MBit/s.
With X-Accel enabled, the parallel test also shows a significant
speedup:
```
real 167.19
user 195.62
sys 78.61
```
That is an average speed of about 5342 MBit/s.
While doing that, I also peaked at the system load which was a bit lower
when using X-Accel. Even though the system was delivering far more data.
But I just looked at the `load1` values and did not build a proper test
for that. That means, I cant provide any definitive data.
Supported Media
---------------
The current implementation works for audio files and book covers. There
are other media files which would benefit from this mechanism like feed
covers or author pictures.
But that's something for a future developer ;-)
This patch slightly changes the behavior of the `AUDIOBOOKSHELF_UID` and
`AUDIOBOOKSHELF_GID` options. Instead of defining a default user and
group, trying to modify files and silently failing if the filesystem
mode cannot be changed, this patch will just skip the entire process in
the first place.
If these options are defined, Audiobookshelf should behave exactly as
before. If they are not defined, Audiobookshelf will now cause fewer
file modifications (or less failures when trying to modify files).
If this patch gets applied, it should probably be highlighted in the
release notes. This usually shouldn't cause problems for migrations
since the Docker guides explicitly configure the options and the
package installations do not seem to use this at all, but there is still
a change that it will and users should be aware of that.
If a problem arises, users can easily fix the problem by either setting
the permissions once manually to the audiobookshelf user or by simply
defining the `AUDIOBOOKSHELF_UID/GID` options.
Sometimes, a book belongs to more than one series. If you listen to and
finish such a book, Audiobookshelf will list the next book in “Continue
Series” twice, right next to each other. That is not helpful.
This patch fixes the problem by not adding books to the list if they are
already in the list.
This patch changes the books displayed in “Continue Series”, avoiding
books if another book from the series is played back right now. This
prevents Audiobookshelf suggesting books to which users will not listen
to because they are still listening to the last one.
Once a book is finished, the next book in the series will pop still be
suggested to the user.
This fixes #1382
The Audiobookshelf logs sometimes contain information about the source
of the log statement, but sometimes they don't This really depends on
developers adding these information to the log messages.
But even then, the information is usually just a hint about the module
logging this, like `[Db]` or [Watcher]`, and finding the exact line can
be hard.
This patch automatically adds the source of the log statement to the
logs. This means if someone calls `Logger.info(…)` in line `22` of
`foo.js`, the log statement will contain this file and line:
```
[2023-01-05 19:04:12[ (LogManager.js:85:18) DEBUG: Daily Log file found 2023-01-05.txt
[2023-01-05 19:04:12] (LogManager.js:59:12) INFO: [LogManager] Init current daily log filename: 2023-01-05.txt
```
This should make it much easier to identify the code where the log
statement originated from.
Long-term, this also means that we can probably remove the manually set
identifiers contained in the log messages, like the `[LogManager]` in
the example above.
If the scanner detects new files with a path containing part of the name
of an already existing library item, the new item will incorrectly be
detected as being a parent directory of the already existing item and
the import will be aborted.
You can follow these steps to reproduce the issue:
```
❯ mkdir audiobooks/author/
❯ mv title\ 10 audiobooks/author
[2022-12-18 22:14:12] DEBUG: [Watcher] File Added /home/lars/dev/audiobookshelf/audiobooks/author/title 10/dictaphone.mp3
[2022-12-18 22:14:16] DEBUG: [DB] Library Items inserted 1
❯ mv title\ 1 audiobooks/author
[2022-12-18 22:15:03] DEBUG: [Watcher] File Added /home/lars/dev/audiobookshelf/audiobooks/author/title 1/dictaphone.mp3
[2022-12-18 22:15:07] WARN: [Scanner] Files were modified in a parent directory of a library item "title 10" - ignoring
```
Since `'title 10'.startsWith('title 1')` is `true`, the current code
makes this false assumption.
This patch fixes the issue by requiring a path separator to be part of
the matching path. This should ensure that only true parent directories
are detected.
This patch requires audiobookshelf to always use Unix file separators.
But that shouldn't be a problem since audiobookshelf always seems to use
these kinds of separators. Even on Windows.
The API for Google Books will return HTTP image URLs when matiching any
books using it as a search provider. In a secure environment, this
causes browser warnings.
All Google image links support HTTPS and we can safely switch to HTTOS
to avoid these warnings.
If no progress had been set before, setting `startedAt` did not work and
it would always been set to `finishedAt` or `Date.now()`. Settings this
if any progress had already been recorded did work.
This fixes the problem so that setting `startedAt` it properly works in
both cases.
This patch updates the batch progress update endpoint to respond with a
`400 Bad Request` instead of a `500 Internal Server Error` if a user
sends an invalid request with no body. This is a user error after all.
```
❯ curl -i -X PATCH \
'http://127.0.0.1:3333/api/me/progress/batch/update' \
-H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5Q_MoRptP0oI' \
-H 'Content-Type: application/json'
HTTP/1.1 400 Bad Request
…
Missing request payload
```
This patch is a minor extension to the update progress and batch update
progress API and allows you to specify `finishedAt` and `startedAt` when
updating a progress.
If not specified, both values are still automatically set to the current
time. If just `finishedAt` is specified, `startedAt` is set to the same
value.
Example API request:
```
❯ curl -i -X PATCH \
'http://127.0.0.1:3333/api/me/progress/li_ywupqxw5d22adcadpa' \
-H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJyb290IiwidXNlcm5hbWUiOiJyb290IiwiaWF0IjoxNjY4OTYxNjAxfQ._NbilCoFy_hfoqy7uvbV4E_0X6qgLYapQ_MoRptP0oI' \
-H 'Content-Type: application/json' \
--data-raw '{"isFinished":true, "finishedAt": 1668556852000, "startedAt": 1668056852000}'
```
This patch drops the log level for logging requests to non-existing
feeds from error to debug. The reasoning behind this is that this is a
client error and a client error is returned and can be handled by the
client.
Otherwise anyone can easily spam the logs with error messages by just
requesting non-existing feeds.
