Add new features, fix .env setup, fix permissions and UID conflicts (#50

)

* add support for Flaresolverr,
automatic timezone detection

* add scripts for cleaning up old users,
and updating the containers

* fix audiobookshelf, re-add jacket support

* add optional folder and permission generation
also use a previously unused function to clean up code

* fix python UID for tautulli and jellyfin

* remove empty file

* docker-compose sample that uses env variables

new .env sample, fixed setup

* change default install directory to current dir

* fix update_containers

* remove obsolete version line

* fix directory ownership

* update README

* address nitpicks

* Update check_running.yml

Updated auto-check workflow to also copy docker-compose sample file, updated syntax to docker compose without "-"

.env.sample

@@ -1,8 +1,34 @@
 # If you're not sure what timezone you should fill in, you can look at the following list:
 # https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
 # Use the column that says "TZ database name".
+# To find your system's preconfigured timezone run: cat /etc/timezone
 TIMEZONE=Europe/Amsterdam
+
 # The PleX claim token is used to connect your server to your account. It can be obtained at
 # https://www.plex.tv/claim/ and is valid for 4 minutes after generating the token.
 PLEX_CLAIM=
-ROOT_DIR=/
+
+# Choose your root directory for keeping all your container configs and data.
+ROOT_DIR=
+
+# USER IDs for each container that requires their own user. The default values will most likely suffice, but in case one of these IDs is already in use on your system it will have to be changed. Any value should work as long as there aren't conflicts with other users.
+# If you've already set up the containers and want to change user IDs after the fact, you have to remove the old users first and re-set permissions. First run remove_old_users.sh and then the steps for manual installation.
+# If you're doing a manual installation make sure to first change the IDs in this file, and then run setup.sh to set up the directory structure and permissions before you start the docker containers.
+MEDIACENTER_GID=13000
+SONARR_UID=13001
+RADARR_UID=13002
+LIDARR_UID=13003
+READARR_UID=13004
+MYLAR_UID=13005
+AUDIOBOOKSHELF_UID=13014
+BAZARR_UID=13013
+PROWLARR_UID=13006
+QBITTORRENT_UID=13007
+JACKETT_UID=13008
+PLEX_UID=13010
+OVERSEERR_UID=13009
+SABNZBD_UID=13011
+JELLYSEERR_UID=13012
+
+# The user used to run docker-compose. To find this run: id -u
+UID=1000 

.github/workflows/check_running.yml

@@ -18,9 +18,12 @@ jobs:
 
       - name: Copy environment file
         run: cp .env.sample .env
+
+      - name: Copy docker-compose file
+        run: cp docker-compose.yml.sample docker-compose.yml
 
       - name: Run setup.sh
         run: ./setup.sh
 
-      - name: Run docker-compose
-        run: docker-compose up -d
+      - name: Run docker compose
+        run: docker compose up -d

README.md

@@ -3,7 +3,8 @@
 
 Ezarr is a project built to make it EZ to deploy a Servarr mediacenter on an Ubuntu server. The
 badge above means that the shell script and docker-compose file in this repository at least *don't
-crash*. It doesn't necessarily mean it will run well on your system ;) It features:
+crash*. It doesn't necessarily mean it will run well on your system ;) 
+It's set up to follow the [TRaSH guidelines](https://trash-guides.info/Hardlinks/How-to-setup-for/Docker/) so it should at least perform optimally. It features:
 - [Sonarr](https://sonarr.tv/) is an application to manage TV shows. It is capable of keeping track
   of what you'd like to watch, at what quality, in which language and more, and can find a place to
   download this if connected to Prowlarr and qBittorrent. It can also reorganize the media you
@@ -24,6 +25,7 @@ crash*. It doesn't necessarily mean it will run well on your system ;) It featur
   less than 1. Less than 1 means that you upload less than you download. Not only is this
   unfriendly towards your fellow users, but it can also get you banned from certain indexers.
 - [Jackett](https://github.com/Jackett/Jackett) is an alternative to Prowlarr. 
+- [FlareSolverr](https://github.com/FlareSolverr/FlareSolverr) is a proxy server to bypass Cloudflare and DDoS-GUARD protection.
 - [qBittorrent](https://www.qbittorrent.org/) can download torrents and provides a bunch more
   features for management.
 - [SABnzbd](https://sabnzbd.org/) can download nzb's
@@ -39,35 +41,63 @@ crash*. It doesn't necessarily mean it will run well on your system ;) It featur
    tool.
 - [Jellyseerr](https://github.com/Fallenbagel/jellyseerr) is like Overseerr, but for Jellyfin.
 
+## Requirements
+Currently this script only works on Linux. There is a chance that the sample docker compose file will work on Windows, although untested.
+The only requirements other than that are **Python 3** and **docker** with **docker-compose-v2**.
+While this script _may_ work on docker-compose-v1 it's made to be and highly recommended to be run using v2.
+The easiest way to install these dependencies on Ubuntu and other Debian-based distors is by running:
+```
+sudo apt-get install python3 docker.io docker-compose-v2
+```
+For other Linux distros you may have to use a different package manager or download directly from docker's website.
+
 ## Using
 ### Using the CLI
 To make things easier, a CLI has been developed. First, clone the repository in a directory of your
-choosing. You can run it by entering `python main.py` and the CLI will guide you through the
-process. Please take a look at [important notes](#important-notes) before you continue.
+choosing. You can run it by entering `python3 main.py` and the CLI will guide you through the
+process. This is the recommended method if you're setting this up for the first time on a new system. 
+Please take a look at [important notes](#important-notes) before you continue. 
+**NOTE: This script will create users for each container with IDs ranging from 13001 to 13014. 
+If you want to choose your own IDs (or some of them are occupied) you have to go through the manual install.**
 
 ### Manually
-1. To get started, clone the repository in a directory of your choosing. **Note: this will be where
-   your installation and media will be as well, so think about this a bit.**
+If you're installing this for the first time simply follow these steps. 
+If you're coming from an older version or reinstalling with different IDs, run `remove_old_users.sh` to clean up old users and then follow these steps.
+1. To get started, clone the repository in a directory of your choosing. `git clone https://github.com/Luctia/ezarr.git`
 2. Copy `.env.sample` to a real `.env` by running `$ cp .env.sample .env`.
-3. Set the environment variables to your liking. Note that `ROOT_DIR` should be the directory you
-   have cloned this in.
-4. Run `setup.sh` as superuser. This will set up your users, a system of directories, ensure
-   permissions are set correctly and sets some more environment variables for docker compose.
-5. Take a look at the `docker-compose.yml` file. If there are services you would like to ignore
+3. Set the environment variables to your liking. Pay special attention `ROOT_DIR` as this is where everything is going to be stored in. 
+   The path in this value needs to be **absolute**. If you leave it empty it's going to install in the directory the .env file is currently in.
+   `UID` should be set to the ID of the user that you want to run docker with. You can find this by running `id -u` from that user's shell.
+4. Run `setup.sh` as superuser. This will set up your users, a system of directories and ensure permissions are set correctly.
+5. Copy `docker-compose.yml.sample` to a real `docker-compose.yml` by running `$ cp docker-compose.yml.sample docker-compose.yml`.
+6. Take a look at the `docker-compose.yml` file. If there are services you would like to ignore
    (for example, running PleX and Jellyfin at the same time is a bit unusual), you can comment them
-   out by placing `#` in front of the lines. This ensures they are ignored by Docker compose.
-6. Run `docker compose up`.
+   out by placing `#` in front of the lines. This ensures they are ignored by Docker compose. 
+   Double check that your .env file is set up properly.
+7. Run `docker compose up -d` to start the containers. If it complains about permissions run the following commands to add your current user to the docker group and apply changes:
+    ```
+    sudo groupadd docker
+    sudo usermod -aG docker $USER
+    newgrp docker
+    ```
+    If it still doesn't work reboot your system.
 
 That's it! Your containers are now up and you can continue to set up the settings in them. Please
 take a look at [important notes](#important-notes) before you continue.
 
 ## Important notes
+- You probably shouldn't run the python script as root. Ideally you should create a brand new user that's just for these services, but any regular user will do.
+  It will need your password for `sudo` to set up the permissions and folder structures, but you shouldn't run it *as* root.
+- If you already used this script previously and want to clean up old users, run `remove_old_users.sh`.
+  This is also recommended if you are updating from an earlier version of this script, since there were previously some conflicts in user IDs.
+- It is recommended to restart your system after script completion, so that newly created users and groups can be loaded properly.
 - When linking one service to another, remember to use the container name instead of `localhost`.
 - Please set the settings of the -arr containers as soon as possible to the following (use
   advanced):
   - Media management:
     - Use hardlinks instead of Copy: `true`
     - Root folder: `/data/media/` and then tv, movies or music depending on service
+  - qBittorrent ships with a default username `admin` and a one-time password that can be viewed by running `docker logs qbittorrent`.
   - Make sure to set a username and password for all servarr services and qBittorrent!
 - In qBittorrent, after connecting it to the -arr services, you can indicate it should move
   torrents in certain categories to certain directories, like torrents in the `radarr` category
@@ -76,6 +106,18 @@ take a look at [important notes](#important-notes) before you continue.
   field: `chmod -R 775 "%F/"`.
 - You'll have to add indexers in Prowlarr by hand. Use Prowlarrs settings to connect it to the
   other -arr apps.
+
+### IMPORTANT IF USING NFS SHARES
+- NFS shares' permissions are mapped by user IDs. If you want to access a file as a client, your user ID needs to match the user ID of the owner (or group) of that file on the NFS server. 
+Note that if you are a group member (and not the owner), having matching group IDs won't be enough, there also needs to be a corresponding user on the NFS server. The easiest way to make sure
+the users and groups are set up on both sides correctly is to run `setup.sh` on both your NFS server and your client. 
+On your server:
+- Copy `.env` and `setup.sh` to your NFS server.
+- You may have to adjust `.env` so that `ROOT_DIR` reflects where it will be stored on your server, which is most likely different from the mapped location on the client.
+- Make sure that the `.env` file is not a .sample. Run `setup.sh`.
+- Now follow all the same steps but on your client machine. Always double-check that `.env` is set correctly, especially `ROOT_DIR`.
+You don't have to do this on your server first but it's recommended. If you are running this script on the client **make sure that you temporarily enable -no-root-squash on your NFS server**, 
+as the script needs superuser privileges to run and by default on NFS the root user is mapped to nowhere to prevent abuse.
   
 ### SABnzbd External internet access denied message
 When you're trying to access SABnzbd the first time you'll come across the message `External
@@ -92,7 +134,10 @@ official SABnzbd website.
 ## FAQ
 
 ### How to update containers
-If you'd like to update containers, you can move to the directory of your `docker-compose.yml` file
+There is an `update_containers.sh` script that takes care of this. Simply run it and it updates
+all containers and removes old images. If you want to keep them, simply comment out the last line of the script.
+It's essentially the following steps but automated:
+If you'd like to it manually, go to the directory of your `docker-compose.yml` file
 and run `(sudo) docker compose pull`. This pulls the newest versions of all images (blueprints for
 containers) listed in the `docker-compose.yml` file. Then, you can run `(sudo) docker compose up
 -d`. This will deploy the new versions without losing uptime. Afterwards, you can run `(sudo)

container_configs.py

@@ -1,3 +1,5 @@
+import os 
+
 class ContainerConfig:
     def __init__(self,
                  root_dir,
@@ -15,6 +17,7 @@ def __init__(self,
         self.comic_dir = root_dir + '/media/comics'
         self.torrent_dir = root_dir + '/data/torrents'
         self.usenet_dir = root_dir + '/data/usenet'
+        self.UID = os.popen('id -u').read().rstrip('\n')
 
     def plex(self):
         return (
@@ -41,7 +44,7 @@ def tautulli(self):
             '    depends_on:\n'
             '      - plex\n'
             '    environment:\n'
-            '      - PUID=${UID}\n'
+            '      - PUID='+ self.UID + '\n'
             '      - PGID=13000\n'
             '      - TZ=' + self.timezone + '\n'
             '    volumes:\n'
@@ -57,7 +60,7 @@ def jellyfin(self):
             '    image: lscr.io/linuxserver/jellyfin:latest\n'
             '    container_name: jellyfin\n'
             '    environment:\n'
-            '      - PUID=${UID}\n'
+            '      - PUID='+ self.UID + '\n'
             '      - PGID=13000\n'
             '      - UMASK=002\n'
             '      - TZ=' + self.timezone + '\n'
@@ -181,13 +184,13 @@ def audiobookshelf(self):
             '    image: ghcr.io/advplyr/audiobookshelf:latest\n'
             '    container_name: audiobookshelf\n'
             '    environment:\n'
-            '      - AUDIOBOOKSHELF_UID=13009\n'
-            '      - AUDIOBOOKSHELF_GID=13000\n'
+            '      - user=13014:13000\n'
+            '      - TZ=' + self.timezone + '\n'
             '    volumes:\n'
             '      - ' + self.config_dir + '/audiobookshelf:/config\n'
-            '      - ' + self.root_dir + '/data/audiobooks:/audiobooks\n'
-            '      - ' + self.root_dir + '/data/podcasts:/podcasts\n'
-            '      - ' + self.root_dir + '/data/metadata:/metadata\n'
+            '      - ' + self.root_dir + '/data/media/audiobooks:/audiobooks\n'
+            '      - ' + self.root_dir + '/data/media/podcasts:/podcasts\n'
+            '      - ' + self.root_dir + '/data/media/audiobookshelf-metadata:/metadata\n'
             '    ports:\n'
             '      - "13378:80"\n'
             '    restart: unless-stopped\n\n'
@@ -282,3 +285,36 @@ def sabnzbd(self):
             '      - "8081:8080"\n'
             '    restart: unless-stopped\n\n'
         )
+
+    def jackett(self):
+        return (
+            '  jackett:\n'
+            '    image: lscr.io/linuxserver/jackett:latest\n'
+            '    container_name: jackett\n'
+            '    environment:\n'
+            '      - PUID=13008\n'
+            '      - PGID=13000\n'
+            '      - UMASK=002\n'
+            '      - TZ=' + self.timezone + '\n'
+            '    volumes:\n'
+            '      - ' + self.config_dir + '/jackett-config:/config\n'
+            '    ports:\n'
+            '      - 9117:9117\n'
+            '    restart: unless-stopped\n\n'
+        )
+
+    def flaresolverr(self):
+        return (
+            '  flaresolverr:\n'
+            '    image: ghcr.io/flaresolverr/flaresolverr:latest\n'
+            '    container_name: flaresolverr\n'
+            '    environment:\n'
+            '      - LOG_LEVEL=${LOG_LEVEL:-info}\n'
+            '      - LOG_HTML=${LOG_HTML:-false}\n'
+            '      - CAPTCHA_SOLVER=${CAPTCHA_SOLVER:-none}\n'
+            '      - TZ=' + self.timezone + '\n'
+            '    ports:\n'
+            '      - "8191:8191"\n'
+            '    restart: unless-stopped\n\n'
+        )
+