Merge pull request #1079 from linuxserver/swag-ondemand-initial

Add swag-ondemand mod

Author: quietsy

.assets/loading-page.png

@@ -2,7 +2,7 @@
 
 FROM scratch
 
-LABEL maintainer="username"
+LABEL maintainer="quietsy"
 
 # copy local files
 COPY root/ /

Dockerfile

@@ -1,25 +1,86 @@
-# Rsync - Docker mod for openssh-server
+# On-demand - Docker mod for SWAG
 
-This mod adds rsync to openssh-server, to be installed/updated during container start.
+This mod gives SWAG the ability to start containers on-demand when accessed through SWAG and stop them after a period of inactivity. It takes a few seconds for containers to start on-demand, you'll need to refresh the tab or add a loading page as detailed below.
 
-In openssh-server docker arguments, set an environment variable `DOCKER_MODS=linuxserver/mods:openssh-server-rsync`
+## Setup:
+- In SWAG's docker arguments, set an environment variable `DOCKER_MODS=linuxserver/mods:swag-ondemand` and either add a volume mapping for `/var/run/docker.sock:/var/run/docker.sock:ro`, or set an environment var `DOCKER_HOST=remoteaddress` (read the security considerations below).
+- Add the label `swag_ondemand=enable` to on-demand containers.
+    ```yaml
+    somecontainer:
+        container_name: somecontainer
+        ...
+        labels:
+            - swag_ondemand=enable
+    ```
+- Replace the following line in `/config/nginx/nginx.conf`:
+    ```nginx
+    access_log /config/log/nginx/access.log;
+    ```
+    With:
+    ```nginx
+    log_format main '$remote_addr - $remote_user [$time_local] '
+                    '"$request_method $scheme://$host$request_uri $server_protocol" '
+                    '$status $body_bytes_sent '
+                    '"$http_referer" "$http_user_agent"';
+    access_log /config/log/nginx/access.log main;
+    ```
+- *Optional* - Additional environment variables
+  - `SWAG_ONDEMAND_STOP_THRESHOLD` - duration of inactivity in seconds before stopping on-demand containers, defaults to `600` (10 minutes).
+  - `SWAG_ONDEMAND_CONTAINER_QUERY_SLEEP` - sleep time in seconds between querying containers, defaults to `5.0`.
+  - `SWAG_ONDEMAND_LOG_READER_SLEEP` - sleep time in seconds between log reads, defaults to `1.0`.
 
-If adding multiple mods, enter them in an array separated by `|`, such as `DOCKER_MODS=linuxserver/mods:openssh-server-rsync|linuxserver/mods:openssh-server-mod2`
+### Loading Page:
 
-# Mod creation instructions
+![loading-page](.assets/loading-page.png)
 
-* Fork the repo, create a new branch based on the branch `template`.
-* Edit the `Dockerfile` for the mod. `Dockerfile.complex` is only an example and included for reference; it should be deleted when done.
-* Inspect the `root` folder contents. Edit, add and remove as necessary.
-* After all init scripts and services are created, run `find ./  -path "./.git" -prune -o \( -name "run" -o -name "finish" -o -name "check" \) -not -perm -u=x,g=x,o=x -print -exec chmod +x {} +` to fix permissions.
-* Edit this readme with pertinent info, delete these instructions.
-* Finally edit the `.github/workflows/BuildImage.yml`. Customize the vars for `BASEIMAGE` and `MODNAME`. Set the versioning logic and `MULTI_ARCH` if needed.
-* Ask the team to create a new branch named `<baseimagename>-<modname>`. Baseimage should be the name of the image the mod will be applied to. The new branch will be based on the `template` branch.
-* Submit PR against the branch created by the team.
+Instead of showing a 502 error page, it can display a loading page and auto-refresh once the container is up.
+  
+Add the following `include` to each proxy-conf where you wish to show the loading page inside the `server` section:
+```nginx
+server {
+    ...
+    include /config/nginx/ondemand.conf;
+    ...
+```
+Or set the following label if using `swag-auto-proxy`:
+```yaml
+    somecontainer:
+        container_name: somecontainer
+        ...
+        labels:
+            - swag_server_custom_directive=include /config/nginx/ondemand.conf;
+```
+### Labels:
+- `swag_ondemand=enable` - required for on-demand.
+- `swag_ondemand_urls=https://wake.domain.com,https://app.domain.com/up` - *optional* - overrides the monitored URLs for starting the container on-demand. Defaults to `https://somecontainer.,http://somecontainer.`.
 
+### URLs:
+- Accessed URLs need to start with one of `swag_ondemand_urls` to be matched, for example, setting `swag_ondemand_urls=https://plex.` will apply to `https://plex.domain.com` and `https://plex.domain.com/something`.
+- `swag_ondemand_urls` default to `https://somecontainer.,http://somecontainer.`, for example `https://plex.,http://plex.`.
+- `swag_ondemand_urls` don't need to be valid, it will work as long as it reaches swag and gets logged by nginx under `/config/log/nginx/access.log`.
+- The same URL can be set on multiple containers and all of them will be started when accessing that URL.
 
-## Tips and tricks
+### Logging:
+The log file can be found under `/config/log/ondemand/ondemand.log`.
 
-* Some images have helpers built in, these images are currently:
-    * [Openvscode-server](https://github.com/linuxserver/docker-openvscode-server/pull/10/files)
-    * [Code-server](https://github.com/linuxserver/docker-code-server/pull/95)
+## Security Consideration:
+Mapping the `docker.sock`, especially in a publicly accessible container is a security liability. Since this mod only needs read-only access to the docker api, the recommended method is to proxy the `docker.sock` via a solution like [our docker socket proxy](https://github.com/linuxserver/docker-socket-proxy), limit the access, and set `DOCKER_HOST=` to point to the proxy address.
+
+Here's a sample compose yaml snippet for `linuxserver/docker-socket-proxy`:
+```yaml
+  socket-proxy:
+    image: lscr.io/linuxserver/socket-proxy:latest
+    container_name: socket-proxy
+    environment:
+      - ALLOW_START=1
+      - ALLOW_STOP=1
+      - CONTAINERS=1
+      - POST=0
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+    restart: unless-stopped
+    read_only: true
+    tmpfs:
+      - /run
+```
+Then the env var in SWAG can be set as `DOCKER_HOST=tcp://socket-proxy:2375`. This will allow docker in SWAG to be able to start/stop existing containers, but it won't be allowed to spin up new containers.

Dockerfile.complex

@@ -0,0 +1,136 @@
+from datetime import datetime
+import docker
+import logging
+import os
+import threading
+import time
+
+ACCESS_LOG_FILE = "/config/log/nginx/access.log"
+LOG_FILE = "/config/log/ondemand/ondemand.log"
+CONTAINER_QUERY_SLEEP = float(os.environ.get("SWAG_ONDEMAND_CONTAINER_QUERY_SLEEP", "5.0"))
+LOG_READER_SLEEP = float(os.environ.get("SWAG_ONDEMAND_LOG_READER_SLEEP", "1.0"))
+STOP_THRESHOLD = int(os.environ.get("SWAG_ONDEMAND_STOP_THRESHOLD", "600"))
+
+last_accessed_urls = set()
+last_accessed_urls_lock = threading.Lock()
+
+class ContainerThread(threading.Thread):
+    def __init__(self):
+        super().__init__()
+        self.daemon = True
+        self.ondemand_containers = {}
+        try:
+            self.docker_client = docker.from_env()
+        except Exception as e:
+            logging.exception(e)
+
+    def process_containers(self):
+        containers = self.docker_client.containers.list(all=True, filters={ "label": ["swag_ondemand=enable"] })
+        container_names = {container.name for container in containers}
+        
+        for container_name in list(self.ondemand_containers.keys()):
+            if container_name in container_names:
+                continue
+            self.ondemand_containers.pop(container_name)
+            logging.info(f"Stopped monitoring {container_name}")
+        
+        for container in containers:
+            container_urls = container.labels.get("swag_ondemand_urls", f"https://{container.name}.,http://{container.name}.")
+            if container.name not in self.ondemand_containers.keys():
+                last_accessed = datetime.now()
+                logging.info(f"Started monitoring {container.name}")
+            else:
+                last_accessed = self.ondemand_containers[container.name]["last_accessed"]
+            self.ondemand_containers[container.name] = { "status": container.status, "urls": container_urls, "last_accessed": last_accessed }
+
+    def stop_containers(self):
+        for container_name in self.ondemand_containers.keys():
+            if self.ondemand_containers[container_name]["status"] != "running":
+                continue
+            inactive_seconds = (datetime.now() - self.ondemand_containers[container_name]["last_accessed"]).total_seconds()
+            if inactive_seconds < STOP_THRESHOLD:
+                continue
+            self.docker_client.containers.get(container_name).stop()
+            logging.info(f"Stopped {container_name} after {STOP_THRESHOLD}s of inactivity")
+    
+    def start_containers(self):
+        with last_accessed_urls_lock:
+            last_accessed_urls_combined = ",".join(last_accessed_urls)
+            last_accessed_urls.clear()
+        
+        for container_name in self.ondemand_containers.keys():
+            accessed = False
+            for ondemand_url in self.ondemand_containers[container_name]["urls"].split(","):
+                if ondemand_url not in last_accessed_urls_combined:
+                    continue
+                self.ondemand_containers[container_name]["last_accessed"] = datetime.now()
+                accessed = True
+            if not accessed or self.ondemand_containers[container_name]["status"] == "running":
+                continue
+            self.docker_client.containers.get(container_name).start()
+            logging.info(f"Started {container_name}")
+            self.ondemand_containers[container_name]["status"] = "running"
+
+    def run(self):
+        while True:
+            try:
+                self.process_containers()
+                self.start_containers()
+                self.stop_containers()
+                time.sleep(CONTAINER_QUERY_SLEEP)
+            except Exception as e:
+                logging.exception(e)
+
+class LogReaderThread(threading.Thread):
+    def __init__(self):
+        super().__init__()
+        self.daemon = True
+    
+    def tail(self, f):
+        f.seek(0,2)
+        inode = os.fstat(f.fileno()).st_ino
+
+        while True:
+            line = f.readline()
+            if not line:
+                time.sleep(LOG_READER_SLEEP)
+                if os.stat(ACCESS_LOG_FILE).st_ino != inode:
+                    f.close()
+                    f = open(ACCESS_LOG_FILE, 'r')
+                    inode = os.fstat(f.fileno()).st_ino
+                continue
+            yield line
+
+    def run(self):
+        while True:
+            try:
+                if not os.path.exists(ACCESS_LOG_FILE):
+                    time.sleep(1)
+                    continue
+
+                logfile = open(ACCESS_LOG_FILE, "r")
+                for line in self.tail(logfile):
+                    for part in line.split():
+                        if not part.startswith("http"):
+                            continue
+                        with last_accessed_urls_lock:
+                            last_accessed_urls.add(part)
+                        break
+            except Exception as e:
+                logging.exception(e)
+                time.sleep(1)
+
+if __name__ == "__main__":
+    os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)
+    logging.basicConfig(filename=LOG_FILE,
+                    filemode='a',
+                    format='%(asctime)s - %(threadName)s - %(levelname)s - %(message)s',
+                    datefmt='%Y-%m-%d %H:%M:%S',
+                    level=logging.INFO)
+    logging.info("Starting swag-ondemand...")
+
+    ContainerThread().start()
+    LogReaderThread().start()
+    
+    while True:
+        time.sleep(1)

README.md

@@ -0,0 +1,19 @@
+proxy_intercept_errors on;
+error_page 502 = @waking_up;
+location @waking_up {
+    add_header Retry-After 1 always;
+    default_type text/html;
+    return 502 '<!DOCTYPE html>
+    <html>
+    <head>
+        <title>Waking Up...</title>
+        <meta http-equiv="refresh" content="1">
+        <style>body{font-family:sans-serif;text-align:center;padding-top:50px;background-color:#1d2022;color:#ffffff;}</style>
+    </head>
+    <body>
+        <h1>Application is sleeping</h1>
+        <p>Please wait while it wakes up...</p>
+        <p>This page will refresh automatically.</p>
+    </body>
+    </html>';
+}