Add backup-tool: dockerized incremental backup with monitoring and Telegram alerts

README.md

@@ -77,6 +77,33 @@ A robust Debian/Ubuntu-friendly tool to update Docker Compose stacks.
 - Sufficient permissions to run docker commands (may require sudo)
 - `flock` command (usually pre-installed on Linux systems)
 
+### backup-tool
+
+A simple, robust backup tool for homelabs. Dockerized with privileges, runs on a Docker Swarm manager node.
+
+**Features:**
+
+- Incremental backups via rsync (SSH hosts) and docker cp (container volumes)
+- Interactive setup wizard for hosts, paths, drives, and Telegram alerts
+- Time & size based retention per host/path
+- Replicate important backups to multiple drives (compressed replicas, no RAID)
+- Telegram bot alerts with exponential back-off (offline, disk space, errors)
+- Offline detection every 30 seconds, independent of backup schedule
+
+**Quick start:**
+
+```bash
+cd packages/backup-tool
+
+# Interactive setup
+docker compose run --rm backup setup
+
+# Start the service
+docker compose up -d
+```
+
+See [packages/backup-tool/README.md](packages/backup-tool/README.md) for full documentation.
+
 ### create-package
 
 A tool to create new package directories with bash script templates.

packages/backup-tool/Dockerfile

@@ -0,0 +1,21 @@
+FROM ubuntu:24.04
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      bash \
+      coreutils \
+      curl \
+      gzip \
+      openssh-client \
+      rsync \
+      tar \
+      docker.io \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /opt/backup-tool
+
+COPY . .
+
+RUN chmod +x backup-tool setup
+
+ENTRYPOINT ["./backup-tool"]
+CMD ["run"]

packages/backup-tool/README.md

@@ -0,0 +1,179 @@
+# backup-tool
+
+A simple, robust backup tool for homelabs. Dockerized with privileges, designed to run on a Docker Swarm manager node as a normal Compose stack on Ubuntu Server.
+
+## Features
+
+- **SSH hosts (rsync)** — incremental backups of remote directories via rsync over SSH
+- **Docker volumes** — backup container volumes using docker commands
+- **Interactive setup** — wizard to configure hosts, paths, drives, and alerts
+- **Time & size based retention** — per host/path TTL policies
+- **Important data replication** — replicate critical backups to multiple drives (no RAID)
+- **Compressed replicas** — replicas are compressed after copy; main backup stays uncompressed for fast rsync
+- **Telegram alerts** — error notifications with exponential back-off (no spam)
+- **Offline detection** — checks every 30 seconds regardless of backup schedule
+- **Timed incremental backups** — configurable interval, link-dest based incremental rsync
+
+## Quick Start
+
+```bash
+# 1. Clone and navigate to the backup-tool directory
+cd packages/backup-tool
+
+# 2. Run the interactive setup wizard
+docker compose run --rm backup setup
+
+# 3. Start the backup service
+docker compose up -d
+
+# 4. View logs
+docker compose logs -f
+```
+
+## Configuration
+
+All configuration lives in the `config/` directory (mounted at `/etc/backup-tool` in the container).
+
+### Main config: `config/backup.conf`
+
+| Key | Description | Default |
+|-----|-------------|---------|
+| `backup_dest` | Where backups are stored | `/backup` |
+| `backup_interval_min` | Backup interval in minutes | `60` |
+| `important_drives` | Comma-separated mount points for replication | |
+| `important_folders` | Comma-separated paths relative to backup\_dest | |
+| `telegram_bot_token` | Telegram bot token | |
+| `telegram_chat_id` | Telegram chat ID | |
+
+### Host configs: `config/hosts.d/<name>.conf`
+
+One file per backup source. See `config/host.conf.example`.
+
+| Key | Description |
+|-----|-------------|
+| `type` | `ssh` (rsync) or `docker` (container volumes) |
+| `address` | SSH address (user@host) |
+| `paths` | Comma-separated paths or volume names |
+| `retention_days` | Days to keep snapshots (0 = forever) |
+| `retention_max_size` | Max total size (e.g. `10G`) |
+
+## How It Works
+
+### Backup Cycle
+
+1. For each configured host, back up each path:
+   - **SSH hosts**: `rsync -a --delete --link-dest=<latest>` for incremental backups
+   - **Docker volumes**: temporary container mount + `docker cp`
+2. Apply retention policies (delete old/oversized snapshots)
+3. Replicate important folders to all important drives
+4. Compress replicas (main copy stays uncompressed)
+
+### Monitoring (runs independently)
+
+- **Offline detection**: pings all SSH hosts every 30 seconds
+- **Disk space**: checks all important drives and backup destination every 60 seconds
+- Alerts go to Telegram with exponential back-off (2min → 4min → 8min → ... → 8h max)
+- Recovery messages are sent when issues clear
+
+### Directory Structure
+
+Backups are stored as timestamped snapshots:
+
+```
+/backup/
+  myhost/
+    var_data/
+      20250101-120000/     ← full snapshot
+      20250102-120000/     ← incremental (hardlinked to previous)
+      latest -> 20250102-120000
+    home_user/
+      ...
+```
+
+Replicas on important drives:
+
+```
+/mnt/usb1/
+  backup-replicas/
+    myhost_var_data.tar.gz   ← compressed copy
+    myhost_home_user.tar.gz
+```
+
+## Commands
+
+```bash
+# Run continuous backup + monitoring (default)
+docker compose run --rm backup run
+
+# Run a single backup cycle
+docker compose run --rm backup once
+
+# Interactive setup wizard
+docker compose run --rm backup setup
+
+# Run only the monitor (offline/disk checks)
+docker compose run --rm backup monitor
+
+# Run only retention cleanup
+docker compose run --rm backup retention
+
+# Run only replication
+docker compose run --rm backup replicate
+```
+
+## Telegram Bot Setup
+
+1. Open Telegram and search for **@BotFather**
+2. Send `/newbot` and follow the prompts
+3. BotFather gives you a **bot token** — e.g. `123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11`
+4. Open a chat with your new bot and send `/start`
+5. Get your **chat ID** by visiting:
+   ```
+   https://api.telegram.org/bot<YOUR_TOKEN>/getUpdates
+   ```
+   Look for `"chat":{"id":NNNNNN}` in the response
+6. Add both values to your config:
+   ```
+   telegram_bot_token=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
+   telegram_chat_id=123456789
+   ```
+
+### Alert Behavior
+
+| Alert | Check interval | Back-off |
+|-------|---------------|----------|
+| Host offline | 30 seconds | 2min → 4min → 8min → ... → 8h |
+| Disk space > 90% | 60 seconds | 2min → 4min → 8min → ... → 8h |
+| Backup/rsync error | Per backup cycle | 2min → 4min → 8min → ... → 8h |
+
+Recovery messages are sent when the condition clears, and the back-off timer resets.
+
+## Requirements
+
+- Docker Engine with docker compose plugin
+- Ubuntu Server (host)
+- SSH key-based authentication to backup targets (for SSH hosts)
+- Docker socket access (for container volume backups)
+
+## Docker Compose Reference
+
+```yaml
+services:
+  backup:
+    build: .
+    privileged: true
+    network_mode: host
+    volumes:
+      - ./config:/etc/backup-tool          # Configuration
+      - backup-state:/var/lib/backup-tool  # Alert state
+      - backup-logs:/var/log/backup-tool   # Logs
+      - /backup:/backup                    # Backup destination
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - ~/.ssh:/root/.ssh:ro               # SSH keys for rsync
+```
+
+The container needs:
+- **privileged** — access to host devices and Docker socket
+- **network_mode: host** — SSH to remote hosts without port mapping
+- **Docker socket** — to query Swarm nodes and copy from volumes
+- **SSH keys** — for passwordless rsync to remote hosts

packages/backup-tool/backup-tool

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# shellcheck shell=bash
+# backup-tool — main entry point.
+# Runs the monitor loop in the background and executes timed incremental backups.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Load libraries
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+# shellcheck source=lib/telegram.sh
+source "${SCRIPT_DIR}/lib/telegram.sh"
+# shellcheck source=lib/monitor.sh
+source "${SCRIPT_DIR}/lib/monitor.sh"
+# shellcheck source=lib/backup.sh
+source "${SCRIPT_DIR}/lib/backup.sh"
+# shellcheck source=lib/retention.sh
+source "${SCRIPT_DIR}/lib/retention.sh"
+# shellcheck source=lib/replication.sh
+source "${SCRIPT_DIR}/lib/replication.sh"
+
+ensure_dirs
+
+# ── Usage ────────────────────────────────────────────────────────────────────
+usage() {
+  cat <<EOF
+backup-tool v${BACKUP_TOOL_VERSION}
+
+Usage:
+  backup-tool run          Start continuous backup + monitoring loop (default)
+  backup-tool once         Run a single backup cycle then exit
+  backup-tool setup        Launch interactive configuration wizard
+  backup-tool monitor      Run only the monitor loop (no backups)
+  backup-tool retention    Run only retention cleanup
+  backup-tool replicate    Run only replication of important folders
+  backup-tool help         Show this help
+
+Environment:
+  BACKUP_CONF_DIR   Config directory   (default: /etc/backup-tool)
+  BACKUP_STATE_DIR  State directory    (default: /var/lib/backup-tool)
+  BACKUP_LOG_DIR    Log directory      (default: /var/log/backup-tool)
+EOF
+}
+
+# ── Single backup cycle ─────────────────────────────────────────────────────
+run_cycle() {
+  local start
+  start=$(date +%s)
+  log_step "Backup cycle starting at $(date)"
+
+  backup_all || true
+  retention_all || true
+  replicate_important || true
+
+  local elapsed=$(( $(date +%s) - start ))
+  log_ok "Backup cycle complete in ${elapsed}s"
+}
+
+# ── Continuous loop ──────────────────────────────────────────────────────────
+run_loop() {
+  local interval_min
+  interval_min=$(cfg_get "$BACKUP_CONF_FILE" backup_interval_min 60)
+  local interval_sec=$(( interval_min * 60 ))
+
+  log_info "backup-tool v${BACKUP_TOOL_VERSION} starting"
+  log_info "Backup interval: ${interval_min}m, config: ${BACKUP_CONF_DIR}"
+
+  # Start monitor in background
+  monitor_loop &
+  local monitor_pid=$!
+  trap 'kill $monitor_pid 2>/dev/null || true; exit 0' EXIT INT TERM
+  log_info "Monitor started (PID $monitor_pid)"
+
+  while true; do
+    run_cycle 2>&1 | tee -a "${BACKUP_LOG_DIR}/backup.log"
+    log_info "Next backup in ${interval_min}m — sleeping"
+    sleep "$interval_sec"
+  done
+}
+
+# ── Main ─────────────────────────────────────────────────────────────────────
+main() {
+  local cmd="${1:-run}"
+  case "$cmd" in
+    run)
+      run_loop
+      ;;
+    once)
+      run_cycle
+      ;;
+    setup)
+      # Re-exec into the setup wizard
+      exec "${SCRIPT_DIR}/setup" "${@:2}"
+      ;;
+    monitor)
+      monitor_loop
+      ;;
+    retention)
+      retention_all
+      ;;
+    replicate)
+      replicate_important
+      ;;
+    help|--help|-h)
+      usage
+      ;;
+    *)
+      log_error "Unknown command: $cmd"
+      usage >&2
+      exit 1
+      ;;
+  esac
+}
+
+main "$@"

packages/backup-tool/config/backup.conf.example

@@ -0,0 +1,25 @@
+# backup-tool example configuration
+#
+# Copy this file to config/backup.conf and edit as needed,
+# or run the interactive setup: docker compose run --rm backup setup
+
+# Where to store backups (must match the volume mount in docker-compose.yml)
+backup_dest=/backup
+
+# How often to run backups (minutes)
+backup_interval_min=60
+
+# Important drives — comma-separated mount points for replication
+# After backing up important folders, latest snapshots are replicated
+# to every drive listed here (no RAID — simple independent copies).
+# Replicas are compressed after copy; main backup stays uncompressed for rsync.
+# important_drives=/mnt/usb1,/mnt/usb2
+
+# Important folders — comma-separated, relative to backup_dest
+# Format: host_label/safe_path  (e.g. nas/var_data, pi4/home_user)
+# important_folders=nas/var_data,pi4/home_user
+
+# Telegram alerts (optional)
+# See README.md for setup instructions.
+# telegram_bot_token=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
+# telegram_chat_id=123456789