Fix bootstrapping, add ETag caching, and 502/404 fallback in pull-config.sh

nginx-reverse-proxy/README.md

@@ -10,11 +10,20 @@ The reverse proxy configuration is automatically synchronized from the create-a-
 
 ### `pull-config.sh`
 Bash script that:
-1. Backs up the current nginx configuration
-2. Downloads the latest configuration from the API endpoint
-3. Tests the new configuration with `nginx -t`
-4. Rolls back to the backup if validation fails
-5. Reloads nginx if validation succeeds
+1. Checks for an existing ETag and sends it with the request (If-None-Match header)
+2. Skips all operations if the server returns 304 Not Modified (no changes)
+3. Falls back to internal cluster URL if the primary URL returns a 502 error
+4. Backs up the current nginx configuration (if it exists)
+5. Downloads the latest configuration from the API endpoint
+6. Tests the new configuration with `nginx -t`
+7. Rolls back to the backup if validation fails (if backup exists)
+8. Saves the ETag for future requests
+9. Reloads nginx if validation succeeds
+
+This approach ensures:
+- **Efficient bandwidth usage**: Only downloads when configuration has changed
+- **High availability**: Automatically falls back to internal URL on gateway errors
+- **Bootstrap-friendly**: Works correctly on first run when no config exists yet
 
 ### `pull-config.cron`
 Cron job definition that runs `pull-config.sh` every minute to keep the nginx configuration synchronized with the database.
@@ -71,7 +80,7 @@ sudo chmod +x /opt/opensource-server/nginx-reverse-proxy/pull-config.sh
 ### 4. Initial Configuration Pull
 ```bash
 # Run script manually to get initial configuration
-sudo touch /etc/nginx/conf.d/reverse-proxy.conf
+# No need to pre-create the config file - the script handles first run when no existing configuration exists
 sudo /opt/opensource-server/nginx-reverse-proxy/pull-config.sh
 ```
 
@@ -93,7 +102,9 @@ sudo tail -f /var/log/syslog | grep pull-config
 The scripts use these default paths (can be modified in the scripts):
 
 - `CONF_FILE`: `/etc/nginx/conf.d/reverse-proxy.conf` - Target nginx config file
-- `CONF_URL`: `https://create-a-container.opensource.mieweb.org/nginx.conf` - API endpoint
+- `ETAG_FILE`: `/etc/nginx/conf.d/reverse-proxy.etag` - Stores ETag for caching
+- `CONF_URL`: `https://create-a-container.opensource.mieweb.org/nginx.conf` - Primary API endpoint
+- `FALLBACK_URL`: `http://create-a-container.cluster.mieweb.org:3000/nginx.conf` - Fallback endpoint (used on 502 errors)
 
 ### Cron Schedule
 By default, the configuration is pulled every minute:
@@ -109,6 +120,22 @@ To change the schedule, edit `/etc/cron.d/nginx-pull-config` with standard cron
 # Every day at midnight: 0 0 * * *
 ```
 
+**Note**: Thanks to ETag-based caching, running the script every minute has minimal overhead. After the first successful run, the script only downloads and reloads nginx when the configuration actually changes. Most runs will exit early with a 304 Not Modified response.
+
+### Optimizations
+
+#### ETag-Based Caching
+The script uses HTTP ETags to avoid unnecessary downloads. On each run:
+- If an ETag exists from a previous run, it's sent with the request
+- If the server returns 304 Not Modified, the script exits immediately
+- This reduces bandwidth usage and prevents unnecessary nginx reloads
+
+#### High Availability Fallback
+If the primary URL (via reverse proxy) returns a 502 Bad Gateway error:
+- The script automatically falls back to the internal cluster URL
+- This handles the bootstrapping problem where the reverse proxy isn't configured yet
+- The internal URL bypasses the reverse proxy and connects directly to the service
+
 ## Troubleshooting
 
 ### Configuration Pull Failures
@@ -185,6 +212,13 @@ ls -lt /etc/nginx/conf.d/reverse-proxy.conf
 sudo /opt/opensource-server/nginx-reverse-proxy/pull-config.sh
 ```
 
+### Force Download (Bypass ETag Cache)
+```bash
+# Remove the ETag file to force a fresh download
+sudo rm -f /etc/nginx/conf.d/reverse-proxy.etag
+sudo /opt/opensource-server/nginx-reverse-proxy/pull-config.sh
+```
+
 ### Disable Automatic Updates
 ```bash
 # Temporarily disable cron job
@@ -193,3 +227,26 @@ sudo chmod 000 /etc/cron.d/nginx-pull-config
 # Or remove it completely
 sudo rm /etc/cron.d/nginx-pull-config
 ```
+
+## Testing
+
+The repository includes comprehensive test scripts to validate the pull-config.sh behavior:
+
+### Run All Tests
+```bash
+cd /opt/opensource-server/nginx-reverse-proxy
+./test-pull-config.sh
+```
+
+This tests:
+- First run without existing configuration
+- ETag caching (304 Not Modified handling)
+- Configuration validation and rollback on failure
+
+### Test 502 Fallback
+```bash
+cd /opt/opensource-server/nginx-reverse-proxy
+./test-502-fallback.sh
+```
+
+This validates the fallback mechanism when the primary URL returns a 502 error.

nginx-reverse-proxy/pull-config.sh

@@ -3,15 +3,101 @@
 set -euo pipefail
 
 CONF_FILE=/etc/nginx/conf.d/reverse-proxy.conf
+ETAG_FILE="${CONF_FILE}.etag"
+TEMP_FILE="${CONF_FILE}.tmp"
+HEADERS_FILE="${CONF_FILE}.headers"
+BACKUP_FILE="${CONF_FILE}.bak"
 CONF_URL=https://create-a-container.opensource.mieweb.org/nginx.conf
+FALLBACK_URL=http://create-a-container.cluster.mieweb.org:3000/nginx.conf
 
-mv "${CONF_FILE}" "${CONF_FILE}.bak"
-curl -fsSL -o "${CONF_FILE}" "${CONF_URL}"
+# Cleanup function
+cleanup() {
+  rm -f "${TEMP_FILE}" "${HEADERS_FILE}" "${BACKUP_FILE}"
+}
 
-if ! nginx -t; then
-  mv "${CONF_FILE}.bak" "${CONF_FILE}"
+# Set trap to always cleanup on exit
+trap cleanup EXIT
+
+# Function to download config and extract ETag
+download_config() {
+  local url="$1"
+
+  # Build curl command as array
+  local curl_cmd=(
+    curl
+    -w "%{http_code}"
+    -D "${HEADERS_FILE}"
+    -o "${TEMP_FILE}"
+    -sSL
+  )
+
+  # Add ETag header if it exists and conf file exists
+  if [[ -f "${CONF_FILE}" ]] && [[ -f "${ETAG_FILE}" ]]; then
+    local etag
+    etag=$(cat "${ETAG_FILE}")
+    curl_cmd+=(-H "If-None-Match: ${etag}")
+  fi
+
+  # Add URL
+  curl_cmd+=("${url}")
+
+  # Execute curl and capture HTTP status code
+  local http_code
+  http_code=$("${curl_cmd[@]}" 2>/dev/null || echo "000")
+
+  # Return the http_code
+  echo "${http_code}"
+}
+
+# Try primary URL
+HTTP_CODE=$(download_config "${CONF_URL}")
+
+# Handle 502, 404 error with fallback
+if [[ ${HTTP_CODE} -eq 502 ]] || [[ ${HTTP_CODE} -eq 404 ]] || [[ ${HTTP_CODE} -eq 000 ]]; then
+  echo "Primary URL failed (HTTP ${HTTP_CODE}), trying fallback URL..." >&2
+  HTTP_CODE=$(download_config "${FALLBACK_URL}")
+fi
+
+# Check if we got a 304 Not Modified
+if [[ ${HTTP_CODE} -eq 304 ]]; then
+  # No changes, exit (cleanup handled by trap)
+  exit 0
+fi
+
+# Check if we got a successful response
+if [[ ${HTTP_CODE} -ne 200 ]]; then
+  echo "Failed to download configuration (HTTP ${HTTP_CODE})" >&2
+  exit 1
+fi
+
+# Extract new ETag from headers
+NEW_ETAG=""
+if [[ -f "${HEADERS_FILE}" ]]; then
+  NEW_ETAG=$(grep -i '^etag:' "${HEADERS_FILE}" | sed 's/^etag: *//i' | tr -d '\r\n' || echo "")
+fi
+
+# Backup existing config if it exists
+if [[ -f "${CONF_FILE}" ]]; then
+  mv "${CONF_FILE}" "${BACKUP_FILE}"
+fi
+
+# Move new config into place
+mv "${TEMP_FILE}" "${CONF_FILE}"
+
+# Test the new configuration (suppress output to avoid cron noise)
+if ! nginx -t >/dev/null 2>&1; then
+  # Restore backup if it exists
+  if [[ -f "${BACKUP_FILE}" ]]; then
+    mv "${BACKUP_FILE}" "${CONF_FILE}"
+  else
+    # No backup, just remove the bad config
+    rm -f "${CONF_FILE}"
+  fi
   exit 1
 fi
 
-rm -f "${CONF_FILE}.bak"
-nginx -s reload
+# Configuration is valid, save new ETag (cleanup handled by trap)
+if [[ -n "${NEW_ETAG}" ]]; then
+  echo "${NEW_ETAG}" > "${ETAG_FILE}"
+fi
+nginx -s reload >/dev/null 2>&1

nginx-reverse-proxy/tests/test-502-fallback.sh

@@ -0,0 +1,174 @@
+#!/usr/bin/env bash
+#
+# Additional test for 502 fallback behavior
+#
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+TEST_DIR="/tmp/nginx-pull-config-502-test-$$"
+
+# Colors
+GREEN='\033[0;32m'
+RED='\033[0;31m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+
+# Setup test environment
+setup() {
+  echo -e "${YELLOW}Setting up test environment for 502 fallback test...${NC}"
+  mkdir -p "${TEST_DIR}/etc/nginx/conf.d"
+  mkdir -p "${TEST_DIR}/bin"
+
+  # Create a mock nginx command
+  cat > "${TEST_DIR}/bin/nginx" <<'EOF'
+#!/bin/bash
+case "$1" in
+  -t)
+    exit 0
+    ;;
+  -s)
+    if [[ "$2" == "reload" ]]; then
+      echo "nginx reloaded"
+      exit 0
+    fi
+    ;;
+esac
+exit 1
+EOF
+  chmod +x "${TEST_DIR}/bin/nginx"
+  export PATH="${TEST_DIR}/bin:${PATH}"
+}
+
+# Cleanup
+cleanup() {
+  echo -e "${YELLOW}Cleaning up...${NC}"
+  rm -rf "${TEST_DIR}"
+}
+
+# Test 502 fallback
+test_502_fallback() {
+  echo -e "\n${YELLOW}Test: 502 error with fallback to internal URL${NC}"
+
+  # Create modified script
+  local test_script="${TEST_DIR}/pull-config-502-test.sh"
+  sed "s|/etc/nginx/conf.d|${TEST_DIR}/etc/nginx/conf.d|g" "${SCRIPT_DIR}/pull-config.sh" > "${test_script}"
+  chmod +x "${test_script}"
+
+  # Create curl mock that returns 502 for primary URL, 200 for fallback
+  cat > "${TEST_DIR}/bin/curl" <<'EOF'
+#!/bin/bash
+url=""
+output_file=""
+headers_file=""
+
+# Parse arguments
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    -o)
+      output_file="$2"
+      shift 2
+      ;;
+    -D)
+      headers_file="$2"
+      shift 2
+      ;;
+    -H|-w)
+      shift 2
+      ;;
+    -sSL)
+      shift
+      ;;
+    *)
+      url="$1"
+      shift
+      ;;
+  esac
+done
+
+# Check which URL is being requested
+if [[ "${url}" == *"create-a-container.opensource.mieweb.org"* ]]; then
+  # Primary URL - return 502
+  if [[ -n "${headers_file}" ]]; then
+    cat > "${headers_file}" <<HEADERS
+HTTP/1.1 502 Bad Gateway
+Content-Type: text/html
+Content-Length: 150
+
+HEADERS
+  fi
+  if [[ -n "${output_file}" ]]; then
+    echo "<html><body>502 Bad Gateway</body></html>" > "${output_file}"
+  fi
+  echo "502"
+elif [[ "${url}" == *"create-a-container.cluster.mieweb.org"* ]]; then
+  # Fallback URL - return 200
+  if [[ -n "${headers_file}" ]]; then
+    cat > "${headers_file}" <<HEADERS
+HTTP/1.1 200 OK
+Content-Type: text/plain
+ETag: "fallback-etag-123"
+Content-Length: 30
+
+HEADERS
+  fi
+  if [[ -n "${output_file}" ]]; then
+    echo "# Config from fallback URL" > "${output_file}"
+  fi
+  echo "200"
+else
+  echo "000"
+fi
+EOF
+  chmod +x "${TEST_DIR}/bin/curl"
+  export PATH="${TEST_DIR}/bin:${PATH}"
+
+  # Run the script
+  if "${test_script}" 2>&1 | tee "${TEST_DIR}/output.log"; then
+    # Check that config was created from fallback
+    if [[ -f "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf" ]]; then
+      if grep -q "fallback URL" "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf"; then
+        echo -e "${GREEN}✓ Test passed: Fallback URL used successfully${NC}"
+
+        # Check that fallback message was logged
+        if grep -q "Primary URL failed" "${TEST_DIR}/output.log"; then
+          echo -e "${GREEN}✓ Test passed: Fallback message logged${NC}"
+          return 0
+        else
+          echo -e "${RED}✗ Test failed: No fallback message in output${NC}"
+          return 1
+        fi
+      else
+        echo -e "${RED}✗ Test failed: Config not from fallback URL${NC}"
+        cat "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf"
+        return 1
+      fi
+    else
+      echo -e "${RED}✗ Test failed: Config file not created${NC}"
+      return 1
+    fi
+  else
+    echo -e "${RED}✗ Test failed: Script exited with error${NC}"
+    cat "${TEST_DIR}/output.log"
+    return 1
+  fi
+}
+
+# Main
+main() {
+  echo -e "${YELLOW}=== Testing 502 Fallback Behavior ===${NC}"
+
+  setup
+
+  if test_502_fallback; then
+    cleanup
+    echo -e "\n${GREEN}502 fallback test passed!${NC}"
+    exit 0
+  else
+    cleanup
+    echo -e "\n${RED}502 fallback test failed${NC}"
+    exit 1
+  fi
+}
+
+main "$@"