diff --git a/nginx-reverse-proxy/README.md b/nginx-reverse-proxy/README.md index 5b529261..1b9b91b2 100644 --- a/nginx-reverse-proxy/README.md +++ b/nginx-reverse-proxy/README.md @@ -10,11 +10,20 @@ The reverse proxy configuration is automatically synchronized from the create-a- ### `pull-config.sh` Bash script that: -1. Backs up the current nginx configuration -2. Downloads the latest configuration from the API endpoint -3. Tests the new configuration with `nginx -t` -4. Rolls back to the backup if validation fails -5. Reloads nginx if validation succeeds +1. Checks for an existing ETag and sends it with the request (If-None-Match header) +2. Skips all operations if the server returns 304 Not Modified (no changes) +3. Falls back to internal cluster URL if the primary URL returns a 502 error +4. Backs up the current nginx configuration (if it exists) +5. Downloads the latest configuration from the API endpoint +6. Tests the new configuration with `nginx -t` +7. Rolls back to the backup if validation fails (if backup exists) +8. Saves the ETag for future requests +9. Reloads nginx if validation succeeds + +This approach ensures: +- **Efficient bandwidth usage**: Only downloads when configuration has changed +- **High availability**: Automatically falls back to internal URL on gateway errors +- **Bootstrap-friendly**: Works correctly on first run when no config exists yet ### `pull-config.cron` Cron job definition that runs `pull-config.sh` every minute to keep the nginx configuration synchronized with the database. @@ -71,7 +80,7 @@ sudo chmod +x /opt/opensource-server/nginx-reverse-proxy/pull-config.sh ### 4. Initial Configuration Pull ```bash # Run script manually to get initial configuration -sudo touch /etc/nginx/conf.d/reverse-proxy.conf +# No need to pre-create the config file - the script handles first run when no existing configuration exists sudo /opt/opensource-server/nginx-reverse-proxy/pull-config.sh ``` @@ -93,7 +102,9 @@ sudo tail -f /var/log/syslog | grep pull-config The scripts use these default paths (can be modified in the scripts): - `CONF_FILE`: `/etc/nginx/conf.d/reverse-proxy.conf` - Target nginx config file -- `CONF_URL`: `https://create-a-container.opensource.mieweb.org/nginx.conf` - API endpoint +- `ETAG_FILE`: `/etc/nginx/conf.d/reverse-proxy.etag` - Stores ETag for caching +- `CONF_URL`: `https://create-a-container.opensource.mieweb.org/nginx.conf` - Primary API endpoint +- `FALLBACK_URL`: `http://create-a-container.cluster.mieweb.org:3000/nginx.conf` - Fallback endpoint (used on 502 errors) ### Cron Schedule By default, the configuration is pulled every minute: @@ -109,6 +120,22 @@ To change the schedule, edit `/etc/cron.d/nginx-pull-config` with standard cron # Every day at midnight: 0 0 * * * ``` +**Note**: Thanks to ETag-based caching, running the script every minute has minimal overhead. After the first successful run, the script only downloads and reloads nginx when the configuration actually changes. Most runs will exit early with a 304 Not Modified response. + +### Optimizations + +#### ETag-Based Caching +The script uses HTTP ETags to avoid unnecessary downloads. On each run: +- If an ETag exists from a previous run, it's sent with the request +- If the server returns 304 Not Modified, the script exits immediately +- This reduces bandwidth usage and prevents unnecessary nginx reloads + +#### High Availability Fallback +If the primary URL (via reverse proxy) returns a 502 Bad Gateway error: +- The script automatically falls back to the internal cluster URL +- This handles the bootstrapping problem where the reverse proxy isn't configured yet +- The internal URL bypasses the reverse proxy and connects directly to the service + ## Troubleshooting ### Configuration Pull Failures @@ -185,6 +212,13 @@ ls -lt /etc/nginx/conf.d/reverse-proxy.conf sudo /opt/opensource-server/nginx-reverse-proxy/pull-config.sh ``` +### Force Download (Bypass ETag Cache) +```bash +# Remove the ETag file to force a fresh download +sudo rm -f /etc/nginx/conf.d/reverse-proxy.etag +sudo /opt/opensource-server/nginx-reverse-proxy/pull-config.sh +``` + ### Disable Automatic Updates ```bash # Temporarily disable cron job @@ -193,3 +227,26 @@ sudo chmod 000 /etc/cron.d/nginx-pull-config # Or remove it completely sudo rm /etc/cron.d/nginx-pull-config ``` + +## Testing + +The repository includes comprehensive test scripts to validate the pull-config.sh behavior: + +### Run All Tests +```bash +cd /opt/opensource-server/nginx-reverse-proxy +./test-pull-config.sh +``` + +This tests: +- First run without existing configuration +- ETag caching (304 Not Modified handling) +- Configuration validation and rollback on failure + +### Test 502 Fallback +```bash +cd /opt/opensource-server/nginx-reverse-proxy +./test-502-fallback.sh +``` + +This validates the fallback mechanism when the primary URL returns a 502 error. diff --git a/nginx-reverse-proxy/pull-config.sh b/nginx-reverse-proxy/pull-config.sh index 85aa7318..e305ed9c 100755 --- a/nginx-reverse-proxy/pull-config.sh +++ b/nginx-reverse-proxy/pull-config.sh @@ -3,15 +3,101 @@ set -euo pipefail CONF_FILE=/etc/nginx/conf.d/reverse-proxy.conf +ETAG_FILE="${CONF_FILE}.etag" +TEMP_FILE="${CONF_FILE}.tmp" +HEADERS_FILE="${CONF_FILE}.headers" +BACKUP_FILE="${CONF_FILE}.bak" CONF_URL=https://create-a-container.opensource.mieweb.org/nginx.conf +FALLBACK_URL=http://create-a-container.cluster.mieweb.org:3000/nginx.conf -mv "${CONF_FILE}" "${CONF_FILE}.bak" -curl -fsSL -o "${CONF_FILE}" "${CONF_URL}" +# Cleanup function +cleanup() { + rm -f "${TEMP_FILE}" "${HEADERS_FILE}" "${BACKUP_FILE}" +} -if ! nginx -t; then - mv "${CONF_FILE}.bak" "${CONF_FILE}" +# Set trap to always cleanup on exit +trap cleanup EXIT + +# Function to download config and extract ETag +download_config() { + local url="$1" + + # Build curl command as array + local curl_cmd=( + curl + -w "%{http_code}" + -D "${HEADERS_FILE}" + -o "${TEMP_FILE}" + -sSL + ) + + # Add ETag header if it exists and conf file exists + if [[ -f "${CONF_FILE}" ]] && [[ -f "${ETAG_FILE}" ]]; then + local etag + etag=$(cat "${ETAG_FILE}") + curl_cmd+=(-H "If-None-Match: ${etag}") + fi + + # Add URL + curl_cmd+=("${url}") + + # Execute curl and capture HTTP status code + local http_code + http_code=$("${curl_cmd[@]}" 2>/dev/null || echo "000") + + # Return the http_code + echo "${http_code}" +} + +# Try primary URL +HTTP_CODE=$(download_config "${CONF_URL}") + +# Handle 502, 404 error with fallback +if [[ ${HTTP_CODE} -eq 502 ]] || [[ ${HTTP_CODE} -eq 404 ]] || [[ ${HTTP_CODE} -eq 000 ]]; then + echo "Primary URL failed (HTTP ${HTTP_CODE}), trying fallback URL..." >&2 + HTTP_CODE=$(download_config "${FALLBACK_URL}") +fi + +# Check if we got a 304 Not Modified +if [[ ${HTTP_CODE} -eq 304 ]]; then + # No changes, exit (cleanup handled by trap) + exit 0 +fi + +# Check if we got a successful response +if [[ ${HTTP_CODE} -ne 200 ]]; then + echo "Failed to download configuration (HTTP ${HTTP_CODE})" >&2 + exit 1 +fi + +# Extract new ETag from headers +NEW_ETAG="" +if [[ -f "${HEADERS_FILE}" ]]; then + NEW_ETAG=$(grep -i '^etag:' "${HEADERS_FILE}" | sed 's/^etag: *//i' | tr -d '\r\n' || echo "") +fi + +# Backup existing config if it exists +if [[ -f "${CONF_FILE}" ]]; then + mv "${CONF_FILE}" "${BACKUP_FILE}" +fi + +# Move new config into place +mv "${TEMP_FILE}" "${CONF_FILE}" + +# Test the new configuration (suppress output to avoid cron noise) +if ! nginx -t >/dev/null 2>&1; then + # Restore backup if it exists + if [[ -f "${BACKUP_FILE}" ]]; then + mv "${BACKUP_FILE}" "${CONF_FILE}" + else + # No backup, just remove the bad config + rm -f "${CONF_FILE}" + fi exit 1 fi -rm -f "${CONF_FILE}.bak" -nginx -s reload \ No newline at end of file +# Configuration is valid, save new ETag (cleanup handled by trap) +if [[ -n "${NEW_ETAG}" ]]; then + echo "${NEW_ETAG}" > "${ETAG_FILE}" +fi +nginx -s reload >/dev/null 2>&1 \ No newline at end of file diff --git a/nginx-reverse-proxy/tests/test-502-fallback.sh b/nginx-reverse-proxy/tests/test-502-fallback.sh new file mode 100755 index 00000000..63ae44a8 --- /dev/null +++ b/nginx-reverse-proxy/tests/test-502-fallback.sh @@ -0,0 +1,174 @@ +#!/usr/bin/env bash +# +# Additional test for 502 fallback behavior +# + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TEST_DIR="/tmp/nginx-pull-config-502-test-$$" + +# Colors +GREEN='\033[0;32m' +RED='\033[0;31m' +YELLOW='\033[1;33m' +NC='\033[0m' + +# Setup test environment +setup() { + echo -e "${YELLOW}Setting up test environment for 502 fallback test...${NC}" + mkdir -p "${TEST_DIR}/etc/nginx/conf.d" + mkdir -p "${TEST_DIR}/bin" + + # Create a mock nginx command + cat > "${TEST_DIR}/bin/nginx" <<'EOF' +#!/bin/bash +case "$1" in + -t) + exit 0 + ;; + -s) + if [[ "$2" == "reload" ]]; then + echo "nginx reloaded" + exit 0 + fi + ;; +esac +exit 1 +EOF + chmod +x "${TEST_DIR}/bin/nginx" + export PATH="${TEST_DIR}/bin:${PATH}" +} + +# Cleanup +cleanup() { + echo -e "${YELLOW}Cleaning up...${NC}" + rm -rf "${TEST_DIR}" +} + +# Test 502 fallback +test_502_fallback() { + echo -e "\n${YELLOW}Test: 502 error with fallback to internal URL${NC}" + + # Create modified script + local test_script="${TEST_DIR}/pull-config-502-test.sh" + sed "s|/etc/nginx/conf.d|${TEST_DIR}/etc/nginx/conf.d|g" "${SCRIPT_DIR}/pull-config.sh" > "${test_script}" + chmod +x "${test_script}" + + # Create curl mock that returns 502 for primary URL, 200 for fallback + cat > "${TEST_DIR}/bin/curl" <<'EOF' +#!/bin/bash +url="" +output_file="" +headers_file="" + +# Parse arguments +while [[ $# -gt 0 ]]; do + case $1 in + -o) + output_file="$2" + shift 2 + ;; + -D) + headers_file="$2" + shift 2 + ;; + -H|-w) + shift 2 + ;; + -sSL) + shift + ;; + *) + url="$1" + shift + ;; + esac +done + +# Check which URL is being requested +if [[ "${url}" == *"create-a-container.opensource.mieweb.org"* ]]; then + # Primary URL - return 502 + if [[ -n "${headers_file}" ]]; then + cat > "${headers_file}" <502 Bad Gateway" > "${output_file}" + fi + echo "502" +elif [[ "${url}" == *"create-a-container.cluster.mieweb.org"* ]]; then + # Fallback URL - return 200 + if [[ -n "${headers_file}" ]]; then + cat > "${headers_file}" < "${output_file}" + fi + echo "200" +else + echo "000" +fi +EOF + chmod +x "${TEST_DIR}/bin/curl" + export PATH="${TEST_DIR}/bin:${PATH}" + + # Run the script + if "${test_script}" 2>&1 | tee "${TEST_DIR}/output.log"; then + # Check that config was created from fallback + if [[ -f "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf" ]]; then + if grep -q "fallback URL" "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf"; then + echo -e "${GREEN}✓ Test passed: Fallback URL used successfully${NC}" + + # Check that fallback message was logged + if grep -q "Primary URL failed" "${TEST_DIR}/output.log"; then + echo -e "${GREEN}✓ Test passed: Fallback message logged${NC}" + return 0 + else + echo -e "${RED}✗ Test failed: No fallback message in output${NC}" + return 1 + fi + else + echo -e "${RED}✗ Test failed: Config not from fallback URL${NC}" + cat "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf" + return 1 + fi + else + echo -e "${RED}✗ Test failed: Config file not created${NC}" + return 1 + fi + else + echo -e "${RED}✗ Test failed: Script exited with error${NC}" + cat "${TEST_DIR}/output.log" + return 1 + fi +} + +# Main +main() { + echo -e "${YELLOW}=== Testing 502 Fallback Behavior ===${NC}" + + setup + + if test_502_fallback; then + cleanup + echo -e "\n${GREEN}502 fallback test passed!${NC}" + exit 0 + else + cleanup + echo -e "\n${RED}502 fallback test failed${NC}" + exit 1 + fi +} + +main "$@" diff --git a/nginx-reverse-proxy/tests/test-pull-config.sh b/nginx-reverse-proxy/tests/test-pull-config.sh new file mode 100755 index 00000000..c9a38bfc --- /dev/null +++ b/nginx-reverse-proxy/tests/test-pull-config.sh @@ -0,0 +1,326 @@ +#!/usr/bin/env bash +# +# Test script for pull-config.sh +# This script validates the three main improvements: +# 1. Handling missing config files (first run) +# 2. ETag-based caching +# 3. Fallback to internal URL on 502 errors + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TEST_DIR="/tmp/nginx-pull-config-test-$$" +TEST_RESULTS=0 + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +# Setup test environment +setup() { + echo -e "${YELLOW}Setting up test environment...${NC}" + mkdir -p "${TEST_DIR}/etc/nginx/conf.d" + mkdir -p "${TEST_DIR}/bin" + + # Create a mock nginx command + cat > "${TEST_DIR}/bin/nginx" <<'EOF' +#!/bin/bash +case "$1" in + -t) + # Always succeed for testing + exit 0 + ;; + -s) + if [[ "$2" == "reload" ]]; then + echo "nginx reloaded" + exit 0 + fi + ;; +esac +exit 1 +EOF + chmod +x "${TEST_DIR}/bin/nginx" + + # Add mock nginx to PATH + export PATH="${TEST_DIR}/bin:${PATH}" + + echo -e "${GREEN}✓ Test environment ready${NC}" +} + +# Cleanup test environment +cleanup() { + echo -e "${YELLOW}Cleaning up test environment...${NC}" + rm -rf "${TEST_DIR}" +} + +# Test 1: First run without existing config +test_first_run() { + echo -e "\n${YELLOW}Test 1: First run without existing config${NC}" + + # Create a modified version of the script that uses our test directory + local test_script="${TEST_DIR}/pull-config-test.sh" + sed "s|/etc/nginx/conf.d|${TEST_DIR}/etc/nginx/conf.d|g" "${SCRIPT_DIR}/pull-config.sh" > "${test_script}" + chmod +x "${test_script}" + + # Replace curl with a mock that simulates a successful download + cat > "${TEST_DIR}/bin/curl" <<'EOF' +#!/bin/bash +# Parse arguments +output_file="" +headers_file="" +while [[ $# -gt 0 ]]; do + case $1 in + -o) + output_file="$2" + shift 2 + ;; + -D) + headers_file="$2" + shift 2 + ;; + *) + shift + ;; + esac +done + +# Write mock config +if [[ -n "${output_file}" ]]; then + echo "# Mock nginx config" > "${output_file}" +fi + +# Write mock headers with ETag +if [[ -n "${headers_file}" ]]; then + cat > "${headers_file}" < "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf" + echo '"mock-etag-12345"' > "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf.etag" + + # Create a modified version of the script + local test_script="${TEST_DIR}/pull-config-etag-test.sh" + sed "s|/etc/nginx/conf.d|${TEST_DIR}/etc/nginx/conf.d|g" "${SCRIPT_DIR}/pull-config.sh" > "${test_script}" + chmod +x "${test_script}" + + # Replace curl with a mock that returns 304 + cat > "${TEST_DIR}/bin/curl" <<'EOF' +#!/bin/bash +# Parse arguments to check for If-None-Match header +has_etag=0 +output_file="" +headers_file="" +while [[ $# -gt 0 ]]; do + case $1 in + -H) + if [[ "$2" == If-None-Match* ]]; then + has_etag=1 + fi + shift 2 + ;; + -o) + output_file="$2" + shift 2 + ;; + -D) + headers_file="$2" + shift 2 + ;; + *) + shift + ;; + esac +done + +# If ETag header was sent, return 304 +if [[ ${has_etag} -eq 1 ]]; then + # Write mock headers + if [[ -n "${headers_file}" ]]; then + cat > "${headers_file}" < "${output_file}" + fi + if [[ -n "${headers_file}" ]]; then + cat > "${headers_file}" < "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf" + echo '"mock-etag-old"' > "${TEST_DIR}/etc/nginx/conf.d/reverse-proxy.conf.etag" + + # Create a modified version of the script + local test_script="${TEST_DIR}/pull-config-rollback-test.sh" + sed "s|/etc/nginx/conf.d|${TEST_DIR}/etc/nginx/conf.d|g" "${SCRIPT_DIR}/pull-config.sh" > "${test_script}" + chmod +x "${test_script}" + + # Replace nginx with one that fails validation + cat > "${TEST_DIR}/bin/nginx" <<'EOF' +#!/bin/bash +case "$1" in + -t) + # Fail validation + echo "nginx: configuration file test failed" + exit 1 + ;; + -s) + if [[ "$2" == "reload" ]]; then + echo "nginx reloaded" + exit 0 + fi + ;; +esac +exit 1 +EOF + chmod +x "${TEST_DIR}/bin/nginx" + + # Replace curl to return a new config + cat > "${TEST_DIR}/bin/curl" <<'EOF' +#!/bin/bash +output_file="" +headers_file="" +while [[ $# -gt 0 ]]; do + case $1 in + -o) + output_file="$2" + shift 2 + ;; + -D) + headers_file="$2" + shift 2 + ;; + *) + shift + ;; + esac +done + +if [[ -n "${output_file}" ]]; then + echo "# Invalid new config" > "${output_file}" +fi +if [[ -n "${headers_file}" ]]; then + cat > "${headers_file}" <