Infrastructure Repository

This repository provides comprehensive Docker-based infrastructure setups for a PostgreSQL High Availability (HA) cluster and a Temporal workflow orchestration engine. These components are designed to work seamlessly together, enabling scalable, fault-tolerant data persistence and workflow management for development and production environments.

Architecture Overview

The infrastructure consists of two main components that can be deployed independently or together:

1. PostgreSQL HA Cluster: A production-ready, three-node PostgreSQL cluster with automatic failover, load balancing, and monitoring capabilities.

2. Temporal Workflow Engine: A distributed workflow orchestration platform that can utilize the HA PostgreSQL cluster for persistent storage.

High-Level Architecture

┌─────────────────┐    ┌─────────────────┐
│   Applications  │────│   Temporal UI   │
│                 │    │   (Port 8080)   │
└─────────────────┘    └─────────────────┘
          │                       │
          └───────────────────────┼──────────────────────┐
                                  │                      │
                    ┌─────────────┴────────────┐   ┌─────┴─────┐
                    │    Temporal Server       │   │ HAProxy   │
                    │    (Port 7233)           │   │ (Ports:   │
                    └──────────────────────────┘   │ 6543,7000 │
                             │                     │ 8404)     │
                             └─────────────────────┼───────────┘
                                                   │
                    ┌──────────────────────────────┼─────────────┐
                    │                              │             │
            ┌───────▼──────┐               ┌───────▼──────┐      │
            │  PostgreSQL  │               │  PostgreSQL  │      │
            │   Node 1     │               │   Node 2     │      │
            │ (Primary)    │               │  (Replica)   │      │
            └──────────────┘               └──────────────┘      │
                    │                              │             │
            ┌───────▼──────┐               ┌───────▼──────┐      │
            │  PostgreSQL  │               │  PostgreSQL  │      │
            │   Node 3     │               │   Node 3     │      │
            │  (Replica)   │               │  (Replica)   │      │
            └──────────────┘               └──────────────┘      │
                    │                              │             │
                    └──────────────────────────────┼─────────────┘
                                                   │
                                             ┌─────▼─────┐
                                             │   etcd    │
                                             │ (Port 2379│
                                             └───────────┘

Components

PostgreSQL HA Cluster (postgres-ha/)

A robust, production-ready PostgreSQL High Availability cluster built with industry-standard tools:

Core Components

• PostgreSQL 15: Enterprise-grade relational database with streaming replication
• Patroni: Cluster management framework handling leader election, failover, and configuration
• etcd: Distributed key-value store providing coordination and consensus for Patroni
• HAProxy: Advanced load balancer with health checks and traffic routing

Key Features

• Automatic Failover: Seamless leader election and failover within seconds
• Read Load Balancing: Distribute read queries across replica nodes
• Health Monitoring: Comprehensive health checks for all cluster components
• Persistent Storage: Docker volumes ensure data persistence across container restarts
• Streaming Replication: Synchronous and asynchronous replication options

Network Ports

• 6543: Primary (read-write) endpoint via HAProxy
• 7000: Replica (read-only) endpoint via HAProxy
• 8404: HAProxy statistics dashboard
• 5433-5435: Direct access to individual PostgreSQL nodes (postgres1, postgres2, postgres3)
• 8008-8010: Patroni REST API endpoints for each node
• 2379/2380: etcd client/peer communication

Management Scripts

Located in postgres-ha/scripts/:

• init-cluster.sh: Initializes the cluster bootstrap process
• health-check.sh: Performs comprehensive health checks on all components
• manage-cluster.sh: Cluster lifecycle management (start/stop/restart/status/logs/failover testing)
• clean-volumes.sh: Resets data volumes for fresh cluster initialization

Temporal Workflow Engine (temporal/)

A complete Temporal workflow orchestration platform deployment:

Core Components

• Temporal Server: Core workflow engine handling workflow execution, task queues, and state management
• Temporal UI: Web-based interface for workflow monitoring, debugging, and management
• Temporal Admin Tools: Command-line utilities for cluster administration and troubleshooting

Key Features

• Auto-Setup Images: Pre-configured containers with automatic database schema setup
• Dynamic Configuration: Runtime configuration overrides via YAML files
• CORS Support: Configurable cross-origin resource sharing for web applications
• Multi-Database Support: Compatible with PostgreSQL, MySQL, Cassandra, and Elasticsearch

Network Ports

• 7233: Temporal server gRPC API
• 8080: Temporal UI web interface

Configuration Variants

• External PostgreSQL: Connects to existing PostgreSQL instances (recommended for HA setups)
• Embedded PostgreSQL: Includes its own PostgreSQL container for development

Prerequisites

• Docker: Version 20.10 or later
• Docker Compose: Version 2.0 or later
• Bash: For running management scripts
• Network Connectivity: Ensure required ports are available and accessible
• System Resources: Minimum 4GB RAM, 2 CPU cores recommended for full deployment

Quick Start

Single Machine Deployment

1. Clone the repository:

   git clone <repository-url>
   cd infra

2. Deploy PostgreSQL HA Cluster:

   cd postgres-ha
   cp .env.example .env
   # Edit .env with your desired passwords
   docker-compose up -d
   ./scripts/health-check.sh

3. Deploy Temporal:

   cd ../temporal
   cp .env.example .env
   # Ensure POSTGRES_HOST points to localhost or appropriate IP
   docker-compose up -d

4. Verify Deployment:

   • Temporal UI: http://localhost:8080
   • HAProxy Stats: http://localhost:8404/stats
   • Run ./scripts/health-check.sh in postgres-ha directory

Detailed Setup

PostgreSQL HA Cluster Configuration

1. Environment Variables: Copy and customize postgres-ha/.env.example:

   cp postgres-ha/.env.example postgres-ha/.env

   Key variables:

   • POSTGRES_PASSWORD: Database superuser password
   • PATRONI_SUPERUSER_PASSWORD: Patroni superuser password
   • PATRONI_REPLICATION_PASSWORD: Replication user password
   • HAPROXY_STATS_USER/PASSWORD: HAProxy statistics credentials

2. Custom Configuration:

   • Patroni: Modify postgres-ha/patroni/patroni.yml for PostgreSQL parameters
   • HAProxy: Customize postgres-ha/haproxy/haproxy.cfg for load balancing rules
   • etcd: Adjust postgres-ha/etcd/etcd.conf for cluster coordination settings

3. Cluster Initialization:

   cd postgres-ha
   ./scripts/manage-cluster.sh init

Temporal Configuration

1. Environment Variables: Copy and customize temporal/.env.example:

   cp temporal/.env.example temporal/.env

   Key variables:

   • POSTGRES_HOST: PostgreSQL cluster endpoint (e.g., host.docker.internal for same machine)
   • POSTGRES_DEFAULT_PORT: HAProxy primary port (6543)
   • POSTGRES_USER/PASSWORD: Database credentials
   • TEMPORAL_VERSION: Temporal server version

2. Dynamic Configuration: Modify YAML files in temporal/dynamicconfig/ for runtime overrides:

   • development-sql.yaml: PostgreSQL-specific settings
   • development-cass.yaml: Cassandra-specific settings (if applicable)

Cross-Machine Deployment

For distributed deployments:

1. Machine A (PostgreSQL HA):

   # Deploy PostgreSQL HA cluster
   cd postgres-ha
   docker-compose up -d
   ./scripts/health-check.sh
   # Note the machine's IP address

2. Machine B (Temporal):

   # Copy temporal directory
   scp -r user@machine-a:/path/to/temporal .
   cd temporal
   # Edit .env to point to Machine A's IP
   sed -i 's/POSTGRES_HOST=.*/POSTGRES_HOST=<MACHINE_A_IP>/' .env
   docker-compose up -d

Monitoring and Management

Health Checks

Run comprehensive health checks:

cd postgres-ha
./scripts/health-check.sh

Cluster Management

Use the management script for common operations:

cd postgres-ha
./scripts/manage-cluster.sh status      # Check cluster status
./scripts/manage-cluster.sh logs        # View all logs
./scripts/manage-cluster.sh logs etcd   # View specific service logs
./scripts/manage-cluster.sh failover    # Test failover mechanism

Access Points

• Temporal UI: http://localhost:8080 (or remote host)
• HAProxy Statistics: http://localhost:8404/stats
• Patroni REST API: http://localhost:8008/health (per node)
• PostgreSQL Direct: localhost:5433-5435 (individual nodes)

Configuration Reference

PostgreSQL HA Cluster

Patroni Configuration (patroni.yml)

• scope: Cluster identifier
• ttl: DCS lock TTL (30s)
• loop_wait: Main loop sleep time (10s)
• postgresql.parameters: PostgreSQL runtime parameters
• bootstrap: Initial cluster configuration

HAProxy Configuration (haproxy.cfg)

• Primary Backend: Routes to current leader (port 5432)
• Replica Backend: Load balances across replicas (port 7000)
• Health Checks: HTTP checks via Patroni REST API

etcd Configuration (etcd.conf)

• name: Cluster member identifier
• data-dir: Data storage directory
• listen-client-urls: Client connection endpoints
• initial-cluster: Bootstrap cluster configuration

Temporal Configuration

Environment Variables

• DB: Database type (postgres12)
• DYNAMIC_CONFIG_FILE_PATH: Path to dynamic configuration
• TEMPORAL_ADDRESS: Server bind address
• TEMPORAL_CORS_ORIGINS: Allowed CORS origins

Dynamic Configuration

Runtime overrides for Temporal server behavior, including:

• Search attribute cache settings
• ID length limits
• Development-specific optimizations

Troubleshooting

Common Issues

1. Cluster Not Starting:

   • Check Docker resources and port availability
   • Verify environment variables in .env files
   • Review container logs: docker-compose logs

2. Replication Lag:

   • Monitor replication status via Patroni API
   • Check network connectivity between nodes
   • Verify PostgreSQL parameters in patroni.yml

3. HAProxy Backend Issues:

   • Access stats page: http://localhost:8404/stats
   • Check health check configurations
   • Verify Patroni REST API accessibility

4. Temporal Connection Issues:

   • Ensure PostgreSQL is accessible from Temporal containers
   • Check database credentials and network connectivity
   • Verify Temporal server logs

Diagnostic Commands

# Check container status
docker-compose ps

# View service logs
docker-compose logs <service-name>

# Test PostgreSQL connectivity
pg_isready -h localhost -p 6543

# Check etcd cluster health
docker exec etcd etcdctl endpoint health

# Verify Patroni cluster status
docker exec postgres1 patronictl -c /patroni.yml list

Security Considerations

Production Deployment

• Change Default Passwords: Use strong, unique passwords in .env files
• Network Security: Restrict access to management ports (8008-8010, 2379, 8404)
• TLS Encryption: Implement TLS for database connections and etcd communication
• Firewall Rules: Configure firewalls to allow only necessary inbound connections
• Secrets Management: Use Docker secrets or external secret management systems

Best Practices

• Regularly update Docker images to latest stable versions
• Monitor cluster health and set up alerting
• Implement backup strategies for data persistence
• Test failover scenarios in staging environments
• Use separate networks for different service tiers

Development and Testing

Local Development

• Use the provided .env.example files as starting points
• Enable debug logging by setting DEBUG_MODE=true
• Utilize the health check scripts for validation

Testing Failover

cd postgres-ha
./scripts/manage-cluster.sh failover

Resetting the Environment

For development testing:

cd postgres-ha
./scripts/clean-volumes.sh

Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes with appropriate documentation
4. Test thoroughly in development environment
5. Submit a pull request with detailed description

License

This project is licensed under the terms specified in the LICENSE file.

Support

For issues and questions:

• Check the troubleshooting section above
• Review Docker and service-specific documentation
• Create an issue in the repository with detailed information