Observability GitOps Platform

A complete observability stack deployed via GitOps (ArgoCD) on Kubernetes, provisioned with Terraform. Demonstrates production-grade monitoring, logging, and alerting patterns for a DevOps portfolio.

Architecture

graph TB
    subgraph "Git Repository"
        GR[gitops/applications/]
        AM[apps/ manifests]
        TF[terraform/ IaC]
    end

    subgraph "ArgoCD"
        RA[Root App]
        RA --> PA[Prometheus App]
        RA --> GA[Grafana App]
        RA --> LA[Loki App]
        RA --> AA[Alertmanager App]
        RA --> PTA[Promtail App]
        RA --> DA[Demo App]
    end

    subgraph "Kubernetes Cluster"
        KC["Kubernetes Cluster"]
        subgraph "monitoring namespace"
            P[Prometheus]
            G[Grafana]
            L[Loki]
            AL[Alertmanager]
            PT[Promtail]
        end
        subgraph "demo namespace"
            D[Demo App]
        end
    end

    GR -->|watches| RA
    TF -->|provisions| KC

    D -->|/metrics| P
    PT -->|collects logs| L
    P -->|alerts| AL
    G -->|queries metrics| P
    G -->|queries logs| L
    P -->|scrapes| D

Loading

Data Flow

1. Metrics: Demo App exposes /metrics -> Prometheus scrapes every 15s -> Grafana visualizes -> Alertmanager fires alerts
2. Logs: Demo App writes JSON to stdout -> Promtail collects from node -> Loki stores and indexes -> Grafana queries via LogQL
3. GitOps: Git push -> ArgoCD detects drift -> Syncs desired state -> Kubernetes reconciles

Why GitOps?

Traditional CI/CD uses a push model where the pipeline pushes changes to the cluster. GitOps uses a pull model where an in-cluster agent (ArgoCD) continuously pulls the desired state from Git and reconciles it. This means Git becomes the single source of truth -- every change is auditable, reversible, and declarative. If someone manually changes the cluster, ArgoCD detects the drift and reverts it automatically.

Stack

Component	Purpose
Terraform	Infrastructure provisioning (k3d local / EKS production)
ArgoCD	GitOps continuous delivery -- syncs Git state to Kubernetes
Prometheus	Metrics collection and alerting rules engine
Grafana	Visualization dashboards for metrics and logs
Loki	Log aggregation system (like Prometheus, but for logs)
Alertmanager	Alert routing, grouping, and notification delivery
Promtail	Log collection agent (ships container logs to Loki)
Demo App	Go HTTP server instrumented with metrics and structured logging

Quick Start

1. Provision a local cluster

cd terraform
make local-init && make local-plan && make local-apply

2. Install ArgoCD

kubectl apply -f gitops/bootstrap/namespace.yaml
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd --values gitops/bootstrap/values.yaml --wait

3. Deploy the observability stack

kubectl apply -f gitops/applications/root-app.yaml

4. Access the dashboards

# Get ArgoCD admin password
make argocd-password

# Port-forward ArgoCD UI
make argocd-port-forward
# Open https://localhost:8080

# Port-forward Grafana
make grafana-port-forward
# Open http://localhost:3000

Project Structure

observability-gitops/
  terraform/                  # Infrastructure as Code
    environments/local/       # k3d cluster for development
    environments/aws/         # EKS cluster for production
    modules/kubernetes-cluster/ # Reusable cluster module
  gitops/                     # GitOps layer (ArgoCD)
    bootstrap/                # ArgoCD installation manifests
    projects/                 # AppProject definitions (RBAC)
    applications/             # App of Apps directory
  apps/                       # Application manifests (consumed by ArgoCD)
    prometheus/               # kube-prometheus-stack values
    grafana/                  # Grafana values + dashboard JSONs
    loki/                     # Loki values
    alertmanager/             # Alertmanager routing config
    promtail/                 # Promtail log collection config
    demo-app/                 # Demo app Kubernetes manifests
  src/demo-app/               # Demo application source code (Go)
  docs/                       # Architecture docs and ADRs

Key Concepts Demonstrated

• App of Apps Pattern -- Single root Application bootstraps the entire stack
• Golden Signals Dashboards -- Latency, Traffic, Errors, Saturation monitoring
• Log Aggregation with Loki -- Structured logging pipeline with Promtail
• Metrics Correlation -- Linking Prometheus metrics with Loki logs in Grafana
• Automated Drift Detection -- ArgoCD self-healing reverts manual changes
• Infrastructure as Code -- Terraform modules for local and cloud environments

License

MIT