Skip to content

Configure Grafana alerting for error spikes #155

@GitAddRemote

Description

@GitAddRemote

Summary

Set up Grafana alert rules to notify on error spikes and other critical log conditions once Grafana Loki is running.

Acceptance criteria

  • Alert fires when error-level logs (pino level >= 50) exceed 10 occurrences in a 5-minute window
  • Alert is delivered to a configured notification channel (email or webhook)
  • Alert rules are defined as code (Grafana provisioning YAML) so they are version-controlled and survive container restarts
  • A silencing window can be applied during planned deployments

Implementation notes

  • Use Grafana's built-in alerting (not the legacy version)
  • Define alert rules in infra/grafana/provisioning/alerting/ so they are provisioned on container start
  • Configure a contact point in infra/grafana/provisioning/alerting/contact-points.yaml
  • Document the alert setup in infra/docs/logging.md

Prerequisites

  • Grafana Loki setup complete (see Loki issue)

Metadata

Metadata

Assignees

No one assigned

    Labels

    tech-storyTechnical implementation storytelemetryLogging, metrics, tracing

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions