This page summarizes how observability is implemented using Prometheus, Grafana, and supporting services.
services/exporter) — custom Prometheus exporter that validates objects in S3 and exposes metrics.services/blackbox) — Prometheus Blackbox exporter used for external endpoint checks.services/prometheus) — main metrics storage and scraping configuration.services/grafana) — dashboarding and alerting UI.These components are wired in docker-compose.prod.yml and Kubernetes manifests under infrastructure/k8s-manifests.
Prometheus is configured via services/prometheus/prometheus.yml to:
Grafana dashboards and datasources are provisioned from:
services/grafana/provisioning/ — defines datasources (Prometheus, Loki, etc.) and base dashboards.services/grafana/snapshots/ — YAML snapshots for alert rules, including:
services/grafana/snapshots/alerts/prod/wealth-api-alerts/.The sync script scripts/grafana/sync_alerts.py and its README (scripts/grafana/README.md) describe how to keep live Grafana alert rules in sync with the snapshot files.
structlog) ensures request IDs and context are available for debugging.services/loki) can be configured as a log backend and integrated into Grafana dashboards.When adding new services or metrics:
services/prometheus/prometheus.yml.services/grafana/provisioning and services/grafana/snapshots.