Design Metrics Monitoring
HardSystem design · Observability
Design a metrics monitoring system like Datadog or Prometheus at cloud scale. Services emit numeric time series (counters, gauges, histograms) with labels such as service, host and endpoint. Engineers build dashboards over them and define alert rules that page on-call when a condition holds. Focus on ingestion, time-series storage, queries and alert evaluation. Out of scope: logs, traces, and the dashboard UI itself.
Requirements
- Ingest 10M samples/sec reliably without slowing the services being monitored
- Support aggregation, downsampling, tiered retention and a bounded number of label combinations (cardinality)
- Serve dashboard queries with low latency, including multi-month ranges
- Evaluate alert rules continuously, deduplicate and group notifications, and keep alerting up when the monitored infrastructure is not