Design an Alerting Service
MediumSystem design · Observability
Design an alerting system for infrastructure metrics and logs. Teams define rules, and the system evaluates them continuously and notifies the right people through channels such as email, SMS, chat or a paging provider, escalating when nobody responds.
In scope: rule evaluation, alert state, deduplication, grouping, silences and inhibition, routing, escalation and acknowledgement. Out of scope: the metrics store itself and the on-call scheduling UI.
Requirements
- Support threshold rules and window-based rules with a minimum duration before an alert fires
- Deduplicate, group, silence, inhibit and escalate alerts, with acknowledgement tracking
- Avoid alert storms while preserving important signals; a page within 60s of an alert firing
- Keep working when the monitored estate is down