Design a Logging System
MediumSystem design · Observability
Design a centralized logging system: services emit structured and unstructured logs, engineers search them by time, service, severity, request id and text, and teams alert on log patterns. Scale: 5TB/day (average 230K lines/s, peak 2M lines/s), 30 days hot and 1 year cold, and under 0.1% of lines ever read. Cover agents and buffering during outages, parsing and redaction, label-based versus full-text indexing and its cost, retention tiers, volume control and sampling, and stream-based alerting.
Requirements
- Logging must never block the application: bounded buffers that drop and count under pressure
- Redact sensitive data at ingest, before anything is written
- Index a small set of labels and store bodies as compressed chunks; full-text indexing is opt-in per stream
- Retention is tiered and set per stream; alerts evaluate on the stream, not by repeated searches