Design Real-Time Analytics
HardSystem design · Analytics · Netflix · Snowflake · TikTok
Design a real-time analytics service: applications emit events, and operators see aggregated metrics (counts, distinct users, percentiles, top-N, grouped by a few dimensions) within seconds on dashboards and via an API. Focus on event-time windowing and watermarks, late and duplicate events, high-cardinality dimensions, approximate sketches and the serving store. Treat the stream processor as a building block; the generic stream processing engine is a separate problem. Out of scope: ad-hoc warehouse analytics and the dashboard UI.
Requirements
- Aggregate by event time with watermarks, durable replayable input and checkpointed state
- Define a policy for late events (drop, or correct) and deduplicate by event id
- Bound high-cardinality dimensions; use mergeable sketches for distinct counts, percentiles and heavy hitters
- Serve low-latency queries from a store that accepts window revisions; support recomputation by replay