Design a Streaming Analytics Platform
HardSystem design · Streaming Analytics · Databricks
Design a managed streaming analytics platform in the style of Databricks Structured Streaming. Customers define transformations over event streams, maintain stateful aggregations with checkpoints, and query near-real-time tables.
In scope: micro-batch execution over a batch engine, offsets and checkpoints, state stores and watermarks, exactly-once into transactional tables, restarts and scaling, schema changes, and monitoring. Out of scope: building the message broker and the table format themselves.
Requirements
- Ingest from durable logs with replayable offsets and checkpoint source positions together with state
- Bound state with watermarks; define how late events are handled
- Exactly-once results via an idempotent or transactional sink committed with the checkpoint
- Survive job restarts and schema changes; autoscale without rebuilding state unnecessarily