Design a Stream Processing Platform

Hard

System design · Data

Design a multi-tenant stream processing platform (Flink-style) on which teams submit long-running stateful jobs over event streams: windowed aggregations, joins and deduplication with event-time semantics. Cover state and checkpointing, exactly-once effects at sinks, watermarks and late data, rescaling, backpressure, and how jobs are submitted, scaled and monitored. ~10M events/s across all jobs, state up to a few TB per job.

Requirements

Support checkpointing and replay
Handle event time, watermarks, and late data
Scale partitions while preserving ordering where needed

Workspace