Design a Message Queue
HardSystem design · Messaging Infrastructure · Amazon
Design a durable message broker in the style of Kafka (and contrast it with SQS or RabbitMQ). Producers publish to topics, consumer groups process messages, and data is retained for replay. Scale: 10M messages/s at 1KB (10GB/s in), replication factor 3, 7 days retention, publish p99 under 10ms. Cover the partitioned log, ordering, replication and acknowledgements, consumer groups and offsets, delivery semantics, retries and dead-letter handling, backpressure, and capacity.
Requirements
- Messages are appended to a partitioned, replicated log and retained by time and size
- Ordering is guaranteed per partition only; parallelism is bounded by partition count
- At-least-once delivery by default, with a clear statement of what exactly-once can and cannot cover
- Retries, dead-letter topics and backpressure for slow consumers and producers