Design a Distributed Cache
HardSystem design · Caching
Design a distributed in-memory cache (Redis Cluster or Memcached-style) that internal services use through a get/set/delete/TTL API. The cache holds about 2TB across the fleet, serves 5M gets/s with p99 under 1ms, and sits in front of a database that can only handle about 5% of that load. Cover partitioning and membership changes, replication and failover, eviction and expiry, and protection of the database from stampedes and hot keys. This is a cache whose data can be rebuilt; a durable key-value database is a separate question.
Requirements
- Partition keys across nodes and add or remove nodes while moving as few keys as possible
- Replicate so that losing a node does not send a surge of misses to the database
- Handle eviction under memory pressure and TTL expiration
- Prevent cache stampede and hot-key saturation, and say how clients find the right node