Design an A/B Testing Platform
HardSystem design · Experimentation
Design an experimentation platform that assigns users to variants and measures the effect on product metrics.
In scope: creating and ramping experiments, deterministic assignment, mutually exclusive experiments, exposure and metric collection, statistical analysis, and automatic guardrails.
Out of scope: the statistics of any one test in depth, and the product UI for editing experiments.
Requirements
- Assign users deterministically with no stored per-user state, and without correlation across experiments
- Support mutually exclusive experiments (layers), ramping, and holdouts without reshuffling users
- Log exposure only when the user actually sees the variant, and attribute delayed conversions within a defined window
- Run guardrail metrics on every experiment and stop harmful ones automatically