Design Content Moderation
HardSystem design · Trust and Safety · TikTok · Meta
Design a content moderation system for a platform with user-generated text, images and video. Content is checked automatically, uncertain items go to human reviewers, enforcement actions are applied, and users can appeal.
In scope: the triage funnel (known-content matching, classifiers, decision engine), review queue prioritisation, graduated enforcement, appeals, policy and model versioning, and reviewer tooling. Out of scope: training the classifiers and account-level spam or bot detection (see the spam detection problem).
Requirements
- Perceptual hash matching of known material before publication, classifiers for the rest
- Prioritise the review queue by severity times predicted reach, not arrival order or raw report count
- Record policy version, model version, scores and actor for every decision; support appeals by a different reviewer
- Separate policy definitions from detection so jurisdiction-specific rules are configuration