Design a Search Engine
HardSystem design · Search
Design a web search engine. A crawler discovers and fetches pages, a pipeline parses, deduplicates and indexes them, and a serving tier answers keyword queries over billions of documents with ranked results in under 200ms. Focus on how crawling and indexing are decoupled, the sharded inverted index and query fan-out, two-phase ranking with link and quality signals, near-duplicate handling, freshness and caching. Crawler internals (frontier, politeness) are covered by the web crawler problem. Out of scope: ads, spell correction, and query understanding with ML.
Requirements
- Crawl politely and avoid duplicate content (canonical URLs, exact and near-duplicate detection)
- Build and serve an inverted index sharded by document, with immutable segments, merging and a real-time tier
- Rank in two phases using relevance, link-based authority and quality signals
- Cut latency and cost with tiered indexes, result caching, hedged requests and per-query deadlines