Design a Web Crawler

Hard

System design · Crawling · Google

Design a web crawler that discovers and downloads pages for a search engine or archive.

Requirements

Avoid fetching the same URL or content repeatedly
Respect robots.txt, crawl delays, and per-host rate limits
Prioritize crawl frontier by freshness and importance
Discuss URL normalization, dedupe, queues, workers, storage, and failure retries

Workspace