ProblemFeeds and search
Web crawler
Fetch billions of web pages and keep a search engine’s copy of the web fresh, week after week.
The brief
Search crawler · suggested for Senior
Design a web crawler
Design a web crawler for a search engine: start from seed sites, discover and fetch pages across the public web without overloading any site, keep the pages fresh, and hand their text and links to the indexer.
A search company is building its own crawler. It starts from about a million seed sites and follows links to discover the rest of the public web, fetching each page, keeping the raw page for 30 days, and handing its text and links to the indexing system, which another team builds. The web is huge and uneven: tens of millions of sites, a few with hundreds of millions of pages, most with a handful, and many on small, slow servers. Site owners publish rules for crawlers in robots.txt, and some sites generate endless addresses, from event calendars to a session id in every link. The same page is often linked from thousands of places. Pages change at very different rates: news front pages every few minutes, most pages rarely. Machines fail every day at this size, and the crawl has a fixed budget for bandwidth and machines. Design the crawler, from a seed URL to fresh text in the indexer.
Functional requirements
- CoreStart from a list of seed sites and discover new pages by following links.
- CoreFetch each page and keep the raw page for 30 days, so it can be read again.
- CoreExtract each page's text and links, and hand them to the indexing system.
- CoreFetch known pages again so the index stays fresh.
- CoreLet operators block a site, with the block taking effect at once.
Non-functional requirements
- QualityNever overload a site: at most one request at a time to any one site, and no more than about one a second, or slower if the site asks.
- QualityObey each site's robots.txt: skip the paths it disallows and honour any crawl delay it sets.
- CostDon't fetch the same page again and again because many pages link to it or it appears under slightly different addresses, and don't hand the indexer the same content twice.
- QualityDon't let sites that generate endless addresses (calendars, filters, session ids) eat the crawl.
- DurabilityA machine crash or a restart loses no discovered URL and never starts the crawl over.
- LatencyFast-changing pages, like news front pages, are fetched again within about an hour of changing, and every page at least once a month.
- AvailabilityThe crawl keeps running when machines, or a whole region, fail.
- CostBandwidth and machines are the main costs: within a fixed budget, the most valuable pages come first.
Scale
- ScaleAbout 10 billion pages in the index, across tens of millions of sites; a few sites have hundreds of millions of pages.
- ScaleAbout 1 billion fetches a day, new and repeat; pages average about 100 KB.
- ScaleAbout 1 million seed sites.
- ScaleRaw pages kept for 30 days.
Nice to mention (below the line)
- Below the lineRendering pages that need JavaScript.
- Below the lineImages, video and PDFs.
- Below the lineA console where site owners submit or remove URLs.
Out of scope
- Out of scopeBuilding the search index, ranking and serving results.
- Out of scopePages behind logins or paywalls.
- Out of scopeJudging spam or page quality, beyond duplicates.
The variant
One fully specified brief, with the level it suits best. You can practise it at any level.
- Search crawlerSuggested for Senior
Crawl the public web for a search index: find pages through links, fetch every site politely, skip repeats and endless sites, and keep billions of pages fresh within a budget.
8 steps · 6 must-pass · about 45 min at Senior
What each level is graded on
Search crawler, step by step. A result at a level needs its must-pass steps answered, and only the steps you answer count. Going up a level adds steps or holds them to a stricter bar.
| Step | Entry level | Mid-level | Your levelSenior | Staff |
|---|---|---|---|---|
| The numbers that shape the crawl (optional) | Graded | Graded | Graded | Graded |
| URLs, pages and where they live | Graded | Graded | Graded | Graded |
| From a seed URL to stored text | Must-pass | Must-pass | Must-pass | Must-pass |
| Every page once | Must-pass | Must-pass | Must-pass | Must-pass |
| One site, a thousand fetchers | Must-passcan block | Must-passcan block | Must-passcan block | Must-passcan block |
| A site's rules | Must-pass | Must-pass | Must-pass | Must-pass |
| A crash mid-crawl | — | Must-pass | Must-pass | Must-pass |
| What next, and when again | — | — | Must-pass | Must-pass |
| Ten billion pages on a budget | — | — | — | Must-pass |
| At this level | 6 steps 4 must-pass about 32 min | 7 steps 5 must-pass about 38 min | 8 steps 6 must-pass about 45 min | 9 steps 7 must-pass about 53 min |
Entry level6 steps · 4 must-pass · about 32 min
- The numbers that shape the crawl (optional)Graded
- URLs, pages and where they liveGraded
- From a seed URL to stored textMust-pass
- Every page onceMust-pass
- One site, a thousand fetchersMust-passcan block
- A site's rulesMust-pass
Mid-level7 steps · 5 must-pass · about 38 min
- The numbers that shape the crawl (optional)Graded
- URLs, pages and where they liveGraded
- From a seed URL to stored textMust-pass
- Every page onceMust-pass
- One site, a thousand fetchersMust-passcan block
- A site's rulesMust-pass
- A crash mid-crawlMust-pass
SeniorYour level8 steps · 6 must-pass · about 45 min
- The numbers that shape the crawl (optional)Graded
- URLs, pages and where they liveGraded
- From a seed URL to stored textMust-pass
- Every page onceMust-pass
- One site, a thousand fetchersMust-passcan block
- A site's rulesMust-pass
- A crash mid-crawlMust-pass
- What next, and when againMust-pass
Staff9 steps · 7 must-pass · about 53 min
- The numbers that shape the crawl (optional)Graded
- URLs, pages and where they liveGraded
- From a seed URL to stored textMust-pass
- Every page onceMust-pass
- One site, a thousand fetchersMust-passcan block
- A site's rulesMust-pass
- A crash mid-crawlMust-pass
- What next, and when againMust-pass
- Ten billion pages on a budgetMust-pass
- Must-pass
- a result at that level needs this step answered.
- Can block
- a miss there is a Blocking issue at that level.
Going up a level
- Mid-level
- “From a seed URL to stored text” is graded more strictly.Adds “A crash mid-crawl”, must-pass.
- Senior
- “A site's rules” is graded more strictly.“A crash mid-crawl” is graded more strictly.Adds “What next, and when again”, must-pass.
- Staff
- Adds “Ten billion pages on a budget”, must-pass.
Skills this problem practises
Your coach tracks readiness per skill across every problem you practise. Each links to the lesson that teaches it.
- Capacity estimation
- Data modelling
- Choosing storage
- Indexing
- Sharding and partitioning
- High-level architecture
- Queues and async messaging
- Networking and protocols
- Caching
- Failure modes and recovery
- Observability and operations
- Rate limiting and abuse
Worked solution
Try it first: practise a variant, then read the walkthrough and compare it with your own design. Reading it first makes the practice much less useful.
Learn: Design a web crawlerYour history
Sign in to see your sessions on this problem.