Loading…

Understanding the problem

A search engine’s crawler starts from seed sites, follows links across the public web, fetches each page, keeps the raw page for a while and hands its text and links to the indexer. The hard parts come from the web: tens of millions of sites that must not be overloaded, one page linked from thousands of places, sites with endless addresses, pages that change at very different rates, and machines that fail every day. This page designs the family’s one variant, Search crawler.

The practice brief states the requirements up front; confirm them in the first few minutes. The core is the loop from a URL in the frontier to stored text and new links.

Functional requirements

CoreStart from a list of seed sites and discover new pages by following links.
CoreFetch each page and keep the raw page for 30 days, so it can be read again.
CoreExtract each page's text and links, and hand them to the indexing system.
CoreFetch known pages again so the index stays fresh.
CoreLet operators block a site, with the block taking effect at once.
Below the lineRendering pages that need JavaScript.
Below the lineImages, video and PDFs.
Below the lineA console where site owners submit or remove URLs.
Below the lineFilling in forms or search boxes to find pages.Those pages sit outside the link graph, and forms can change things on the site.
Below the lineDeciding what to index.The indexer’s job. The crawler hands over text and links, minus duplicates.

Non-functional requirements

QualityNever overload a site: at most one request at a time to any one site, and no more than about one a second, or slower if the site asks.
QualityObey each site's robots.txt: skip the paths it disallows and honour any crawl delay it sets.
CostDon't fetch the same page again and again because many pages link to it or it appears under slightly different addresses, and don't hand the indexer the same content twice.
QualityDon't let sites that generate endless addresses (calendars, filters, session ids) eat the crawl.
DurabilityA machine crash or a restart loses no discovered URL and never starts the crawl over.
Latency

Scale

From the briefNumbers here are given in the practice brief.

ScaleAbout 10 billion pages in the index, across tens of millions of sites; a few sites have hundreds of millions of pages.
ScaleAbout 1 billion fetches a day, new and repeat; pages average about 100 KB.
ScaleAbout 1 million seed sites.
ScaleRaw pages kept for 30 days.

Out of scope

Out of scopeBuilding the search index, ranking and serving results.
Out of scopePages behind logins or paywalls.
Out of scopeJudging spam or page quality, beyond duplicates.

Check yourself

The brief says one request at a time per site. Why is that harder with more machines?
Show answerHide answer
Each fetcher can be well behaved on its own while hundreds of them pick the same site. The limit has to be decided in one place per site, not inside each fetcher.

The numbers that shape the crawl

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Work out only the numbers this design depends on, and say the page size you assume. For scale, Common Crawl’s August 2025 archive holds 2.44 billion pages from 38.5 million registered domains, crawled in about two weeks (Common Crawl (opens in a new tab)).

Back of the envelope for the search crawler

Quantity	Value	Working
Fetches	About 11,600 a second	A billion a day ÷ 86,400; the crawler sets its own pace.
Downloads	About 9 Gbps	About 100 KB a page × 11,600 a second.
Raw pages stored	About 100 TB a day	Before compression; about 3 PB for 30 days.
Fetches in flight	About 11,600	If a fetch takes about a second.
Sites being fetched at once	At least 11,600	One request at a time per site.
URLs known after a month	Say 30 billion	Pages link to many pages not fetched yet.

What the numbers tell you

The crawl is limited by how many sites it keeps busy at once.
Known URLs, and the record of which were seen, are too big for one machine.

Read:Learn: Numbers and capacity estimation

Try it: The numbers that shape the crawlGuided practice · Web crawler, Search crawler · Senior

Check yourself

Which number tells you the frontier needs a queue per site?
Show answerHide answer
Sites in progress at once. At one request at a time per site, about 11,600 fetches in flight means at least 11,600 sites must have their turn at any moment, so the frontier has to hand out work by site.

Core entities

Start from the reads: look up a URL, find its latest fetch, read a stored page back to parse it again, and check a site’s rules and pace. Records are billions of small rows read by key; pages are large, written once and read by the parser.

The records that matter

Record	Key fields	Why it exists
URL	`url_hash`, url, site, `last_fetched_at`, status	What is known about one address, one lookup away.
Fetch	`url_hash` + crawl cycle (the `fetch_id`), status, `fetched_at`, page key, text key, fingerprint	One row per fetch, newest first; a repeat writes the same row.
Link	`from_url`, `to_url`	Which pages are linked to most.
Site	registered domain, robots rules, crawl delay, next allowed time, budget	The rules and pace the deep dives rely on.
Frontier entry	site, url, priority, due time, lease expiry	A URL waiting, or leased to a fetcher.

Where each kind of data lives

Data	Store	Why
Records	A wide-column or key-value store such as Bigtable or Cassandra	Single-row reads and writes by key, no transactions across rows.
Raw pages and text	Object storage such as S3 or Google Cloud Storage	Large, written once, cheap, deleted after 30 days.
Frontier	A replicated log or table, split by site	Survives crashes; leases make redelivery safe.

Architecture diagramSmall records in one store, page bytes in another

Read:Learn: Data model design Learn: Storage choice justification Learn: S3 / Blob Storage Learn: Apache Cassandra

Try it: URLs, pages and where they liveGuided practice · Web crawler, Search crawler · Senior

Check yourself

Why not keep the raw page in the fetch row?
Show answerHide answer
Pages are far larger than a record and are read in bulk by the parser. In object storage they are cheap, and a fetch row just points at them.

The interfaces

A crawler has no public API. Its interfaces are the frontier’s calls, the messages between stages and one operator call. Messages carry a page’s location and a stable fetch_id, not the page itself.

The frontier, the stages and the operator

Call or message	Carries	Returns	Notes
`POST /frontier/urls`	Discovered URLs	How many were new	After normalising and the seen check.
`POST /frontier/lease`	Fetcher id, batch size	URLs with `fetch_id` and lease expiry	Only sites whose turn has come.
`POST /frontier/ack`	`fetch_id`, outcome, next due time	`204`	After the page, record and parse task are written.
Parse task	`fetch_id`, URL, page key	Acknowledged by the parser	After the text and links are out.
`PUT /admin/blocks/{site}`	Reason	`204`	Checked before every fetch.

Read:Learn: API contract design Learn: Idempotency and retries

Check yourself

A fetcher’s ack is lost in the network, and the lease expires. What stops the page being stored twice?
Show answerHide answer
The next fetch of that URL carries the same fetch_id, so the page and record are written under the same keys, and the link check stops its links being queued twice.

High-level design

Build the loop one stage at a time; the deep dives then harden the parts an interviewer pushes on.

Fetch a seed page and store it

The frontier holds URLs waiting to be fetched, starting with the seeds. A fetcher only waits on the network, so it hands everything after the download to other stages.

The seed list goes into the frontier.
A fetcher takes the next URL whose site may be fetched now, resolves it through a DNS cache and downloads the page, with time and size limits.
It writes the raw page to object storage under the fetch’s id, plus a fetch record.
It leaves a small parse task with the page’s location, and moves on.

Architecture diagramFetch a seed page and store it

The fetcher stores the raw page, records the fetch and leaves a parse task, then takes its next URL.

Diagram as text

Components: DNS cache (Cache); Seed list (Component); Frontier (Queue); Fetcher (Worker); Website (External API); Parse queue (Queue); Fetch records (NoSQL DB); Object storage (Object store). Connections: Seed list → Frontier (1. seed URLs); Frontier → Fetcher (2. next URL); Fetcher → DNS cache (3. resolve); Fetcher → Website (4. GET); Fetcher → Object storage (5. raw page); Fetcher → Fetch records (6. record); Fetcher → Parse queue (7. parse task), asynchronous.

Read:Learn: Producer-consumer / work queue Learn: S3 / Blob Storage

Try it: From a seed URL to stored textGuided practice · Web crawler, Search crawler · Senior

Keep reading+1 min

What a fetch asks for

Ask for compressed responses, send a User-Agent that names the crawler, follow a few redirects at most, and stop reading at a maximum page size. Record the final URL after redirects too, because it is the one the seen set should know.

Keys for raw pages

Key each stored page by its fetch id, derived from the URL and the crawl cycle, so a repeated fetch overwrites the same object instead of adding a copy. Packing pages into large archive files, such as WARC files, keeps the object count down.

Check yourself

Parse it and send the links back

A parser reads the stored page and extracts its text and links. Which links get in, and how fast each site is fetched, are the first deep dives.

A parser takes a task and reads the raw page from object storage.
It sends the extracted text to the indexer’s queue and records where the text is kept.
It sends the links back toward the frontier, which decides which are new.

Architecture diagramParse it and send the links back

Design a web crawler

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

The numbers that shape the crawl

Core entities

The interfaces

High-level design

Fetch a seed page and store it

What a fetch asks for

Keys for raw pages

Parse it and send the links back

Sign in with Google to read the rest

Practise this

Why messages carry references

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

01Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

02The numbers that shape the crawl

03Core entities

04The interfaces

05High-level design

Fetch a seed page and store it

What a fetch asks for

Keys for raw pages

Parse it and send the links back

Sign in with Google to read the rest

12Practise this

Why messages carry references

Understanding the problem

The numbers that shape the crawl

Core entities

The interfaces

High-level design

Practise this