Web crawler: how the architecture works

How it works, step by step

The walkthrough above, as text. Each step lights up the components named under it.

1. 01

   A crawl starts from seed URLs placed in the URL Frontier with a priority. The frontier is the crawler's to-do list; everything else exists to drain it politely and to refill it with links found on the way.

   Components URL Frontier · Via seeds, revisits

2. 02

   The frontier has priority queues in front and one queue per host behind them, as in the published Mercator design, so any one host's URLs are fetched serially. The Politeness Scheduler takes a URL only from a host whose waiting time has passed, which stops the crawler from hammering a small site.

   Components Politeness Scheduler · Via take next URL

3. 03

   The scheduler dispatches URLs to Fetcher Workers. There are many workers, but each host has at most one request in flight, so total speed comes from crawling many hosts at once rather than one host fast.

   Components Fetcher Workers · Via dispatch

4. 04

   Before fetching a page, the worker checks the host's robots.txt rules in the Robots Cache, fetching and caching the file if needed. RFC 9309 says to avoid using a cached copy for more than 24 hours where possible, to treat a 4xx answer for the file as no restrictions, and to assume complete disallow if the file cannot be reached because of a server error.

   Components Robots Cache · Via allowed? fill

5. 05

   The worker resolves the host name through a caching DNS Resolver. Name resolution is a known bottleneck for crawlers because each new host needs a lookup, so resolvers cache results and run lookups asynchronously.

   Components DNS Resolver · Via resolve

6. 06

   The worker sends the HTTP request with a clear user agent, timeouts, a maximum size and a limit on redirects, and uses conditional headers (such as the ETag from the last visit) on revisits so unchanged pages cost little. Rate-limit or server-error responses make the host's delay longer.

   Components Websites · Via HTTP GET

7. 07

   The page is written to the Raw Content Store, typically packed into large archive files because billions of tiny files would be costly to manage, and its outcome (status, time, ETag) is recorded in Crawl Metadata.

   Components Raw Content Store, Crawl Metadata · Via store page

8. 08

   A parse job goes to the Parser Workers, which extract links and text. Each link is made absolute and canonicalised (for example lower-casing the host and dropping fragments) so that different spellings of one URL are recognised as the same.

   Components Parser Workers · Via parse job

9. 09

   The content-seen test compares a fingerprint of the page with those already stored. The same page often exists under several URLs and on mirrors, and a match avoids processing or storing it again. Fingerprints, not full pages, are stored to keep the test cheap.

   Components Content Fingerprints · Via seen content?

10. 10

    The URL-seen test filters out links already discovered. It stores fixed-size checksums of canonical URLs rather than the text, and puts a cache or probabilistic filter in front, because the set is too large for memory and lookups have little locality.

    Components URL Seen Store · Via seen URL?

11. 11

    Only unseen URLs are added to the frontier, given a priority from signals such as link depth and site importance. Per-host limits on depth and URL count help stop spider traps, where a site generates endless distinct URLs (such as calendar pages).

    Components URL Frontier · Via new URLs

12. 12

    Pages change, so the Crawl Scheduler reads the metadata to decide what is due for a revisit, giving frequently changing pages shorter intervals and static ones longer. Revisits go back into the frontier like any other URL.

    Components Crawl Scheduler, URL Frontier · Via due pages

13. 13

    Downstream consumers such as an indexer read the stored pages in batches. Keeping crawling separate from indexing lets each be scaled and re-run on its own, for example re-indexing without fetching again.

    Components Downstream Consumers · Via batch read

14. 14

    Observability tracks pages fetched per second, error and timeout rates per host, frontier size and the share of duplicates. A growing frontier with a falling fetch rate usually points to a politeness bottleneck or a trap.

    Components Observability, Parser Workers, URL Frontier · Via metrics

Sequence diagram

The same walkthrough as a UML sequence diagram, one lane per layer or group (14 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.

Key components

Related walkthroughs

• URL shortenerHow a link shortener creates unique short keys, serves redirects from a cache in a few milliseconds, and counts clicks off the redirect path.Web fundamentals
• Distributed rate limiterHow an API rate limiter shares token buckets across gateways, answers 429 with a retry hint, fails safely and sheds load when the system is stressed.Web fundamentals
• Distributed cacheHow a distributed cache places keys with consistent hashing, replicates shards, evicts entries, invalidates on writes and resists stampedes.Web fundamentals