{"id":"design-web-crawler","title":"Design a web crawler","sections":[{"id":"overview","title":"Overview","html":"<p>A <strong>web crawler</strong> is a program that downloads web pages, finds the links on them, and then downloads the pages those links point to. For example, the crawler fetches <code>https://example.com/</code>, finds links to <code>/about</code> and <code>https://news.example.org/today</code>, adds both to its to-do list, and later fetches each of them, finding more links as it goes. Search engines run crawlers so they have a copy of the web to search.</p>\n<p>In this chapter the crawler feeds a small search service. A user types <code>hello world</code> and gets back a list of pages, each with a title, a short <strong>snippet</strong> (a sentence or two of text from the page) and a link. The crawler's job is to keep the data behind that list complete and up to date.</p>\n<p>It's a good interview question because the web is hostile to naive code. Pages link in cycles, so a careless crawler visits the same pages forever. Many URLs serve the same content, and some sites generate endless URLs. Every site belongs to someone else, and a crawler that fetches too fast can overload it. And a billion pages refreshed weekly comes to petabytes a month.</p>\n<p>The chapter follows the four steps from <a href=\"/learn/system-design-interview\">The system design interview, step by step</a>. The <a href=\"/learn/design-search-engine\">search engine design</a> covers ranking and the index in more depth. This chapter spends its depth on the crawler.</p>","headings":[]},{"id":"step-1-outline-use-cases-constraints-and-assumptions","title":"Step 1: Outline use cases, constraints and assumptions","html":"<h3 id=\"step-1-outline-use-cases-constraints-and-assumptions\">Step 1: Outline use cases, constraints and assumptions</h3>\n<h4 id=\"use-cases\">Use cases</h4>\n<ul>\n<li><strong>The service crawls a list of URLs.</strong> While doing so it:\n<ul>\n<li>builds a <strong>reverse index</strong> (also called an <strong>inverted index</strong>): a map from each word to the pages that contain it, so a search can find matching pages without reading every page, and</li>\n<li>generates a title and a snippet for each page. They are <strong>static</strong>: generated once per crawl, and the same whatever the search query.</li>\n</ul>\n</li>\n<li><strong>A user enters a search term and sees a list of relevant pages,</strong> with the titles and snippets the crawler generated. We'll only sketch the components for this use case, without going deep.</li>\n<li><strong>The service is highly available.</strong></li>\n</ul>\n<h4 id=\"out-of-scope\">Out of scope</h4>\n<ul>\n<li>Search analytics, such as dashboards of what people search for.</li>\n<li>Personalized search results.</li>\n<li>Computing <strong>PageRank</strong>, a score for how important a page is based on which pages link to it. We order the crawl by a popularity score, but we treat that score as an input we're given. The search engine chapter explains <a href=\"/learn/design-search-engine#link-authority-pagerank\">how PageRank works</a>.</li>\n</ul>\n<h4 id=\"assumptions\">Assumptions</h4>\n<ul>\n<li><strong>Traffic is uneven.</strong> Some searches are very popular, and others are run only once.</li>\n<li><strong>All users are anonymous.</strong></li>\n<li><strong>Search results must come back fast.</strong></li>\n<li><strong>The crawler must not get stuck in an infinite loop.</strong> The web is a graph with cycles: page A links to B, B links to C, and C links back to A. A crawler that doesn't remember where it has been goes around that cycle forever.</li>\n<li><strong>There are 1 billion links to crawl.</strong></li>\n<li><strong>Pages are recrawled regularly to stay fresh,</strong> on average once a week, and popular sites more often. That gives about <strong>4 billion fetches a month</strong> (1 billion pages × about 4 weeks).</li>\n<li><strong>The average stored size of a page is 500 KB.</strong> To keep it simple, a changed page counts the same as a new one: every fetch stores a full copy.</li>\n<li><strong>There are 100 billion searches a month.</strong></li>\n<li><strong>We build from ordinary parts:</strong> databases, queues and services, without an existing search platform such as <a href=\"https://solr.apache.org/\" target=\"_blank\" rel=\"noopener noreferrer\">Apache Solr</a> or crawler such as <a href=\"https://nutch.apache.org/\" target=\"_blank\" rel=\"noopener noreferrer\">Apache Nutch</a>.</li>\n</ul>\n<p>The estimates below need a few more assumptions of our own. State them out loud in an interview:</p>\n<ul>\n<li>An average page has <strong>50 outgoing links</strong>.</li>\n<li>A page's visible text, without markup, scripts and styles, averages <strong>10 KB</strong>.</li>\n<li>A title, snippet and URL take about <strong>500 bytes</strong>, so a page of 10 results is about <strong>5 KB</strong>.</li>\n<li>Search traffic peaks at about <strong>3 times</strong> its average.</li>\n<li>An average fetch, including DNS lookup, connecting and downloading, takes about <strong>2 seconds</strong>.</li>\n</ul>\n<h4 id=\"calculate-usage\">Calculate usage</h4>\n<p>A 30-day month has 30 × 86,400 = 2,592,000 seconds, about 2.6 million.</p>\n<p><strong>Crawling</strong></p>\n<ul>\n<li>Fetch rate: 4,000,000,000 ÷ 2,592,000 s ≈ <strong>1,540 pages a second</strong>. We control the crawl, so it can run close to this average all day, with no peak.</li>\n<li>Inbound bandwidth: 1,540 × 500 KB ≈ 770 MB a second, which is 770 × 8 ≈ <strong>6.2 Gbit/s</strong>, around the clock.</li>\n<li>Fetches in flight: by <a href=\"/learn/performance-and-latency#littles-law\">Little's Law</a>, that's the rate times the time each one takes: 1,540 a second × 2 s ≈ <strong>3,100 open fetches</strong>.</li>\n<li>New links found: 1,540 pages a second × 50 links ≈ <strong>77,000 links a second</strong>, each checked against the URLs we already know.</li>\n</ul>\n<p><strong>Storage</strong></p>\n<ul>\n<li>Page content: 4,000,000,000 × 500 KB = 2,000,000,000,000 KB = <strong>2 PB a month</strong>.</li>\n<li>Over 3 years: 2 PB × 36 months = <strong>72 PB</strong>, if we keep every copy of every page.</li>\n<li>Latest copy of each page only: 1,000,000,000 × 500 KB = <strong>500 TB</strong>.</li>\n<li>Titles and snippets: 1,000,000,000 × 500 bytes = <strong>500 GB</strong>.</li>\n<li>Reverse index: 1,000,000,000 × 10 KB of text = 10 TB of text. A compressed inverted index is usually smaller than the text it indexes, so budget <strong>10 TB</strong>.</li>\n<li>Crawl bookkeeping, at about 200 bytes per URL: 1,000,000,000 × 200 bytes = <strong>200 GB</strong>.</li>\n</ul>\n<p><strong>Searching</strong></p>\n<ul>\n<li>Search rate: 100,000,000,000 ÷ 2,592,000 s ≈ <strong>38,600 a second</strong> on average, and 3 × 38,600 ≈ <strong>116,000 a second</strong> at peak.</li>\n<li>Outbound bandwidth: 38,600 × 5 KB ≈ 193 MB a second, about <strong>1.5 Gbit/s</strong>.</li>\n<li>Reads to writes: 38,600 searches for every 1,540 pages crawled, roughly <strong>25 to 1</strong>.</li>\n</ul>\n<p><strong>A sanity check.</strong> Common Crawl, a nonprofit that publishes a free copy of the web, reported that its <a href=\"https://commoncrawl.org/blog/august-2026-crawl-archive-now-available\" target=\"_blank\" rel=\"noopener noreferrer\">August 2026 crawl</a> fetched 2.14 billion pages between August 7 and August 20, with 360 TiB of uncompressed content. That's 2,140,000,000 ÷ (14 × 86,400 s) ≈ 1,770 pages a second, close to our 1,540. It's also about 185 KB a page (360 TiB ≈ 396,000,000,000,000 bytes ÷ 2.14 billion), so our 500 KB is generous by almost 3 times. That's fine for a capacity plan.</p>\n<p><strong>What these numbers tell us.</strong> Search and crawling are two systems with different shapes. Search is read-heavy and latency-sensitive at tens of thousands of requests a second, so it needs caching and replicas. Crawling is a steady, throughput-bound job limited by bandwidth (6.2 Gbit/s), open connections (3,100) and \"have we seen this URL?\" checks (77,000 a second). Page content dominates storage, so the retention policy is the biggest storage decision: 72 PB for every version over three years, or 500 TB for the latest only. The index and snippets are small by comparison.</p>","headings":[{"id":"step-1-outline-use-cases-constraints-and-assumptions","title":"Step 1: Outline use cases, constraints and assumptions","depth":3},{"id":"use-cases","title":"Use cases","depth":4},{"id":"out-of-scope","title":"Out of scope","depth":4},{"id":"assumptions","title":"Assumptions","depth":4},{"id":"calculate-usage","title":"Calculate usage","depth":4}]},{"id":"step-2-create-a-high-level-design","title":"Step 2: Create a high-level design","html":"<h3 id=\"step-2-create-a-high-level-design\">Step 2: Create a high-level design</h3>\n<figure class=\"diagram\" data-text=\"User -> Web server (reverse proxy); Web server (reverse proxy) -> Query API; Query API -> Document service (titles, snippets); Query API -> Reverse index service (matching pages); Job queues -> Document service; Job queues -> Reverse index service; Crawler service -> Job queues (index jobs); Crawler service <-> NoSQL database; Crawler service -> Web sites (fetch)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -10 800 333\" width=\"800\" role=\"img\" aria-label=\"First design: a crawler that feeds a search service\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"400\" y1=\"69\" x2=\"400\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"560\" y1=\"135\" x2=\"560\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"489\" y1=\"162\" x2=\"473\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"560\" y1=\"253\" x2=\"560\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"487\" y1=\"282\" x2=\"473\" y2=\"282\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"282\" x2=\"647\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">User</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Web server</tspan><tspan x=\"240\" y=\"54.5\">(reverse proxy)</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"46.5\">Query API</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Document</tspan><tspan x=\"560\" y=\"54.5\">service</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"158.5\">Reverse index</tspan><tspan x=\"400\" y=\"174.5\">service</tspan></text><rect class=\"dg-node dg-queue\" x=\"492\" y=\"138\" width=\"136\" height=\"48\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"612\" y1=\"146\" x2=\"612\" y2=\"178\"></line><line class=\"dg-line\" x1=\"602\" y1=\"146\" x2=\"602\" y2=\"178\"></line><line class=\"dg-line\" x1=\"592\" y1=\"146\" x2=\"592\" y2=\"178\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"546\" y=\"166.5\">Job queues</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"278.5\">Crawler</tspan><tspan x=\"560\" y=\"294.5\">service</tspan></text><path class=\"dg-node dg-store\" d=\"M332,260 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,260 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"289.5\">NoSQL database</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"258\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"286.5\">Web sites</tspan></text><rect class=\"dg-edge-bg\" x=\"424.4\" y=\"-6\" width=\"109.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"7\" text-anchor=\"middle\">titles, snippets</text><rect class=\"dg-edge-bg\" x=\"351.6\" y=\"91\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"400\" y=\"104\" text-anchor=\"middle\">matching pages</text><rect class=\"dg-edge-bg\" x=\"524\" y=\"213\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"226\" text-anchor=\"middle\">index jobs</text><rect class=\"dg-edge-bg\" x=\"618.5\" y=\"234\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"639\" y=\"247\" text-anchor=\"middle\">fetch</text></svg></div><figcaption>First design: a crawler that feeds a search service</figcaption></figure>\n<ul>\n<li><strong>Crawler service:</strong> a loop that takes the next URL, fetches the page and hands it off for indexing. It's the only part that talks to the outside web.</li>\n<li><strong>NoSQL database:</strong> holds <code>links_to_crawl</code>, the to-do list ranked by priority, and <code>crawled_links</code>, a record of every page fetched with its signature and fetch time. Every access is by URL, which suits a key-value store.</li>\n<li><strong>Job queues:</strong> let the crawler hand off pages without waiting for indexing, which is slower than fetching. See <a href=\"/learn/queues-and-async\">queues and async work</a>.</li>\n<li><strong>Reverse index service:</strong> turns page text into the word-to-pages index, and answers \"which pages contain these words?\"</li>\n<li><strong>Document service:</strong> generates and stores each page's static title and snippet, and returns them by page ID.</li>\n<li><strong>Web server:</strong> a <a href=\"/learn/reverse-proxy-and-api-gateway#what-a-reverse-proxy-is\">reverse proxy</a> that accepts connections from users and forwards requests.</li>\n<li><strong>Query API:</strong> parses the search text, asks the reverse index for matching pages, and asks the document service for their titles and snippets.</li>\n</ul>","headings":[{"id":"step-2-create-a-high-level-design","title":"Step 2: Create a high-level design","depth":3}]},{"id":"step-3-design-core-components","title":"Step 3: Design core components","html":"<h3 id=\"step-3-design-core-components\">Step 3: Design core components</h3>\n<h4 id=\"use-case-service-crawls-a-list-of-urls\">Use case: Service crawls a list of URLs</h4>\n<h5 id=\"the-seed-list-and-the-two-tables\">The seed list and the two tables</h5>\n<p>A crawler needs a starting list, the <strong>seed list</strong>. We assume we're given an initial <code>links_to_crawl</code> ranked by overall site popularity. If not, we can seed it with popular sites that link out to a lot of other content, such as large directories and news front pages, and discover the rest by following links.</p>\n<p><code>links_to_crawl</code> is the <strong>frontier</strong>: the URLs waiting to be fetched, each with a priority score. The crawler always takes the highest-scoring URL next. A <a href=\"https://redis.io/docs/latest/develop/data-types/sorted-sets/\" target=\"_blank\" rel=\"noopener noreferrer\">Redis sorted set</a> fits: unique strings, each with a numeric score, kept in score order. Adding a URL (<code>ZADD</code>) costs O(log N), and <a href=\"https://redis.io/docs/latest/commands/zpopmax/\" target=\"_blank\" rel=\"noopener noreferrer\"><code>ZPOPMAX</code></a> removes and returns the highest-scoring member, also in O(log N).</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>links_to_crawl   (sorted set, score = priority)</span></span>\n<span class=\"line\"><span>  https://en.wikipedia.org/wiki/Main_Page        0.97</span></span>\n<span class=\"line\"><span>  https://www.bbc.com/news                        0.95</span></span>\n<span class=\"line\"><span>  https://example.com/about                       0.31</span></span></code></pre>\n<p><code>crawled_links</code> records what we have fetched, keyed by URL, with a second index on <code>signature</code>:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>crawled_links    (key-value, key = normalized URL)</span></span>\n<span class=\"line\"><span>  url            https://example.com/about</span></span>\n<span class=\"line\"><span>  signature      9f2c41a07be3d655        # fingerprint of the page content</span></span>\n<span class=\"line\"><span>  crawled_at     2026-10-12T08:14:03Z</span></span>\n<span class=\"line\"><span>  next_crawl_at  2026-10-19T08:14:03Z</span></span>\n<span class=\"line\"><span>  change_count   3                       # recrawls that found new content</span></span></code></pre>\n<p>Relational or NoSQL? Every access is by one URL, there are no joins, and a billion rows must be split across machines, which suits a key-value store. See <a href=\"/learn/nosql-databases#sql-or-nosql-making-the-choice\">SQL or NoSQL</a>.</p>\n<h5 id=\"the-crawler-loop\">The crawler loop</h5>\n<p>For each URL, the crawler:</p>\n<ol>\n<li>Takes the highest-priority URL from <code>links_to_crawl</code>.</li>\n<li>Checks that the site's <code>robots.txt</code> allows it (see <a href=\"/learn/design-web-crawler#politeness-robotstxt-and-crawl-rate-per-host\">politeness</a>).</li>\n<li>Fetches the page.</li>\n<li>Computes the page's <strong>signature</strong>, a short fingerprint of its content.</li>\n<li>Looks in <code>crawled_links</code> for a page with the same or a very similar signature.\n<ul>\n<li>If there is one, this URL is a copy of a page we already have. It isn't indexed again. It goes back into <code>links_to_crawl</code> with a lower priority, so the crawler doesn't keep returning to it. This also stops a cycle of links between copies from trapping the crawler.</li>\n<li>If not, it adds a job to the reverse index service's queue and one to the document service's queue, and records the URL and signature in <code>crawled_links</code>.</li>\n</ul>\n</li>\n<li>Extracts the page's links, normalizes them, and adds the new ones to <code>links_to_crawl</code>.</li>\n</ol>\n<p>As a Python sketch, with storage hidden behind a small class:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> time</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> urllib.parse </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> urljoin, urlsplit, urlunsplit</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">WEEK</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 7</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 86_400</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> normalize</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(url):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Make equal URLs compare equal: lowercase scheme and host,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    drop default ports and #fragments.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    parts </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> urlsplit(url)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    scheme </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> parts.scheme.lower()</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    host </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (parts.hostname </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">or</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">).lower()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> parts.port </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> parts.port </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">!=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"http\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">80</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"https\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">443</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}.get(scheme):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        host </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">host</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">parts.port</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> urlunsplit((scheme, host, parts.path </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">or</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"/\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, parts.query, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> signature</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(text):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Exact-content fingerprint. Near-duplicates need more (see below).\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    words </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \" \"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(text.lower().split())</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib.sha256(words.encode()).hexdigest()[:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">16</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> LinkStore</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"links_to_crawl in a Redis sorted set, crawled_links in a key-value store.\"\"\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, redis, kv):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.redis, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.kv </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> redis, kv</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> next_url</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        popped </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.redis.zpopmax(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"links_to_crawl\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> popped[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">][</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> popped </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> add_if_new</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, url, priority):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.kv.get(url) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">            # nx=True: never overwrite a URL that's already queued</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">            self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.redis.zadd(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"links_to_crawl\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {url: priority}, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">nx</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> demote</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, url, priority</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.01</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.redis.zadd(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"links_to_crawl\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {url: priority})</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> copy_of_another</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, url, sig):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        other </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.kv.find_by_signature(sig)   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># URL stored with this signature</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> other </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> other </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">!=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> url</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> record</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, url, sig):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        now </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> time.time()</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.kv.put(url, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"signature\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: sig, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"crawled_at\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: now,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                          \"next_crawl_at\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: now </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> WEEK</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">})</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> crawl</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(store, fetcher, robots, index_queue, doc_queue, score):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    while</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        url </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> store.next_url()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> url </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            time.sleep(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)                   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># frontier empty for now</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            continue</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> robots.allowed(url):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            continue</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        page </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> fetcher.get(url)             </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># waits for this host's turn</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> page </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:                    </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># timeout, error, not HTML</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            continue</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        sig </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> signature(page.text)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> store.copy_of_another(url, sig):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            store.demote(url)               </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># a copy: don't index it again</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            continue</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        index_queue.put({</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"url\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: url, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: page.text})</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        doc_queue.put({</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"url\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: url, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"html\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: page.html})</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        store.record(url, sig)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> href </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> page.links:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            link </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> normalize(urljoin(url, href))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            store.add_if_new(link, score(link))</span></span></code></pre>\n<p>The loop checks for repeats at two moments:</p>\n<ul>\n<li><strong>Before fetching, by URL.</strong> <code>add_if_new</code> refuses URLs already queued or crawled. This is cheap and stops most cycles: when C links back to A, A is already in <code>crawled_links</code>.</li>\n<li><strong>After fetching, by content.</strong> A signature needs the content, so this check must wait for the download. It catches one page reached through different URLs, such as <code>http://</code> and <code>https://</code> versions, or an article with and without a tracking parameter.</li>\n</ul>\n<p>One production detail: <code>ZPOPMAX</code> removes the URL before it's fetched, so a crawler that crashes mid-fetch loses it. The fix is a lease. Move the URL to an \"in progress\" set with a deadline, and put it back if it isn't recorded in time, like a <a href=\"/learn/queues-and-async#acknowledgements-and-visibility-timeouts\">visibility timeout</a> in a message queue.</p>\n<h5 id=\"what-happens-to-the-jobs\">What happens to the jobs</h5>\n<p>The <strong>reverse index service</strong> reads its queue. For each page it splits the text into lowercase words and adds the page's ID to each word's list of pages, called a <strong>posting list</strong>:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>\"hello\"  -> [17, 42, 108, ...]</span></span>\n<span class=\"line\"><span>\"world\"  -> [42, 99, 108, ...]</span></span></code></pre>\n<p>To answer <code>hello world</code>, it intersects the lists (pages 42 and 108), ranks the matches and returns the best. Updating posting lists one page at a time is slow, so real indexers batch pages into new index segments and merge them in the background. See <a href=\"/learn/design-search-engine#the-inverted-index-pipeline\">the inverted index pipeline</a>.</p>\n<p>The <strong>document service</strong> reads its own queue. It takes the title from the HTML <code>&#x3C;title></code> element, and the snippet from the meta description or the first 160 or so characters of the main text, and stores them by page ID:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>documents</span></span>\n<span class=\"line\"><span>  doc_id      42</span></span>\n<span class=\"line\"><span>  url         https://foo.com</span></span>\n<span class=\"line\"><span>  title       foo's title</span></span>\n<span class=\"line\"><span>  snippet     foo's snippet: the first sentence or two of the page.</span></span>\n<span class=\"line\"><span>  crawled_at  2026-10-12T08:14:03Z</span></span></code></pre>\n<p>Because snippets are static, all the work happens at crawl time, and a search only needs a lookup by ID.</p>\n<h4 id=\"handling-duplicates\">Handling duplicates</h4>\n<p>There are two kinds of duplicates, and they need different tools.</p>\n<h5 id=\"duplicate-urls\">Duplicate URLs</h5>\n<p>The same URL arrives many times: from every page that links to it, from seed lists merged with discovered links, and from crawler restarts. Before a batch of URLs goes into the frontier, we remove repeats.</p>\n<p>For a small list, <code>sort -u urls.txt</code> on one machine is enough. With 1 billion links we use <strong>MapReduce</strong>, a way to run a job over data spread across many machines. A <em>map</em> step turns each record into key-value pairs. The framework groups all pairs with the same key onto one machine (the <em>shuffle</em>). A <em>reduce</em> step combines each group into an output. The <a href=\"/learn/batch-processing#mapreduce-the-foundational-model\">batch processing chapter</a> explains the model.</p>\n<p>Here the key is the normalized URL. Every copy of a URL lands in the same group, and the reducer writes it out once.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> collections </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> defaultdict</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> map_urls</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(lines):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Map: emit (normalized URL, 1) for every line.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> line </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> lines:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> line.strip():</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            yield</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> normalize(line.strip()), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> shuffle</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(pairs):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"What the framework does between map and reduce.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    groups </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> defaultdict(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">list</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> key, value </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pairs:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        groups[key].append(value)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> groups</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> reduce_urls</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(groups):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Reduce: one output per distinct URL. The count is how often</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    the URL was linked, a useful signal for its priority.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> url, ones </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> groups.items():</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        yield</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> url, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(ones)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">lines </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"https://Example.com/a#top\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"https://example.com/a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"https://example.com/b\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(reduce_urls(shuffle(map_urls(lines)))))</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># {'https://example.com/a': 2, 'https://example.com/b': 1}</span></span></code></pre>\n<p>Watch the reducer. A tempting version emits only URLs whose count is 1. That silently drops every URL that appears more than once, which is the opposite of what we want. Emit every key once, whatever its count.</p>\n<p>Between batch runs, the crawler still needs a fast \"seen this URL?\" check for 77,000 links a second. A <strong>Bloom filter</strong> is a compact bit array that answers \"definitely not seen\" or \"probably seen\". At a 1% false-positive rate it needs about 9.6 bits per URL (−ln 0.01 ÷ (ln 2)² ≈ 4.6 ÷ 0.48), so 1,000,000,000 × 9.6 bits ≈ 1.2 GB, which fits in one machine's memory. The cost: about 1 new URL in 100 is wrongly reported as seen and skipped. The <a href=\"/learn/cache#cache-penetration\">cache chapter</a> uses a Bloom filter the same way.</p>\n<h5 id=\"duplicate-content\">Duplicate content</h5>\n<p>An exact fingerprint, like the SHA-256 hash in the loop, only catches pages that match character for character. Two copies of an article that differ by a timestamp get completely different hashes. Such copies are common: <em>Introduction to Information Retrieval</em> notes that <a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html\" target=\"_blank\" rel=\"noopener noreferrer\">by some estimates as many as 40% of web pages</a> duplicate other pages.</p>\n<p>To catch near-duplicates, compare pages by similarity:</p>\n<ol>\n<li>Break each page into <strong>shingles</strong>, every run of k consecutive words. With k = 4, \"the quick brown fox jumps\" gives \"the quick brown fox\" and \"quick brown fox jumps\".</li>\n<li>Measure overlap with the <strong>Jaccard index</strong>: the size of the intersection of two shingle sets divided by the size of their union. Identical pages score 1.0. Pages above a threshold such as 0.9 are near-duplicates.</li>\n<li>Comparing every pair of a billion pages is impossible, so shrink each page's shingles into a fixed-size <strong>sketch</strong> with <strong>MinHash</strong>: hash every shingle with many hash functions and keep the smallest value for each. The chance that two pages share the minimum for one function equals their Jaccard index, so the fraction of matching minimums estimates it.</li>\n</ol>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> shingles</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(text, k</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    words </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> text.lower().split()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\" \"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(words[i:i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> k]) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(words) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> jaccard</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(a, b):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(a </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> b) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(a </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">|</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> b) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (a </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">or</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> b) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1.0</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> minhash</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(shingle_set, num_hashes</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">200</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.from_bytes(hashlib.blake2b(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">seed</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.encode(),</span></span>\n<span class=\"line\"><span style=\"color:#E36209;--shiki-dark:#FFAB70\">                                               digest_size</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">).digest(), </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"big\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> shingle_set)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> seed </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(num_hashes)]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> estimated_similarity</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(sketch_a, sketch_b):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> y </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x, y </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> zip</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(sketch_a, sketch_b)) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(sketch_a)</span></span></code></pre>\n<p>The sketch is 200 numbers however long the page is, so it fits in <code>crawled_links</code> beside the signature. <strong>Cosine similarity</strong> is an alternative: it treats each page as a vector of word counts and measures the angle between two vectors, so it weighs how often words appear and ignores their order.</p>\n<p>Large crawlers often use <strong>SimHash</strong>, which packs a page into one 64-bit fingerprint such that similar pages differ in only a few bits. The search engine chapter explains <a href=\"/learn/design-search-engine#near-duplicate-detection-with-simhash\">SimHash and how Google used it</a>. With any of these, the loop's \"seen similar content?\" check becomes a search for fingerprints close to the new one.</p>\n<h4 id=\"determining-when-to-update-the-crawl-results\">Determining when to update the crawl results</h4>\n<p>Pages change, so they must be fetched again. Each row in <code>crawled_links</code> has a <code>crawled_at</code> timestamp. A simple policy: after a default period, say one week, every page is due again, and popular or frequently updated sites sooner. A scheduler finds rows whose <code>next_crawl_at</code> has passed and puts those URLs back in <code>links_to_crawl</code>.</p>\n<p>A better policy learns each page's rate of change. Every recrawl tells us whether the signature changed, so over several visits we can estimate the mean time between changes and set the interval from it:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">DAY</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 86_400</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">MIN_INTERVAL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">MAX_INTERVAL</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> DAY</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">30</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> DAY</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> next_interval</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(previous, changed):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Recrawl sooner after a change, later after no change.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    interval </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> previous </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> changed </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> previous </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1.5</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">MIN_INTERVAL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">MAX_INTERVAL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, interval))</span></span></code></pre>\n<p>It's tempting to recrawl every page in direct proportion to how often it changes. Cho and Garcia-Molina tested this on 270 web sites over four months, in <a href=\"https://www.kth.se/social/files/554f1d5bf27654514fd00393/synchronizing-a-database-to-improve-freshness.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Synchronizing a database to improve freshness</a> (SIGMOD 2000). Revisiting every page at the same rate kept the copy fresher than revisiting in proportion to change rate, because a page that changes many times a day goes stale again almost at once. With their crawl budget and real change data, the predicted share of fresh pages was 0.12 for the proportional policy, 0.57 for the uniform one and 0.62 for their optimal one. Their conclusion: \"to improve freshness, we should penalize the elements that change too often.\" That's why <code>next_interval</code> has a floor. A search engine may still revisit a popular, fast-changing page often, because a stale copy of an important page affects more users.</p>\n<p>Two web standards make recrawling cheaper:</p>\n<ul>\n<li><strong>Conditional requests.</strong> Send the <code>ETag</code> or last-modified date from the previous fetch in an <code>If-None-Match</code> or <code>If-Modified-Since</code> header. An unchanged page comes back as <a href=\"https://www.rfc-editor.org/rfc/rfc9110#section-15.4.5\" target=\"_blank\" rel=\"noopener noreferrer\"><code>304 Not Modified</code></a> with no body, saving the 500 KB download. Our storage estimate counted every refresh as a new page, so it's pessimistic.</li>\n<li><strong>Sitemaps.</strong> A site can list its URLs in an XML sitemap with a <code>&#x3C;lastmod></code> date for each. Google says it <a href=\"https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap\" target=\"_blank\" rel=\"noopener noreferrer\">uses <code>&#x3C;lastmod></code> if it's consistently accurate, and ignores <code>&#x3C;changefreq></code> and <code>&#x3C;priority></code></a>. Site-reported dates help, but the crawler should check them against what it sees.</li>\n</ul>\n<h4 id=\"politeness-robotstxt-and-crawl-rate-per-host\">Politeness: robots.txt and crawl rate per host</h4>\n<p>Every fetch costs someone else's server some work, and to a small site a crawler sending hundreds of requests a second looks like an attack. Politeness has two parts: obeying the site's rules, and limiting the rate.</p>\n<h5 id=\"robotstxt\">robots.txt</h5>\n<p>The <strong>Robots Exclusion Protocol</strong> lets a site tell crawlers which paths they may fetch. It was standardized as <a href=\"https://www.rfc-editor.org/rfc/rfc9309\" target=\"_blank\" rel=\"noopener noreferrer\">RFC 9309</a> in September 2022. The rules that shape our design:</p>\n<ul>\n<li><strong>Location and groups.</strong> The rules live at <code>/robots.txt</code> at the top of the host, such as <code>https://example.com/robots.txt</code>. The file is split into groups that start with <code>user-agent</code> lines. A crawler uses the group matching its name, compared case-insensitively, and otherwise the <code>*</code> group.</li>\n<li><strong>Matching.</strong> Each group has <code>allow</code> and <code>disallow</code> path rules. <code>*</code> matches any run of characters, and <code>$</code> marks the end of the path. The most specific matching rule (the one with the most characters) wins, and <code>allow</code> should win a tie. If nothing matches, the URL is allowed.</li>\n<li><strong>Errors.</strong> A 4xx response, such as 404, means the file is \"unavailable\", and the crawler may fetch anything. A 5xx response or an unreachable server means the crawler \"MUST assume complete disallow\". If that lasts a long time (the RFC's example is 30 days), the crawler may treat the file as unavailable or keep using a cached copy.</li>\n<li><strong>Limits.</strong> Crawlers should follow at least five redirects to the file, must parse at least 500 KiB of it, and \"SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable.\"</li>\n<li><strong>It isn't security.</strong> The RFC says it \"is not a substitute for valid content security measures.\" A polite crawler honors it anyway.</li>\n</ul>\n<p>An example:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>User-agent: *</span></span>\n<span class=\"line\"><span>Disallow: /search</span></span>\n<span class=\"line\"><span>Disallow: /*?sessionid=</span></span>\n<span class=\"line\"><span>Allow: /search/about$</span></span></code></pre>\n<p>Our crawler may fetch <code>/news/today</code>. It may not fetch <code>/search?q=x</code> or <code>/cart?sessionid=abc</code>. It may fetch <code>/search/about</code>, because that <code>Allow</code> rule is longer than <code>Disallow: /search</code>.</p>\n<p>So the design needs a <strong>robots cache</strong>: parsed rules per host, with the time they were fetched. A host's <code>robots.txt</code> is fetched before any other URL on it and refreshed at most every 24 hours. Many URLs share a host, so <em>Introduction to Information Retrieval</em> finds such a cache <a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/crawler-architecture-1.html\" target=\"_blank\" rel=\"noopener noreferrer\">still \"highly effective\"</a>.</p>\n<p>Many files contain <code>Crawl-delay: 10</code>, asking for 10 seconds between requests. RFC 9309 doesn't define it, though it lets crawlers interpret extra records, and Google <a href=\"https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt\" target=\"_blank\" rel=\"noopener noreferrer\">doesn't support <code>crawl-delay</code></a>. Our crawler can honor it as a minimum gap, but it can't rely on sites setting it.</p>\n<h5 id=\"crawl-rate-per-host\">Crawl rate per host</h5>\n<p>So the crawler needs its own per-host limit. The textbook design, from the Mercator crawler and described in <a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/the-url-frontier-1.html\" target=\"_blank\" rel=\"noopener noreferrer\"><em>Introduction to Information Retrieval</em></a>, has two rules:</p>\n<ol>\n<li>Keep <strong>only one connection open to a host at a time</strong>.</li>\n<li>Before the next request to a host, wait about <strong>ten times as long as the last fetch took</strong>. A site that took 0.5 s to answer is asked again 5 s later, so a struggling site is automatically visited less.</li>\n</ol>\n<p>The frontier implements this with two layers of queues. <strong>Front queues</strong> order URLs by priority. Each <strong>back queue</strong> holds URLs from a single host, and a heap orders hosts by the earliest time each may be contacted again. In our design, <code>links_to_crawl</code> plays the front queues, and each crawler keeps back queues and the heap in memory. The search engine chapter <a href=\"/learn/design-search-engine#the-crawl-frontier-priority-and-politeness\">walks through this frontier</a>.</p>\n<p>The crawler should also back off when a site pushes back. A <code>429 Too Many Requests</code> or <code>503 Service Unavailable</code> means \"too fast\", and a <a href=\"https://www.rfc-editor.org/rfc/rfc9110#section-10.2.3\" target=\"_blank\" rel=\"noopener noreferrer\"><code>Retry-After</code></a> header says how long to wait. Google documents the same signal from the site's side: <a href=\"https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate\" target=\"_blank\" rel=\"noopener noreferrer\">returning 500, 503 or 429 makes Googlebot slow down</a> across the whole hostname.</p>\n<p>Politeness changes the capacity math. At one request per host every 10 seconds, each host supplies at most 0.1 pages a second. To reach 1,540 pages a second, the crawler must work on at least 1,540 ÷ 0.1 = <strong>15,400 hosts at once</strong>. That's why step 4 shards the crawler by host.</p>\n<h4 id=\"crawler-traps\">Crawler traps</h4>\n<p>A <strong>crawler trap</strong> (or spider trap) is a set of URLs that never runs out. <em>Introduction to Information Retrieval</em> describes <a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/features-a-crawler-must-provide-1.html\" target=\"_blank\" rel=\"noopener noreferrer\">pages \"that mislead crawlers into getting stuck fetching an infinite number of pages\"</a>, and notes that many are accidents of \"faulty website development\". Google's <a href=\"https://developers.google.com/search/docs/crawling-indexing/url-structure\" target=\"_blank\" rel=\"noopener noreferrer\">URL structure guidelines</a> list common causes:</p>\n<ul>\n<li><strong>Calendars</strong> that link to the next and previous day with no end: <code>/calendar?date=2026-10-20</code> links to <code>2026-10-21</code>, and so on forever.</li>\n<li><strong>Filters and sort orders</strong> that combine, so a shop with 20 filters produces millions of listing URLs.</li>\n<li><strong>Session IDs in URLs</strong>, which give every visit a fresh set of URLs.</li>\n<li><strong>Broken relative links</strong>, which produce <code>/docs/section/section/section/...</code> if the server answers every path with a page instead of a 404.</li>\n</ul>\n<p>Some responses never end, such as an audio stream. Common Crawl <a href=\"https://commoncrawl.org/blog/august-2026-crawl-archive-now-available\" target=\"_blank\" rel=\"noopener noreferrer\">truncates every fetch at 5 MiB</a> for this reason.</p>\n<p>Content checks catch some traps, but a calendar's pages really do differ. Add limits:</p>\n<ul>\n<li><strong>Cap each download</strong> in size and time.</li>\n<li><strong>Cap URLs</strong> by length, path depth and repeated path segments.</li>\n<li><strong>Budget each host:</strong> a page budget per crawl cycle, scaled by popularity. A small site that suddenly has 10 million queued URLs is much more likely a trap than a success.</li>\n<li><strong>Strip known session parameters</strong> while normalizing URLs.</li>\n<li><strong>Rely on priority.</strong> Pages deep in a trap have few links from other sites, so a good score sinks them.</li>\n</ul>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> urllib.parse </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> urlsplit</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> looks_like_trap</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(url, max_len</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, max_depth</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">15</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, max_repeats</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    segments </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> urlsplit(url).path.split(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"/\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(url) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_len</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            or</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(segments) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_depth</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            or</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> any</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(segments.count(s) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_repeats </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> set</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(segments)))</span></span></code></pre>\n<h4 id=\"use-case-user-inputs-a-search-term-and-sees-a-list-of-relevant-pages-with-titles-and-snippets\">Use case: User inputs a search term and sees a list of relevant pages with titles and snippets</h4>\n<ol>\n<li>The <strong>client</strong> sends the search to the <strong>web server</strong>, running as a <a href=\"/learn/reverse-proxy-and-api-gateway#what-a-reverse-proxy-does-for-you\">reverse proxy</a>.</li>\n<li>The web server forwards it to a <strong>Query API</strong> server.</li>\n<li>The Query API parses the query: it removes markup, breaks the text into terms, fixes typos, normalizes capitalization, and converts the query to boolean operations.</li>\n<li>It asks the <strong>reverse index service</strong> for matching documents. That service ranks the matches and returns the IDs of the best ones.</li>\n<li>It asks the <strong>document service</strong> for their titles and snippets, and returns the results.</li>\n</ol>\n<p>Step 3 on the input <code>&#x3C;b>Helo&#x3C;/b> WORLD</code>:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Step</th><th>Result</th></tr></thead><tbody><tr><td>Remove markup</td><td><code>Helo WORLD</code></td></tr><tr><td>Break into terms</td><td><code>Helo</code>, <code>WORLD</code></td></tr><tr><td>Fix typos</td><td><code>Hello</code>, <code>WORLD</code> (the index knows \"hello\" is common and \"helo\" is rare)</td></tr><tr><td>Normalize capitalization</td><td><code>hello</code>, <code>world</code></td></tr><tr><td>Boolean form</td><td><code>hello AND world</code></td></tr></tbody></table></div>\n<p>The public API is <a href=\"/learn/protocols-and-apis#rest-resources-and-representations\">REST</a>: a <code>GET</code> with the query in the URL.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>$ curl \"https://search.com/api/v1/search?query=hello+world\"</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">{</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">  \"query\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"hello world\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">  \"results\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    {</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"title\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"foo's title\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"snippet\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"foo's snippet\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"link\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"https://foo.com\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">},</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    {</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"title\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"bar's title\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"snippet\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"bar's snippet\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"link\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"https://bar.com\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">},</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    {</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"title\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"baz's title\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"snippet\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"baz's snippet\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"link\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"https://baz.com\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">  ],</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">  \"next_page\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"/api/v1/search?query=hello+world&#x26;page=2\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>Because it's a <code>GET</code> with everything in the URL, browsers, CDNs and our own cache can all cache it by the normalized query.</p>\n<p>Between the Query API and the internal services, use <a href=\"/learn/protocols-and-apis#rpc-calling-a-function-on-another-machine\">RPC</a>, such as gRPC. The calls are fixed functions (\"find documents for these terms\", \"get documents by ID\") between services we own, and compact binary messages over persistent connections save time on a path that must be fast.</p>","headings":[{"id":"step-3-design-core-components","title":"Step 3: Design core components","depth":3},{"id":"use-case-service-crawls-a-list-of-urls","title":"Use case: Service crawls a list of URLs","depth":4},{"id":"handling-duplicates","title":"Handling duplicates","depth":4},{"id":"determining-when-to-update-the-crawl-results","title":"Determining when to update the crawl results","depth":4},{"id":"politeness-robotstxt-and-crawl-rate-per-host","title":"Politeness: robots.txt and crawl rate per host","depth":4},{"id":"crawler-traps","title":"Crawler traps","depth":4},{"id":"use-case-user-inputs-a-search-term-and-sees-a-list-of-relevant-pages-with-titles-and-snippets","title":"Use case: User inputs a search term and sees a list of relevant pages with titles and snippets","depth":4}]},{"id":"step-4-scale-the-design","title":"Step 4: Scale the design","html":"<h3 id=\"step-4-scale-the-design\">Step 4: Scale the design</h3>\n<figure class=\"diagram\" data-text=\"User -> DNS; User -> Load balancer; Load balancer -> Web servers; Web servers -> Query API servers; Query API servers -> Memory cache; Query API servers -> Reverse index (sharded); Query API servers -> Document svc (sharded); Job queues -> Reverse index (sharded); Job queues -> Document svc (sharded); Crawler fleet (by host) -> Job queues; Crawler fleet (by host) <-> NoSQL frontier, crawled_links; MapReduce dedup -> NoSQL frontier, crawled_links; Crawler fleet (by host) -> DNS cache, robots cache; Crawler fleet (by host) -> Page store; Crawler fleet (by host) -> Web sites (fetch)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 445\" width=\"800\" role=\"img\" aria-label=\"Scaled design. Internal load balancers are left out to reduce clutter.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"80\" y1=\"69\" x2=\"80\" y2=\"133\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"560\" y1=\"71\" x2=\"560\" y2=\"126\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"598.7\" y1=\"71\" x2=\"672\" y2=\"126\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"622\" y1=\"255\" x2=\"584\" y2=\"198\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"658\" y1=\"255\" x2=\"696\" y2=\"198\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"471\" y1=\"282\" x2=\"567\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"327\" y1=\"282\" x2=\"313\" y2=\"282\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"282\" x2=\"167\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"400\" y1=\"311\" x2=\"400\" y2=\"366\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"458\" y1=\"311\" x2=\"572\" y2=\"368\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"361.3\" y1=\"311\" x2=\"278.7\" y2=\"373\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">User</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">DNS</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"46.5\">Load balancer</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"46.5\">Web servers</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Query API</tspan><tspan x=\"560\" y=\"54.5\">servers</tspan></text><path class=\"dg-node dg-store\" d=\"M652,20 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,20 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"49.5\">Memory cache</tspan></text><path class=\"dg-node dg-store\" d=\"M492,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"161.5\">Reverse index</tspan><tspan x=\"560\" y=\"177.5\">(sharded)</tspan></text><path class=\"dg-node dg-store\" d=\"M652,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"161.5\">Document svc</tspan><tspan x=\"720\" y=\"177.5\">(sharded)</tspan></text><rect class=\"dg-node dg-queue\" x=\"572\" y=\"258\" width=\"136\" height=\"48\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"692\" y1=\"266\" x2=\"692\" y2=\"298\"></line><line class=\"dg-line\" x1=\"682\" y1=\"266\" x2=\"682\" y2=\"298\"></line><line class=\"dg-line\" x1=\"672\" y1=\"266\" x2=\"672\" y2=\"298\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"626\" y=\"286.5\">Job queues</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"278.5\">Crawler fleet</tspan><tspan x=\"400\" y=\"294.5\">(by host)</tspan></text><path class=\"dg-node dg-store\" d=\"M172,258 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M172,258 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"281.5\">NoSQL frontier,</tspan><tspan x=\"240\" y=\"297.5\">crawled_links</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"278.5\">MapReduce</tspan><tspan x=\"80\" y=\"294.5\">dedup</tspan></text><path class=\"dg-node dg-store\" d=\"M332,378 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,378 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"401.5\">DNS cache,</tspan><tspan x=\"400\" y=\"417.5\">robots cache</tspan></text><path class=\"dg-node dg-store\" d=\"M572,380 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M572,380 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"640\" y=\"409.5\">Page store</tspan></text><rect class=\"dg-node dg-client\" x=\"172\" y=\"378\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"406.5\">Web sites</tspan></text><rect class=\"dg-edge-bg\" x=\"299.5\" y=\"333\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"346\" text-anchor=\"middle\">fetch</text></svg></div><figcaption>Scaled design. Internal load balancers are left out to reduce clutter.</figcaption></figure>\n<p><strong>Don't jump straight to the final design.</strong> Say that you would (1) <strong>benchmark</strong> and load-test, (2) <strong>profile</strong> to find bottlenecks, (3) fix them one at a time while weighing alternatives and trade-offs, and (4) repeat. <a href=\"/learn/design-scaling-aws\">Design a system that scales to millions of users</a> shows that loop on a smaller system. Here, the step-1 numbers predict the bottlenecks:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Number from step 1</th><th>What breaks</th></tr></thead><tbody><tr><td>116,000 searches a second at peak</td><td>One web server, API server or index server can't take it.</td></tr><tr><td>10 TB index, 500 GB of documents</td><td>The index doesn't fit in one machine's memory, so latency depends on disk.</td></tr><tr><td>Uneven, repetitive search traffic</td><td>Popular queries redo the same work thousands of times a second.</td></tr><tr><td>3,100 open fetches across 15,400 hosts</td><td>One crawler machine runs out of connections, and DNS lookups pile up.</td></tr><tr><td>77,000 new links a second</td><td>The frontier database takes a lookup and a write for every link.</td></tr><tr><td>6.2 Gbit/s inbound, 2 PB a month</td><td>Network and storage must be sized for the crawl, not only for users.</td></tr></tbody></table></div>\n<h4 id=\"dns-load-balancer-and-web-servers\">DNS, load balancer and web servers</h4>\n<p>Users reach the service through <a href=\"/learn/dns-and-cdn#how-dns-works\">DNS</a>, which returns a <strong>load balancer</strong>'s address. The load balancer spreads requests over many identical <strong>web servers</strong>. Adding copies of a stateless server is <strong>horizontal scaling</strong>, and it answers the 116,000-a-second peak. It costs more machines to run, and every server must stay stateless; see <a href=\"/learn/load-balancing#horizontal-scaling-and-what-it-costs\">horizontal scaling and what it costs</a>. Run the load balancers in pairs so they aren't a single point of failure.</p>\n<p>The web servers act as <a href=\"/learn/reverse-proxy-and-api-gateway#what-a-reverse-proxy-does-for-you\">reverse proxies</a>: they terminate TLS, compress responses and forward requests to the <strong>Query API servers</strong>, a separate <a href=\"/learn/reverse-proxy-and-api-gateway#separating-the-web-and-application-layers\">application layer</a>. Splitting the layers lets the web layer scale with connections and the API layer scale with query work.</p>\n<h4 id=\"a-memory-cache-for-popular-queries\">A memory cache for popular queries</h4>\n<p>Some searches are very popular, and others run once. Serve popular ones from a <strong>memory cache</strong> such as Redis or Memcached, keyed by the normalized query. That cuts response time, keeps load off the reverse index and document services, and absorbs <a href=\"/learn/cache#caches-smooth-out-uneven-load\">uneven traffic and spikes</a>, such as a news event that sends millions of people to one search.</p>\n<p>Memory is much faster than storage. In the widely shared <a href=\"https://gist.github.com/jboner/2841832\" target=\"_blank\" rel=\"noopener noreferrer\">latency numbers table</a>, reading 1 MB sequentially takes about 250 µs from memory, 1 ms from an SSD (4 times as long) and 20 ms from a spinning disk (80 times as long).</p>\n<p>Ten million cached result pages at 5 KB each come to 50 GB, which a small Redis cluster holds easily. How much traffic that absorbs depends on how skewed queries are, so measure the hit rate. Use <a href=\"/learn/cache#cache-aside\">cache-aside</a> with a TTL of minutes to an hour, so results catch up with recrawls, and <a href=\"/learn/cache#when-the-cache-is-full\">LRU eviction</a> so rare queries fall out. When a very popular entry expires, thousands of requests can hit the index at once; <a href=\"/learn/cache#cache-stampede-thundering-herd\">cache stampede</a> covers the fixes.</p>\n<h4 id=\"sharding-and-federation-for-the-index-and-document-services\">Sharding and federation for the index and document services</h4>\n<p>The reverse index service and document service need heavy use of sharding and federation to handle their data size and request load.</p>\n<ul>\n<li><strong>Federation</strong> splits data by function into separate databases. The reverse index, the documents and the crawl tables already live in separate stores owned by separate services, so each can be sized and scaled for its own load. See <a href=\"/learn/relational-databases#federation-one-database-per-feature\">federation</a>.</li>\n<li><strong>Sharding</strong> splits one dataset across many machines. See <a href=\"/learn/relational-databases#sharding-splitting-one-table-across-databases\">sharding</a>.\n<ul>\n<li>The <strong>document service</strong> shards by document ID. Ten results touch at most 10 shards, in parallel. 500 GB over 10 shards is 50 GB each, which fits in memory.</li>\n<li>The <strong>reverse index</strong> can be split by document (each shard indexes some pages, and every query asks every shard) or by term (each shard owns some words, and a query asks only those shards). By document is the usual choice because it balances load and survives a lost shard better; see <a href=\"/learn/design-search-engine#partitioning-the-index-by-document-or-by-term\">partitioning the index</a>. 10 TB over 50 shards is 200 GB each, enough to keep each shard's hot part in memory.</li>\n</ul>\n</li>\n<li>Each shard has <strong>replicas</strong>, which add read capacity for the peak and keep the service up when a machine fails.</li>\n</ul>\n<p>The cost is complexity. A query fans out to many shards and waits for the slowest, so <a href=\"/learn/performance-and-latency#the-tail-at-scale\">tail latency</a> matters, and resharding as the index grows takes planning.</p>\n<h4 id=\"scaling-the-frontier-and-crawled_links\">Scaling the frontier and crawled_links</h4>\n<p><code>links_to_crawl</code> and <code>crawled_links</code> stay in a <a href=\"/learn/nosql-databases#key-value-stores\">key-value NoSQL store</a>, sharded by <strong>hash of the host name</strong>. All of a host's URLs then sit on one shard, which is what per-host politeness needs: the crawler that owns a shard enforces the rate for its hosts without asking anyone. The trade-off is skew, since a huge site makes its shard heavier. Place hosts with <a href=\"/learn/consistent-hashing\">consistent hashing</a> and split the largest out.</p>\n<p>The 77,000 new links a second never hit the database directly. Each crawler's Bloom filter drops URLs it has seen, and the MapReduce job removes repeats from large URL lists before they're merged into the frontier.</p>\n<p>These tables can be <strong>eventually consistent</strong>. If one crawler briefly misses that another just recorded a URL, the worst case is fetching a page twice, a small price for fast, available writes. See <a href=\"/learn/cap-and-consistency#the-consistency-spectrum\">the consistency spectrum</a>.</p>\n<h4 id=\"scaling-the-crawler-service\">Scaling the crawler service</h4>\n<ul>\n<li><strong>Partition crawlers by host.</strong> Each crawler machine owns a range of host hashes, matching the frontier shards. A failed machine's range is reassigned.</li>\n<li><strong>Keep a DNS cache.</strong> Every fetch needs the host's IP address, and DNS lookups can become the bottleneck, so the crawler keeps its own cache, refreshed periodically. Google's 1998 crawler did the same: <a href=\"http://infolab.stanford.edu/~backrub/google.html\" target=\"_blank\" rel=\"noopener noreferrer\">each crawler kept \"its own DNS cache\"</a> because DNS lookup was \"a major performance stress\". Keep the robots cache beside it.</li>\n<li><strong>Keep many connections open.</strong> Fetching is mostly waiting, so each machine runs thousands of fetches at once with asynchronous I/O. Google's early crawlers each kept about 300 connections open, and four of them peaked at over 100 pages a second. Our 3,100 open fetches spread over a few dozen machines. Reusing a connection for the next request to the same host (<strong>connection pooling</strong>, or keep-alive) saves a TCP and TLS handshake each time; see <a href=\"/learn/protocols-and-apis#what-the-guarantees-cost\">connection pooling</a>.</li>\n<li><strong>Use UDP where it applies.</strong> Switching to UDP can help, but only in two places. DNS queries already use it: <a href=\"https://www.rfc-editor.org/rfc/rfc1035#section-4.2.1\" target=\"_blank\" rel=\"noopener noreferrer\">RFC 1035</a> calls UDP \"the recommended method for standard queries\". Page fetches use HTTP/1.1 or HTTP/2 over TCP, except for servers that support HTTP/3, which runs over QUIC on UDP and needs fewer handshake round trips. See <a href=\"/learn/protocols-and-apis#http3-streams-move-into-the-transport\">HTTP/3</a>.</li>\n<li><strong>Provision bandwidth.</strong> Crawling is bandwidth-intensive: 6.2 Gbit/s on average, all day. Size network links for that with headroom. Conditional requests and compressed responses (<code>Accept-Encoding: gzip</code>) reduce it.</li>\n<li><strong>Put raw pages in object storage.</strong> The 2 PB a month go to a cheap, replicated object store. Decide early between every version (72 PB over 3 years) and the latest only (500 TB).</li>\n</ul>\n<h4 id=\"keeping-the-service-available\">Keeping the service available</h4>\n<ul>\n<li><strong>Search path:</strong> replicated web servers, API servers, cache nodes and index shards behind load balancers, spread across failure domains. If one index shard is down, the Query API returns results from the others, which is better than an error. See <a href=\"/learn/availability#graceful-degradation\">graceful degradation</a>.</li>\n<li><strong>Databases:</strong> replicas with automatic <a href=\"/learn/availability#fail-over-active-passive\">fail-over</a> for the document and crawl stores.</li>\n<li><strong>Crawl path:</strong> durable job queues, so a crashed indexer loses nothing, and URL leases, so a crashed crawler's URLs return to the frontier. Search keeps serving the existing index even if the crawl stops for an hour, so the crawl can accept weaker availability than search.</li>\n</ul>","headings":[{"id":"step-4-scale-the-design","title":"Step 4: Scale the design","depth":3},{"id":"dns-load-balancer-and-web-servers","title":"DNS, load balancer and web servers","depth":4},{"id":"a-memory-cache-for-popular-queries","title":"A memory cache for popular queries","depth":4},{"id":"sharding-and-federation-for-the-index-and-document-services","title":"Sharding and federation for the index and document services","depth":4},{"id":"scaling-the-frontier-and-crawled_links","title":"Scaling the frontier and crawled_links","depth":4},{"id":"scaling-the-crawler-service","title":"Scaling the crawler service","depth":4},{"id":"keeping-the-service-available","title":"Keeping the service available","depth":4}]},{"id":"trade-offs-and-follow-up-questions","title":"Trade-offs and follow-up questions","html":"<h3 id=\"trade-offs-and-follow-up-questions\">Trade-offs and follow-up questions</h3>\n<p><strong>1. How to store the frontier?</strong> A Redis sorted set gives one priority order and is easy to explain, but knows nothing about hosts. The Mercator front and back queues handle priority and politeness together, with more machinery. A log such as Kafka is durable and fast but has no priority order. Start with the sorted set sharded by host, with per-host back queues as the politeness layer. See <a href=\"/learn/design-search-engine#the-crawl-frontier-priority-and-politeness\">the crawl frontier</a>.</p>\n<p><strong>2. Exact or approximate \"seen URL\" checks?</strong> A database lookup is exact but costs a read per link, 77,000 a second. A Bloom filter holds a billion URLs in 1.2 GB but skips about 1% of new URLs. Use the Bloom filter as the fast first pass, and confirm \"probably seen\" answers in the database if losing 1% of new pages matters.</p>\n<p><strong>3. Exact or near-duplicate content detection?</strong> SHA-256 is cheap and never wrong about identical pages, but misses a changed timestamp. MinHash or SimHash catch near-duplicates, with a threshold that will sometimes merge pages that really differ. Use the exact hash first, then a near-duplicate check before indexing.</p>\n<p><strong>4. How often to recrawl?</strong> A fixed weekly cycle is simple and predictable. Adaptive intervals spend the budget on pages that change. Cho and Garcia-Molina's results warn against recrawling purely in proportion to change rate. Use adaptive intervals with a floor and a ceiling, weighted by popularity.</p>\n<p><strong>5. Every version or only the latest?</strong> 72 PB versus 500 TB over three years. Unless the product needs page history, keep the latest copy plus the signature history, which is tiny.</p>\n<p><strong>6. Static or query-dependent snippets?</strong> Static snippets are cheap to serve and cache. Snippets that highlight the search words are more useful but need page text at query time. Start static, as required, and mention the upgrade.</p>\n<p><strong>7. How polite is polite enough?</strong> A stricter per-host rate protects sites but slows the crawl of large ones. The ten-times-the-fetch-time rule adapts by itself. Honor <code>robots.txt</code>, <code>Crawl-delay</code>, <code>429</code>, <code>503</code> and <code>Retry-After</code>, then use sitemaps and conditional requests to spend a large site's budget on pages that changed. The <a href=\"/learn/rate-limiting\">rate limiting chapter</a> covers the algorithms a per-host limiter can use.</p>","headings":[{"id":"trade-offs-and-follow-up-questions","title":"Trade-offs and follow-up questions","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>The product team wants the 100 million most popular pages refreshed daily, and the other 900 million weekly. What fetch rate and bandwidth does that need?</summary>\n<p>Daily pages: 100,000,000 × 30 = 3,000,000,000 fetches a month. Weekly pages: 900,000,000 × 30 ÷ 7 ≈ 3,860,000,000. Together, about 6,860,000,000 fetches a month.</p>\n<p>6,860,000,000 ÷ 2,592,000 s ≈ 2,650 fetches a second, up from 1,540. At 500 KB each, that's about 1.3 GB a second, or roughly 10.6 Gbit/s. With a 2-second fetch, Little's Law puts about 5,300 fetches in flight. Check politeness before agreeing, because popular pages cluster on a few large sites. Conditional requests matter too, since most daily refetches will return <code>304 Not Modified</code>.</p>\n</details>\n<details>\n<summary>Overnight, the frontier grew by 40 million URLs from one small site, all like /events?date=2031-04-17. What happened, and what do you change?</summary>\n<p>The crawler fell into a calendar trap. Each date page links to the next and previous dates with no end, and each day's page is slightly different, so the content checks didn't stop it.</p>\n<p>Remove the site's queued URLs. Then add a per-host page budget scaled by popularity, so a small site can't flood the frontier, and URL limits on length, depth and repeated segments. Also check the site's <code>robots.txt</code>, since many sites disallow their calendars.</p>\n</details>\n<details>\n<summary>A host's robots.txt has returned 503 for three days. What should the crawler do under RFC 9309? What if it had returned 404?</summary>\n<p>A 5xx status makes the file \"unreachable\", and the crawler must assume complete disallow: fetch nothing from that host, and keep retrying <code>robots.txt</code>. If it lasts a long time (the RFC's example is 30 days), the crawler may treat the file as unavailable or keep using a cached copy.</p>\n<p>A 404 makes the file \"unavailable\", which means no restrictions. The crawler may fetch any URL on the host, still under its own per-host rate limit.</p>\n</details>\n<details>\n<summary>A colleague's MapReduce job for removing duplicate URLs emits a URL only when its count is 1. It outputs 300 million URLs from an input with 1 billion distinct ones. Why?</summary>\n<p>The reducer drops every URL that appeared two or more times. Popular URLs appear many times, because many pages link to them, so the job threw away most of the URLs that matter. Emit every key once, whatever its count. Keep the count, though: it's how often the URL was linked, which can feed its crawl priority.</p>\n</details>\n<details>\n<summary>A site with 50 million pages allows one request a second. Can the crawler refresh it weekly? What would you do?</summary>\n<p>No. 50,000,000 requests at one a second take 50,000,000 seconds, about 579 days for one pass.</p>\n<p>Spend the budget where it counts. Use the site's sitemaps and <code>&#x3C;lastmod></code> dates to find changed pages, checking them since sites aren't always accurate. Send conditional requests so unchanged pages cost a <code>304</code> with no body. Prioritize by popularity and measured change rate, and accept that the long tail is refreshed rarely. It's also worth asking the site for a higher rate or a feed of changes.</p>\n</details>\n<details>\n<summary>A crawler machine crashes. Which URLs are lost, and how does the design recover?</summary>\n<p>With plain <code>ZPOPMAX</code>, every URL the machine had popped but not recorded is lost: it left <code>links_to_crawl</code> and never reached <code>crawled_links</code>. That's around a hundred URLs per crash.</p>\n<p>With leases, popping moves a URL into an in-progress set with a deadline, and a sweeper returns expired ones to the frontier. The machine's host range goes to another crawler, which rebuilds its per-host queues from the frontier shard. Pages already queued for indexing are safe in the durable queues. A page may be fetched twice after a crash, so indexing must be idempotent: indexing the same URL and signature twice gives the same result.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://www.rfc-editor.org/rfc/rfc9309\" target=\"_blank\" rel=\"noopener noreferrer\">RFC 9309: Robots Exclusion Protocol</a>: the 2022 standard for <code>robots.txt</code>. Short, and it settles matching, error and caching questions.</li>\n<li><a href=\"https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt\" target=\"_blank\" rel=\"noopener noreferrer\">How Google interprets the robots.txt specification</a>: how one large crawler applies RFC 9309, including ignoring <code>crawl-delay</code>.</li>\n<li><a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/the-url-frontier-1.html\" target=\"_blank\" rel=\"noopener noreferrer\">Introduction to Information Retrieval: The URL frontier</a>: the front and back queue design for priority and politeness. Its neighbors on <a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/crawler-architecture-1.html\" target=\"_blank\" rel=\"noopener noreferrer\">crawler architecture</a>, <a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/features-a-crawler-must-provide-1.html\" target=\"_blank\" rel=\"noopener noreferrer\">what a crawler must provide</a> and <a href=\"https://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html\" target=\"_blank\" rel=\"noopener noreferrer\">near-duplicates and shingling</a> cover the rest of this chapter's crawler.</li>\n<li><a href=\"http://infolab.stanford.edu/~backrub/google.html\" target=\"_blank\" rel=\"noopener noreferrer\">The Anatomy of a Large-Scale Hypertextual Web Search Engine</a> (Brin and Page, 1998): section 4.3 describes Google's first crawler, its DNS cache, and the social side of crawling.</li>\n<li><a href=\"https://www.kth.se/social/files/554f1d5bf27654514fd00393/synchronizing-a-database-to-improve-freshness.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Synchronizing a database to improve freshness</a> (Cho and Garcia-Molina, SIGMOD 2000): why recrawling in proportion to change rate backfires.</li>\n<li><a href=\"https://commoncrawl.org/blog/august-2026-crawl-archive-now-available\" target=\"_blank\" rel=\"noopener noreferrer\">Common Crawl: August 2026 crawl archive</a>: real figures for a crawl of about 2 billion pages, for sanity-checking estimates.</li>\n<li><a href=\"https://developers.google.com/search/docs/crawling-indexing/url-structure\" target=\"_blank\" rel=\"noopener noreferrer\">Google: URL structure best practices</a>: the common sources of infinite URL spaces, from the site owner's side.</li>\n<li><a href=\"https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate\" target=\"_blank\" rel=\"noopener noreferrer\">Google: Reduce the Googlebot crawl rate</a> and <a href=\"https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap\" target=\"_blank\" rel=\"noopener noreferrer\">Build and submit a sitemap</a>: how a large crawler reacts to 429 and 503, and which sitemap fields it trusts.</li>\n<li><a href=\"https://www.rfc-editor.org/rfc/rfc9110\" target=\"_blank\" rel=\"noopener noreferrer\">RFC 9110: HTTP Semantics</a>: conditional requests, <code>304 Not Modified</code> and <code>Retry-After</code>, which make recrawling cheaper and politer.</li>\n<li><a href=\"https://redis.io/docs/latest/develop/data-types/sorted-sets/\" target=\"_blank\" rel=\"noopener noreferrer\">Redis sorted sets</a>: the data structure behind <code>links_to_crawl</code>.</li>\n<li><a href=\"https://gist.github.com/jboner/2841832\" target=\"_blank\" rel=\"noopener noreferrer\">Latency numbers every programmer should know</a>: the memory, SSD and disk figures behind the case for a memory cache.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}