{"id":"design-pastebin","title":"Design a text-sharing service like Pastebin","sections":[{"id":"overview","title":"Overview","html":"<p>A text-sharing service lets someone paste a block of text into a web form and get back a short link, such as <code>https://paste.example/r7k2x9a</code>. Anyone who opens the link sees the text. People use it to share a stack trace with a colleague, a config file in a chat, or a log excerpt in a bug report. Pastebin.com is the best-known example.</p>\n<p>The traffic in this exercise is modest, but the design still makes you decide real things. Each paste has two kinds of data. One is a small record: the paste's short code, when it was created and when it expires. The other is the text itself, which can be a few bytes or hundreds of kilobytes. Where each one lives, how expired pastes get cleaned up, and how to serve a paste that suddenly gets popular are the interesting decisions.</p>\n<p>The same design covers a URL shortener such as Bitly. A shortener stores a long URL where this service stores text, and it answers a visit with a redirect instead of a page. The handbook covers that problem in <a href=\"/learn/design-url-shortener\">Design a URL shortener</a>, which goes deep on short codes, redirect status codes and redirect caching. This chapter treats short codes briefly. It spends its depth on storing paste contents, size limits, expiry, caching the read path, and counting page views.</p>\n<p>The chapter follows the four steps from <a href=\"/learn/system-design-interview\">The system design interview, step by step</a>.</p>","headings":[]},{"id":"step-1-outline-use-cases-constraints-and-assumptions","title":"Step 1: Outline use cases, constraints and assumptions","html":"<h3 id=\"step-1-outline-use-cases-constraints-and-assumptions\">Step 1: Outline use cases, constraints and assumptions</h3>\n<p>There's no interviewer to answer questions here, so we'll state the use cases and assumptions ourselves.</p>\n<h4 id=\"use-cases\">Use cases</h4>\n<ul>\n<li><strong>A user enters a block of text and gets a randomly generated link.</strong>\n<ul>\n<li>By default, a paste never expires.</li>\n<li>The user can choose a timed expiration instead, such as one hour or one week.</li>\n</ul>\n</li>\n<li><strong>A user enters a paste's URL and views its contents.</strong></li>\n<li><strong>Users are anonymous.</strong> Nobody signs in.</li>\n<li><strong>The service tracks analytics for each page</strong>, as monthly visit counts.</li>\n<li><strong>The service deletes expired pastes.</strong></li>\n<li><strong>The service is highly available.</strong> A paste link that doesn't open is a broken promise.</li>\n</ul>\n<h4 id=\"out-of-scope\">Out of scope</h4>\n<ul>\n<li>Registering for an account, and verifying an email address.</li>\n<li>Logging in, and editing a paste after it's created.</li>\n<li>Choosing a paste's visibility, such as public, unlisted or private.</li>\n<li>Choosing your own short link (a \"custom alias\"). We'll note where it would fit.</li>\n</ul>\n<h4 id=\"assumptions\">Assumptions</h4>\n<p>These numbers are made-up inputs for the exercise. They don't describe any real service.</p>\n<ul>\n<li>Traffic isn't evenly distributed. Some pastes get far more views than others, and traffic comes in spikes.</li>\n<li>Following a short link should be fast.</li>\n<li>Pastes are text only.</li>\n<li>Page-view analytics don't need to be real time.</li>\n<li>There are 10 million users.</li>\n<li>There are 10 million new pastes a month.</li>\n<li>There are 100 million paste reads a month, so reads outnumber writes 10 to 1.</li>\n</ul>\n<h4 id=\"calculate-usage\">Calculate usage</h4>\n<p>The size of one paste, counting 1 KB as 1,000 bytes:</p>\n<ul>\n<li>Paste contents: 1 KB on average, so 1,000 bytes.</li>\n<li><code>shortlink</code>: 7 bytes, one per character of the code.</li>\n<li><code>expiration_length_in_minutes</code>: 4 bytes, a 32-bit integer.</li>\n<li><code>created_at</code>: 5 bytes, a compact date and time.</li>\n<li><code>paste_path</code>: up to 255 bytes, where the text lives in storage.</li>\n<li>Total: 1,000 + 7 + 4 + 5 + 255 = 1,271 bytes, or about <strong>1.27 KB per paste</strong>.</li>\n</ul>\n<p>The 255 bytes for the path is an upper bound, since a variable-length column stores only the characters it holds. Using the upper bound keeps the estimate safe.</p>\n<p>Storage over time:</p>\n<ul>\n<li>Each month: 1.27 KB × 10 million pastes = <strong>12.7 GB</strong>.</li>\n<li>Over three years: 12.7 GB × 36 months = 457 GB, so about <strong>450 GB</strong>.</li>\n<li>Pastes over three years: 10 million × 36 = <strong>360 million short links</strong>. Pastes can't be edited, so every write is a new paste.</li>\n</ul>\n<p>Split that into its two parts, because they'll live in different stores:</p>\n<ul>\n<li>Metadata (everything except the text): 271 bytes × 360 million ≈ <strong>98 GB</strong> over three years.</li>\n<li>Text: 1,000 bytes × 360 million = <strong>360 GB</strong> over three years.</li>\n</ul>\n<p>Request rates. A month has about 2.5 million seconds. (The exact figure is 30.4 days × 86,400 s ≈ 2.63 million; the round number is fine for an estimate.)</p>\n<ul>\n<li>Writes: 10 million ÷ 2.5 million s = <strong>4 new pastes a second</strong> on average.</li>\n<li>Reads: 100 million ÷ 2.5 million s = <strong>40 reads a second</strong> on average.</li>\n</ul>\n<p>A handy conversion guide:</p>\n<ul>\n<li>2.5 million seconds in a month.</li>\n<li>1 request a second = 2.5 million requests a month.</li>\n<li>40 requests a second = 100 million requests a month.</li>\n<li>400 requests a second = 1 billion requests a month.</li>\n</ul>\n<p>A few more numbers for step 4:</p>\n<ul>\n<li>Read bandwidth: 40 reads a second × about 1.3 KB ≈ 52 KB a second, or about 0.4 megabits a second.</li>\n<li>Peaks: if the busiest minute runs at 10 times the average, that's 40 writes and 400 reads a second.</li>\n<li>Cache: 100 million ÷ 30 ≈ 3.3 million reads a day. Even if every read were for a different paste, holding them all in memory would take 3.3 million × 1.3 KB ≈ 4.3 GB.</li>\n<li>Logs: at about 200 bytes per access log line, 100 million reads produce about 20 GB of logs a month.</li>\n</ul>\n<p><strong>What these numbers tell us.</strong> On average this is a small system, and one server could handle the load. So the design is shaped by other things. The first is availability, because one server is one machine that can fail. The second is skew: one paste shared on a busy forum can get thousands of views a second while the average stays at 40. The third is growth, with 360 million pastes after three years. Note too that the text is almost four times the size of the metadata. That hints that the two belong in different stores. The <a href=\"/learn/estimation\">estimation</a> chapter covers this kind of arithmetic.</p>","headings":[{"id":"step-1-outline-use-cases-constraints-and-assumptions","title":"Step 1: Outline use cases, constraints and assumptions","depth":3},{"id":"use-cases","title":"Use cases","depth":4},{"id":"out-of-scope","title":"Out of scope","depth":4},{"id":"assumptions","title":"Assumptions","depth":4},{"id":"calculate-usage","title":"Calculate usage","depth":4}]},{"id":"step-2-create-a-high-level-design","title":"Step 2: Create a high-level design","html":"<h3 id=\"step-2-create-a-high-level-design\">Step 2: Create a high-level design</h3>\n<figure class=\"diagram\" data-text=\"Client -> Web server (reverse proxy); Web server (reverse proxy) -> Write API (create); Web server (reverse proxy) -> Read API (view); Write API -> Object store (paste text); Write API -> SQL database (pastes table); Read API -> Object store (paste text); Read API -> SQL database (pastes table); Web server (reverse proxy) -> Log analysis (MapReduce) (access logs); Log analysis (MapReduce) -> Analytics database (monthly counts)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 640 445\" width=\"640\" role=\"img\" aria-label=\"The first design. Paste text goes to the object store; everything else goes to the SQL database.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"278.7\" y1=\"133\" x2=\"361.3\" y2=\"71\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"278.7\" y1=\"191\" x2=\"361.3\" y2=\"253\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"418\" y1=\"69\" x2=\"536\" y2=\"246\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"418\" y1=\"255\" x2=\"536\" y2=\"78\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"282\" x2=\"487\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"240\" y1=\"191\" x2=\"240\" y2=\"371\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"402\" x2=\"487\" y2=\"402\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Client</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"158.5\">Web server</tspan><tspan x=\"240\" y=\"174.5\">(reverse proxy)</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"46.5\">Write API</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"258\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"286.5\">Read API</tspan></text><path class=\"dg-node dg-store\" d=\"M492,18 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,18 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"41.5\">Object store</tspan><tspan x=\"560\" y=\"57.5\">(paste text)</tspan></text><path class=\"dg-node dg-store\" d=\"M492,258 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,258 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"281.5\">SQL database</tspan><tspan x=\"560\" y=\"297.5\">(pastes table)</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"376\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"398.5\">Log analysis</tspan><tspan x=\"240\" y=\"414.5\">(MapReduce)</tspan></text><path class=\"dg-node dg-store\" d=\"M492,378 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,378 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"401.5\">Analytics</tspan><tspan x=\"560\" y=\"417.5\">database</tspan></text><rect class=\"dg-edge-bg\" x=\"296.4\" y=\"93\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"106\" text-anchor=\"middle\">create</text><rect class=\"dg-edge-bg\" x=\"302.6\" y=\"213\" width=\"34.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"226\" text-anchor=\"middle\">view</text><rect class=\"dg-edge-bg\" x=\"200.9\" y=\"272\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"285\" text-anchor=\"middle\">access logs</text><rect class=\"dg-edge-bg\" x=\"350.6\" y=\"393\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"399\" y=\"406\" text-anchor=\"middle\">monthly counts</text></svg></div><figcaption>The first design. Paste text goes to the object store; everything else goes to the SQL database.</figcaption></figure>\n<ul>\n<li><strong>Client.</strong> A browser or a command-line tool such as <code>curl</code>.</li>\n<li><strong>Web server.</strong> It runs as a <strong>reverse proxy</strong>, a server that accepts client requests and forwards them to application servers. It terminates HTTPS, rejects oversized requests and logs every request. See <a href=\"/learn/reverse-proxy-and-api-gateway#what-a-reverse-proxy-is\">reverse proxies</a>.</li>\n<li><strong>Write API.</strong> It validates a new paste, stores it, picks a short code and returns the link.</li>\n<li><strong>Read API.</strong> It looks up a code, checks expiry and returns the text. Reads outnumber writes 10 to 1, so separate servers let each side scale on its own.</li>\n<li><strong>SQL database.</strong> It holds one row per paste, mapping the short code to the expiry settings and to where the text lives.</li>\n<li><strong>Object store.</strong> It holds each paste's text as one file. An <strong>object store</strong> is a storage service that keeps whole files (\"objects\") under a key, with put, get and delete over HTTP. Amazon S3 is the best-known example.</li>\n<li><strong>Log analysis and analytics database.</strong> A daily batch job reads the access logs and writes monthly view counts to a separate database, so analytics never loads the database that serves pastes.</li>\n</ul>","headings":[{"id":"step-2-create-a-high-level-design","title":"Step 2: Create a high-level design","depth":3}]},{"id":"step-3-design-core-components","title":"Step 3: Design core components","html":"<h3 id=\"step-3-design-core-components\">Step 3: Design core components</h3>\n<h4 id=\"use-case-user-enters-a-block-of-text-and-gets-a-randomly-generated-link\">Use case: User enters a block of text and gets a randomly generated link</h4>\n<ol>\n<li>The client sends a create request to the web server.</li>\n<li>The web server forwards it to a Write API server.</li>\n<li>The Write API server:\n<ol>\n<li>checks that the text isn't empty or over the size limit,</li>\n<li>works out when the paste expires, if at all,</li>\n<li>stores the text in the object store,</li>\n<li>generates a short code and saves a row in the <code>pastes</code> table, retrying with a new code if that one is taken,</li>\n<li>returns the link.</li>\n</ol>\n</li>\n</ol>\n<h5 id=\"where-the-text-lives\">Where the text lives</h5>\n<p>We need something like a huge hash table: give it a code, get back the paste. There are three reasonable places for the text.</p>\n<ol>\n<li><strong>In the database row</strong>, next to the metadata. One store, one write, one read.</li>\n<li><strong>On file servers you run</strong>, with the row holding a server and path. You handle full disks, replication and failed machines yourself.</li>\n<li><strong>In a managed object store</strong>, with the row holding the object's key. A <a href=\"/learn/nosql-databases#document-stores\">NoSQL document store</a> could play the same role.</li>\n</ol>\n<p>This design uses option 3.</p>\n<ul>\n<li><strong>Durability comes built in.</strong> Amazon's documentation says S3 Standard stores objects across at least three Availability Zones (separate data centers in one region) and is \"designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.\" <a href=\"/learn/availability#availability-vs-durability\">Availability</a> explains the difference between the two numbers.</li>\n<li><strong>The database stays small.</strong> Three years of metadata is about 98 GB, which keeps backups, replicas and failover quick.</li>\n<li><strong>Large pastes don't crowd out small ones.</strong> A 500 KB value makes every read of its row heavier and fills the database's memory with rarely read text.</li>\n<li><strong>Raw pastes become cacheable files.</strong> A CDN can serve them, as step 4 shows.</li>\n</ul>\n<p>The costs are real. A cache miss now makes two round trips, and the second is slow: Amazon's performance guide puts small-object reads at \"roughly 100–200 milliseconds\" and suggests a cache in front when you need single-digit milliseconds. And a create writes to two stores that can't share a transaction.</p>\n<p>A middle path is worth raising in an interview: keep small pastes in the row and move only large ones out. PostgreSQL does a version of this already. Its documentation says that when a row is wider than about 2 kB, it compresses large values and moves them \"out-of-line\" to a separate table. Keeping 1 KB pastes inline would save the second round trip for most reads, at the cost of two code paths.</p>\n<h5 id=\"size-limits\">Size limits</h5>\n<p>A size limit protects everything else. Without one, a single 2 GB upload could exhaust a server's memory. Pastebin's FAQ says free accounts can create pastes up to 512 kilobytes and PRO members up to 10 megabytes. We'll use 512 KB, enforced twice: at the web server, as a maximum request body size, and in the Write API, by counting the decoded bytes. Both reject with <code>413 Content Too Large</code>.</p>\n<p>The limit barely moves the average, but it sets the worst case. One 512 KB paste read 1,000 times a second is 512 MB a second, about 4 gigabits a second of outgoing traffic. That's what the CDN in step 4 is for.</p>\n<h5 id=\"the-pastes-table\">The <code>pastes</code> table</h5>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">CREATE</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> TABLE</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> pastes</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    shortlink                     </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">CHAR</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">7</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)      </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">NOT NULL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    expiration_length_in_minutes  </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">INT</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">          NULL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,      </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">-- NULL means never expires</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    created_at                    </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">DATETIME</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">     NOT NULL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    expires_at                    </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">DATETIME</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">     NULL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,      </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">-- created_at + length, or NULL</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    paste_path                    </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">VARCHAR</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">255</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">NOT NULL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">-- key in the object store</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    PRIMARY KEY</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (shortlink),</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    INDEX</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> idx_expires_at (expires_at)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span></code></pre>\n<ul>\n<li><strong><code>shortlink</code> is the primary key.</strong> Every read looks a paste up by its code. The primary key also creates an index that the database uses to enforce uniqueness, so two pastes can never share a code.</li>\n<li><strong>\"Never expires\" needs a value.</strong> It's the default, so the length column allows <code>NULL</code>.</li>\n<li><strong><code>expires_at</code> is stored and indexed.</strong> The cleanup job asks which pastes have expired, which depends on <code>created_at + expiration_length_in_minutes</code>. An index on <code>created_at</code> alone can't answer that, because two pastes created together can expire a minute or a year apart. Indexing the computed <code>expires_at</code> lets the job find expired rows without scanning the table.</li>\n<li><strong>Indexes keep lookups fast and in memory.</strong> An index turns a full scan into a search whose steps grow with the logarithm of the table size, and a small, busy index stays in RAM. Jonas Bonér's widely shared latency table lists reading 1 MB sequentially at about 250 microseconds from memory, about 4 times longer from an SSD, and about 80 times longer from a spinning disk. See <a href=\"/learn/relational-databases#use-good-indexes\">use good indexes</a>.</li>\n</ul>\n<p>The table could live in a <a href=\"/learn/nosql-databases#key-value-stores\">NoSQL key-value store</a> instead, since every read is \"get by key\" with no joins. The cleanup job's \"everything that expired before now\" is easier in SQL, where it's a range scan on an index. We'll use a relational database. <a href=\"/learn/nosql-databases#sql-or-nosql-making-the-choice\">SQL or NoSQL</a> covers the choice in general.</p>\n<h5 id=\"generating-the-short-code\">Generating the short code</h5>\n<p>The code must be short, safe in a URL, and hard to guess.</p>\n<ol>\n<li><strong>Make a large random-looking number.</strong> For example, take an MD5 hash of the client's IP address and the current time. MD5 turns any input into a 128-bit number spread evenly across its range. Hashing random bytes works as well.</li>\n<li><strong>Write it in Base62</strong>, the 62 characters <code>0-9</code>, <code>a-z</code> and <code>A-Z</code>, which are all safe in URLs. Base64 adds <code>+</code> and <code>/</code>, which have special meanings in URLs. Encoding is deterministic, and a <em>k</em>-digit result takes <em>k</em> steps.</li>\n<li><strong>Keep seven characters.</strong> 62⁷ is about 3.52 trillion codes. After three years we'd use 360 million of them, about 0.01%.</li>\n</ol>\n<p>Two details matter in practice. First, reduce the number modulo 62⁷ before encoding. A 128-bit number in Base62 has up to 22 characters, and since 2¹²⁸ is only about one eighth of 62²², its first character can take only about 8 values. Cutting the first 7 characters therefore uses only about 8 × 62⁶ ≈ 450 billion codes. Second, add randomness. Two people behind one office network share an IP address, and if they create pastes in the same millisecond, hashing IP and time gives both the same code every time they retry.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> os</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> time</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">ALPHABET</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\"</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CODE_LENGTH</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 7</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> to_base62</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(num: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> num </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> ALPHABET</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    digits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> []</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    while</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> num </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        num, remainder </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> divmod</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(num, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">62</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        digits.append(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">ALPHABET</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[remainder])</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">reversed</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(digits))</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> new_shortlink</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(client_ip: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    seed </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">client_ip</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">|</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">time.time_ns()</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">|</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">os.urandom(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">).hex()</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    number </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.from_bytes(hashlib.md5(seed.encode()).digest(), </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"big\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> to_base62(number </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">%</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 62</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> **</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> CODE_LENGTH</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">).rjust(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CODE_LENGTH</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">ALPHABET</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span></span></code></pre>\n<p>Random codes sometimes collide with a taken one. With 360 million pastes stored, a new code collides about once in 9,800 tries (360 million ÷ 3.52 trillion). Over all three years that's roughly 18,000 collisions (360 million² ÷ (2 × 3.52 trillion)). The fix is cheap: insert, let the primary key reject a duplicate, and retry with a new code. A custom alias, if the product adds one, goes through the same insert and the same check.</p>\n<p>The alternative is to encode numbers from a counter, which never collide. <a href=\"/learn/design-url-shortener#generating-the-short-code\">Generating the short code</a> compares the approaches and how to hand out unique numbers across servers.</p>\n<h5 id=\"writing-to-two-stores\">Writing to two stores</h5>\n<p>A crash between the object store write and the database write leaves one without the other. The order decides which.</p>\n<ul>\n<li><strong>Row first:</strong> a crash leaves a row that points at nothing, and readers get an error.</li>\n<li><strong>Text first:</strong> a crash leaves a file no row points to. Nobody has its code, so nobody can reach it. It only wastes a little storage.</li>\n</ul>\n<p>So write the text first, under a key that doesn't depend on the short code, such as a UUID. Otherwise a retry after a code collision would overwrite another paste's text. A periodic sweep deletes objects that no row references once they're a day old. S3 supports this order: it gives \"strong read-after-write consistency for PUT and DELETE requests,\" and \"if a PUT request is successful, your data is safely stored.\" <a href=\"/learn/distributed-transactions\">Distributed transactions</a> covers keeping two stores in step more generally.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> uuid</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime, timedelta</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">MAX_PASTE_BYTES</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 512</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1000</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> create_paste</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(request, client_ip, db, object_store):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    text </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> request[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"paste_contents\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(text.encode(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"utf-8\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> or</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> MAX_PASTE_BYTES</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        raise</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> PayloadTooLarge(size)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    minutes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> request.get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"expiration_length_in_minutes\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># None: never expires</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    created_at </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime.utcnow()</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    expires_at </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> created_at </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> timedelta(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">minutes</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">minutes) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> minutes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # 1. Text first, under a key that doesn't depend on the code.</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    paste_path </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">expiry_prefix(minutes)</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">uuid.uuid4().hex</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    object_store.put(paste_path, text, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">content_type</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text/plain; charset=utf-8\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # 2. Claim a code. The primary key rejects duplicates.</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> _ </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        shortlink </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> new_shortlink(client_ip)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            db.execute(</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                \"INSERT INTO pastes VALUES (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">)\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                (shortlink, minutes, created_at, expires_at, paste_path),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            )</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"shortlink\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: shortlink}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        except</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> db.UniqueViolation:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            continue</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"no free short code\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<p><code>expiry_prefix</code> puts each paste under a prefix for its expiry option, such as <code>ttl-7d/</code> or <code>ttl-never/</code>. The deletion use case explains why.</p>\n<h5 id=\"the-api\">The API</h5>\n<p>The public API follows REST: each paste is a resource with its own URL, acted on with standard HTTP methods. See <a href=\"/learn/protocols-and-apis#rest-resources-and-representations\">REST</a>.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>$ curl -X POST https://paste.example/api/v1/pastes \\</span></span>\n<span class=\"line\"><span>    -H \"Content-Type: application/json\" \\</span></span>\n<span class=\"line\"><span>    -d '{\"paste_contents\": \"Hello World!\", \"expiration_length_in_minutes\": 60}'</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>HTTP/1.1 201 Created</span></span>\n<span class=\"line\"><span>Content-Type: application/json</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>{</span></span>\n<span class=\"line\"><span>  \"shortlink\": \"r7k2x9a\",</span></span>\n<span class=\"line\"><span>  \"url\": \"https://paste.example/r7k2x9a\",</span></span>\n<span class=\"line\"><span>  \"raw_url\": \"https://paste.example/raw/r7k2x9a\"</span></span>\n<span class=\"line\"><span>}</span></span></code></pre>\n<p>Leaving out <code>expiration_length_in_minutes</code> creates a paste that never expires.</p>\n<p>Inside the system, services can talk over <strong>remote procedure calls</strong> (RPC), where one server calls a function on another as if it were local. RPC is compact for traffic you control, and REST suits outside users because any HTTP client works. <a href=\"/learn/protocols-and-apis#rpc-calling-a-function-on-another-machine\">RPC</a> compares them.</p>\n<h4 id=\"use-case-user-enters-a-pastes-url-and-views-the-contents\">Use case: User enters a paste's URL and views the contents</h4>\n<ol>\n<li>The client requests the paste from the web server.</li>\n<li>The web server forwards it to a Read API server.</li>\n<li>The Read API server looks up the code in the <code>pastes</code> table.\n<ul>\n<li>If the row exists and hasn't expired, it fetches the text from the object store and returns it.</li>\n<li>Otherwise it returns an error.</li>\n</ul>\n</li>\n</ol>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>$ curl https://paste.example/api/v1/pastes/r7k2x9a</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>HTTP/1.1 200 OK</span></span>\n<span class=\"line\"><span>Content-Type: application/json</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>{</span></span>\n<span class=\"line\"><span>  \"paste_contents\": \"Hello World!\",</span></span>\n<span class=\"line\"><span>  \"created_at\": \"2026-10-19T14:02:11Z\",</span></span>\n<span class=\"line\"><span>  \"expiration_length_in_minutes\": 60,</span></span>\n<span class=\"line\"><span>  \"expires_at\": \"2026-10-19T15:02:11Z\"</span></span>\n<span class=\"line\"><span>}</span></span></code></pre>\n<p>An unknown code gets <code>404 Not Found</code>. An expired one can get <code>410 Gone</code>, which says the paste existed and was removed on purpose.</p>\n<p>Readers also use two other URLs. <code>https://paste.example/r7k2x9a</code> is an HTML page showing the text. <code>https://paste.example/raw/r7k2x9a</code> returns only the text, as <code>text/plain</code>, for scripts and terminals. The raw URL is often the busiest, and it's the easiest to cache, because every reader gets the same bytes.</p>\n<p>The Read API must check <code>expires_at</code> on every read. Cleanup runs in the background and can run late, so a paste that expired a second ago has to look gone even if its row and file still exist.</p>\n<h5 id=\"caching-the-read-path\">Caching the read path</h5>\n<p>Step 4 adds a memory cache, used <strong>cache-aside</strong>: check the cache, and on a miss read the database and object store, then fill the cache. <a href=\"/learn/cache#cache-aside\">Cache-aside</a> explains the pattern.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">DEFAULT_TTL</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3600</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   # seconds</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">MISSING</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"missing\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> get_paste</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(shortlink, cache, replica, primary, object_store):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    key </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"paste:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">shortlink</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    entry </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cache.get(key)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> entry </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        row </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> replica.find_paste(shortlink) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">or</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> primary.find_paste(shortlink)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> row </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            cache.set(key, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">MISSING</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">ttl</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">60</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 404</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        entry </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">            \"paste_contents\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: object_store.get(row.paste_path),</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">            \"created_at\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: row.created_at,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">            \"expires_at\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: row.expires_at,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        }</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        cache.set(key, entry, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">ttl</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">ttl_for(row.expires_at))</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> entry </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> MISSING</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 404</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> entry[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"expires_at\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> entry[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"expires_at\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime.utcnow():</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 410</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 200</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, entry</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> ttl_for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(expires_at):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Never cache a paste past its own expiry.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> expires_at </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> DEFAULT_TTL</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    seconds_left </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (expires_at </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime.utcnow()).total_seconds()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">DEFAULT_TTL</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(seconds_left)))</span></span></code></pre>\n<ul>\n<li><strong>The cache TTL never outlives the paste.</strong> A paste expiring in five minutes is cached for at most five minutes.</li>\n<li><strong>Unknown codes are cached briefly.</strong> Otherwise someone requesting random codes sends every request to the database, a problem called <a href=\"/learn/cache#cache-penetration\">cache penetration</a>.</li>\n<li><strong>A replica miss falls back to the primary.</strong> A <strong>read replica</strong> is a database copy that receives the primary's changes a moment late. A paste created a second ago may not be there yet, and its creator is the most likely person to open it. See <a href=\"/learn/relational-databases#replication-lag-and-reading-your-own-writes\">replication lag</a>.</li>\n</ul>\n<h4 id=\"use-case-service-tracks-analytics-of-pages\">Use case: Service tracks analytics of pages</h4>\n<p>We need monthly visit counts per paste, and they don't need to be real time. Every view already passes through a web server that writes an access log line, such as:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>203.0.113.7 - - [19/Oct/2026:14:02:11 +0000] \"GET /raw/r7k2x9a HTTP/1.1\" 200 1311</span></span></code></pre>\n<p>So the cheapest design counts from the logs after the fact, with nothing added to the read path. Each web server ships its logs to the object store hourly. Once a day, a <strong>MapReduce</strong> job counts views. Dean and Ghemawat's 2004 MapReduce paper from Google uses this very task as an example: counting how often each URL appears in \"logs of web page requests.\" You write two functions.</p>\n<ul>\n<li>The <strong>mapper</strong> reads one log line and, for each successful paste view, outputs the key <code>(month, shortlink)</code> with the value 1.</li>\n<li>The framework groups all values with the same key onto one machine. This step is the <strong>shuffle</strong>.</li>\n<li>The <strong>reducer</strong> receives one key with all its values and adds them up.</li>\n</ul>\n<p>An optional <strong>combiner</strong> adds up counts on each mapper machine before the shuffle, so a popular paste sends one number per machine instead of one per view. The paper describes combiners for reduce functions that are \"commutative and associative,\" and addition is both.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> re</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">VIEW</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> re.compile(</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    r</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'</span><span style=\"color:#22863A;--shiki-light-font-weight:bold;--shiki-dark:#85E89D;--shiki-dark-font-weight:bold\">\\[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">(</span><span style=\"color:#22863A;--shiki-dark:#85E89D\">?P&#x3C;ts></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">[</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">^</span><span style=\"color:#22863A;--shiki-light-font-weight:bold;--shiki-dark:#85E89D;--shiki-dark-font-weight:bold\">\\]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">]</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">)</span><span style=\"color:#22863A;--shiki-light-font-weight:bold;--shiki-dark:#85E89D;--shiki-dark-font-weight:bold\">\\]</span><span style=\"color:#032F62;--shiki-dark:#DBEDFF\"> \"GET /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">(?:</span><span style=\"color:#032F62;--shiki-dark:#DBEDFF\">raw/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">?</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">(</span><span style=\"color:#22863A;--shiki-dark:#85E89D\">?P&#x3C;code></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">[0-9A-Za-z]</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">{7}</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">)</span><span style=\"color:#032F62;--shiki-dark:#DBEDFF\"> HTTP/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">[\\d.]</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#032F62;--shiki-dark:#DBEDFF\">\" </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">(</span><span style=\"color:#22863A;--shiki-dark:#85E89D\">?P&#x3C;status></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\d</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">{3}</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">)</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> MonthlyPasteViews</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> mapper</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, _, line):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"\"\"Emit ((month, shortlink), 1) per successful view:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        (\"2026-10\", \"r7k2x9a\"), 1</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        (\"2026-10\", \"r7k2x9a\"), 1</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        (\"2026-10\", \"bQ81zz0\"), 1</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        match </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> VIEW</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.search(line)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> match </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> or</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> match[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"status\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">!=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"200\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        month </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime.strptime(match[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"ts\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%d</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%b</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">/%Y:%H:%M:%S %z\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">).strftime(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"%Y-%m\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        yield</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (month, match[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"code\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> combiner</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, key, counts):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        yield</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> key, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(counts)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> reducer</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, key, counts):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"\"\"Total per paste per month:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        (\"2026-10\", \"r7k2x9a\"), 2</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        (\"2026-10\", \"bQ81zz0\"), 1</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        yield</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> key, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(counts)</span></span></code></pre>\n<p>The output, one row per paste per month, goes into the analytics database. The input is about 20 GB of logs a month, so this is a small job. Decide what a \"visit\" means before writing the mapper: this one counts every successful request, but you may want to drop bots or count each visitor once a day. Only the mapper changes. <a href=\"/learn/batch-processing#mapreduce-the-foundational-model\">MapReduce</a> covers the shuffle and fault tolerance in more depth.</p>\n<h5 id=\"a-streaming-alternative\">A streaming alternative</h5>\n<p>The other approach counts as views happen. The Read API publishes a small event, <code>{shortlink, timestamp}</code>, to a <strong>log</strong> such as Apache Kafka, an append-only sequence of events that consumers read at their own pace. A consumer keeps running totals for a few seconds, then writes the increments to the analytics store. <a href=\"/learn/event-streaming\">Event streaming</a> explains these logs.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th></th><th>Batch over access logs</th><th>Streaming view events</th></tr></thead><tbody><tr><td>Freshness</td><td>A day old</td><td>Seconds old</td></tr><tr><td>Change to the read path</td><td>None</td><td>Publish one event per view</td></tr><tr><td>New infrastructure</td><td>A scheduled job</td><td>A log cluster and always-on consumers</td></tr><tr><td>Fixing a counting bug</td><td>Re-run over the stored logs</td><td>Replay the log, if it still holds the events</td></tr><tr><td>Duplicates</td><td>Rare</td><td>Need care: see <a href=\"/learn/event-streaming#delivery-semantics\">delivery semantics</a></td></tr></tbody></table></div>\n<p>Real-time counts aren't required, so batch fits better here. Streaming earns its cost if the product wants a live view counter, and many systems run both, as described under <a href=\"/learn/batch-processing#combining-batch-and-streaming-lambda-vs-kappa\">Lambda vs Kappa</a>.</p>\n<p>One trap applies to both. Once step 4 puts a CDN in front of raw pastes, many views are answered by the CDN and never reach a web server. The job must also read the CDN's request logs, or the counts drop the day the CDN goes live.</p>\n<h4 id=\"use-case-service-deletes-expired-pastes\">Use case: Service deletes expired pastes</h4>\n<p>Deleting an expired paste means removing its row and file and making sure no cache still serves it. The read path already hides expired pastes, so cleanup only reclaims space and doesn't need to be instant. If half of all pastes set an expiry, that's 5 million ÷ 2.5 million s = 2 deletions a second on average, which is easy as long as nothing scans the whole table.</p>\n<h5 id=\"the-cleanup-job\">The cleanup job</h5>\n<p>A job runs every few minutes and deletes in small batches, using the <code>expires_at</code> index:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> delete_expired_pastes</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(db, object_store, cache, batch_size</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    while</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        rows </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> db.query(</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">            \"SELECT shortlink, paste_path FROM pastes\"</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">            \" WHERE expires_at &#x3C;= NOW() ORDER BY expires_at LIMIT </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            batch_size,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        )</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> rows:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> row </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> rows:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            db.execute(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"DELETE FROM pastes WHERE shortlink = </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, row.shortlink)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            cache.delete(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"paste:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">row.shortlink</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            object_store.delete(row.paste_path)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># a failure leaves a harmless orphan</span></span></code></pre>\n<p>Small batches keep each delete short, so locks are held briefly and replicas aren't flooded with changes. The row goes first, so readers stop finding the paste. If the file delete fails, the orphan sweep from the write path removes it later.</p>\n<p>The job could instead mark rows as expired, a <strong>soft delete</strong>. That keeps a record for abuse reports and lets the read path return <code>410 Gone</code>, but the table never shrinks. A common middle path marks the row at once and removes it 30 days later.</p>\n<h5 id=\"letting-the-object-store-expire-files\">Letting the object store expire files</h5>\n<p>S3 can delete files on a schedule with <strong>lifecycle rules</strong>. A rule selects objects by key prefix or tag and expires them a set number of days after creation. That's why the write path files each paste under a prefix for its expiry option: one rule per prefix, such as <code>ttl-7d/</code>, cleans up those files. Amazon's documentation sets some limits:</p>\n<ul>\n<li><strong>Rules count in days.</strong> S3 adds the days to the creation time and rounds up \"to the next day at midnight UTC,\" so rules can't do a 10-minute or one-hour expiry.</li>\n<li><strong>Deletion can run late.</strong> Expired objects are queued and removed asynchronously, possibly \"days or even weeks later.\" Amazon advises against relying on lifecycle rules alone for deletions with a deadline.</li>\n<li><strong>Billing stops on time anyway:</strong> \"after the object expiration time, you aren't charged for storage, even if the object isn't deleted immediately.\"</li>\n</ul>\n<p>So lifecycle rules are a cheap backstop for day-sized options. They don't remove database rows or replace the read-path check. Pastes that never expire go under a prefix with no rule.</p>\n<h5 id=\"caches-and-the-cdn\">Caches and the CDN</h5>\n<p>An expired paste can linger in two caches. The memory cache already caps its TTL at the paste's remaining life. The CDN needs the same rule: raw responses carry <code>Cache-Control: max-age</code> set to the smaller of the default and the seconds left. CloudFront's documentation says <code>max-age</code> sets how long, in seconds, it keeps an object before asking the origin again, capped by its own maximum TTL. Without the header, CloudFront falls back to the default TTL, which is 24 hours unless you change it. That would keep a 10-minute paste visible for a day.</p>","headings":[{"id":"step-3-design-core-components","title":"Step 3: Design core components","depth":3},{"id":"use-case-user-enters-a-block-of-text-and-gets-a-randomly-generated-link","title":"Use case: User enters a block of text and gets a randomly generated link","depth":4},{"id":"use-case-user-enters-a-pastes-url-and-views-the-contents","title":"Use case: User enters a paste's URL and views the contents","depth":4},{"id":"use-case-service-tracks-analytics-of-pages","title":"Use case: Service tracks analytics of pages","depth":4},{"id":"use-case-service-deletes-expired-pastes","title":"Use case: Service deletes expired pastes","depth":4}]},{"id":"step-4-scale-the-design","title":"Step 4: Scale the design","html":"<h3 id=\"step-4-scale-the-design\">Step 4: Scale the design</h3>\n<figure class=\"diagram\" data-text=\"Client -> DNS; Client -> Load balancer; Client -> CDN (raw pastes); CDN (raw pastes) -> Load balancer (on a miss); Load balancer -> Web servers; Web servers -> Write API; Web servers -> Read API; Write API -> SQL primary; SQL primary -> SQL read replicas; Read API -> SQL read replicas; Read API -> Object store; Read API -> Memory cache; Web servers -> Log analysis (logs); Log analysis -> Analytics DB\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 443\" width=\"800\" role=\"img\" aria-label=\"The scaled design. Internal load balancers are left out to reduce clutter.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"80\" y1=\"135\" x2=\"80\" y2=\"71\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"80\" y1=\"189\" x2=\"80\" y2=\"251\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"118.7\" y1=\"253\" x2=\"201.3\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"436\" y1=\"135\" x2=\"521.3\" y2=\"71\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"436\" y1=\"189\" x2=\"521.3\" y2=\"253\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"720\" y1=\"74\" x2=\"720\" y2=\"126\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"596\" y1=\"255\" x2=\"672\" y2=\"198\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"282\" x2=\"647\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"596\" y1=\"309\" x2=\"674.7\" y2=\"368\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"382\" y1=\"189\" x2=\"259.3\" y2=\"373\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"402\" x2=\"327\" y2=\"402\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">DNS</tspan></text><rect class=\"dg-node dg-client\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Client</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"278.5\">CDN</tspan><tspan x=\"80\" y=\"294.5\">(raw pastes)</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"166.5\">Load balancer</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"166.5\">Web servers</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"46.5\">Write API</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"258\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"286.5\">Read API</tspan></text><path class=\"dg-node dg-store\" d=\"M652,20 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,20 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"49.5\">SQL primary</tspan></text><path class=\"dg-node dg-store\" d=\"M652,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"161.5\">SQL read</tspan><tspan x=\"720\" y=\"177.5\">replicas</tspan></text><path class=\"dg-node dg-store\" d=\"M652,260 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,260 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"289.5\">Object store</tspan></text><path class=\"dg-node dg-store\" d=\"M652,380 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,380 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"409.5\">Memory cache</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"378\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"406.5\">Log analysis</tspan></text><path class=\"dg-node dg-store\" d=\"M332,380 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,380 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"409.5\">Analytics DB</tspan></text><rect class=\"dg-edge-bg\" x=\"127.1\" y=\"213\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"160\" y=\"226\" text-anchor=\"middle\">on a miss</text><rect class=\"dg-edge-bg\" x=\"303.3\" y=\"272\" width=\"34.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320.7\" y=\"285\" text-anchor=\"middle\">logs</text></svg></div><figcaption>The scaled design. Internal load balancers are left out to reduce clutter.</figcaption></figure>\n<p>The Write API also writes to the object store; that edge is left out so lines don't cross.</p>\n<p>Don't jump straight to this design. Scaling is a loop: <strong>benchmark</strong> with a load test, <strong>profile</strong> to find the bottleneck, <strong>fix</strong> it while weighing alternatives, and <strong>repeat</strong>. <a href=\"/learn/design-scaling-aws\">Design a system that scales to millions of users on AWS</a> walks through that loop.</p>\n<h4 id=\"finding-the-bottlenecks\">Finding the bottlenecks</h4>\n<p>Four writes and 40 reads a second won't strain any one component. The problems are elsewhere:</p>\n<ol>\n<li><strong>Every component is a single point of failure</strong>, and high availability is a requirement.</li>\n<li><strong>A cache miss costs two round trips</strong>, and small S3 reads take roughly 100–200 ms.</li>\n<li><strong>Traffic is skewed.</strong> One popular paste can get thousands of reads a second, and large pastes turn that into gigabits a second.</li>\n<li><strong>Data keeps growing</strong>: about 2.7 GB of metadata and 10 million objects a month.</li>\n</ol>\n<p>Each component below fixes one of these, and each has a cost.</p>\n<h4 id=\"dns-and-a-cdn-for-raw-pastes\">DNS and a CDN for raw pastes</h4>\n<p><strong>DNS</strong> turns <code>paste.example</code> into the load balancers' addresses, and with health checks it can steer clients away from a failed site (bottleneck 1).</p>\n<p>A <strong>CDN</strong> is a network of caching servers in many locations that serves content from a server near the reader. Use it as a <strong>pull CDN</strong> for raw pastes: on a miss it fetches the paste from you once, then serves its copy until it expires. A viral raw paste becomes a handful of requests to your servers (bottleneck 3), and distant readers get lower latency. Point the CDN at the load balancer. Then every miss still passes the Read API's expiry check and gets the right <code>Cache-Control</code> header, which wouldn't happen if the CDN read the object store directly.</p>\n<p>The costs: a paste removed for abuse stays visible at the edge until its copy expires, unless you also send the CDN an invalidation; and CDN-served views vanish from your web server logs. See <a href=\"/learn/dns-and-cdn#content-delivery-networks\">CDNs</a> and <a href=\"/learn/dns-and-cdn#http-caching-headers\">HTTP caching headers</a>.</p>\n<h4 id=\"a-load-balancer-and-more-web-servers\">A load balancer and more web servers</h4>\n<p>A <strong>load balancer</strong> spreads requests across identical web servers and stops sending traffic to any that fail health checks. Running many copies this way is <strong>horizontal scaling</strong>, and it removes the web tier as a single point of failure. It works because the servers are <strong>stateless</strong>: users are anonymous, so there are no sessions to share. The load balancer is now critical, so run a pair. See <a href=\"/learn/load-balancing\">load balancing</a>.</p>\n<h4 id=\"separate-write-api-and-read-api-servers\">Separate Write API and Read API servers</h4>\n<p>Behind the web servers sits an <strong>application layer</strong> of API servers, split into a write pool and a read pool. Reads are ten times as frequent and writes do more work each, so separate pools scale and deploy independently, and a burst of reads can't starve the create path. The cost is more services to run, plus internal load balancers. See <a href=\"/learn/reverse-proxy-and-api-gateway#separating-the-web-and-application-layers\">separating the web and application layers</a>.</p>\n<h4 id=\"a-memory-cache-for-hot-pastes\">A memory cache for hot pastes</h4>\n<p>A <strong>memory cache</strong> such as Redis or Memcached keeps recently read pastes in RAM. Popular pastes come back in about a millisecond without touching the database or object store (bottleneck 2), and spikes are absorbed before they reach the database (bottleneck 3). Sizing is easy: a whole day's distinct reads would take about 4.3 GB, and the hottest 20% under 1 GB, so one modest server with a replica is enough.</p>\n<p>The costs: cached data can be stale, though pastes are never edited, so only deletion and expiry matter, and the TTL rules bound both. A cold cache sends a burst of misses, and a hot key that expires can make thousands of requests rebuild it together. <a href=\"/learn/cache#cache-stampede-thundering-herd\">Cache stampede</a> lists the fixes.</p>\n<h4 id=\"read-replicas-and-failover-for-the-database\">Read replicas and failover for the database</h4>\n<p>The database becomes a <strong>primary</strong> that takes all writes and <strong>read replicas</strong> that copy its changes, an arrangement often called master-slave replication.</p>\n<ul>\n<li><strong>Replicas serve cache misses.</strong> One or two are plenty, as long as they have spare capacity to keep up with replicating the writes.</li>\n<li><strong>Failover</strong> promotes a replica if the primary dies. Until then new pastes can't be created, but existing ones stay readable from the cache and replicas.</li>\n</ul>\n<p>Four writes a second, more at peak, is well within one primary. The costs are replica lag, which is why reads of brand-new codes fall back to the primary, and promotion, which must never leave two primaries accepting writes. See <a href=\"/learn/relational-databases#primary-replica-replication\">primary-replica replication</a>, <a href=\"/learn/availability#fail-over-active-passive\">active-passive failover</a>, <a href=\"/learn/cap-and-consistency\">CAP and consistency</a> and <a href=\"/learn/availability\">Availability</a>.</p>\n<h4 id=\"object-store-growth\">Object store growth</h4>\n<p>S3 handles 12.7 GB of new text a month comfortably, with no disks to add. What grows is the number of objects and requests. Amazon's documentation says S3 supports \"at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix,\" and that scaling up \"happens gradually,\" with <code>503 Slow Down</code> errors meanwhile. At 4 writes a second we're far below that.</p>\n<p>One trap: Amazon's storage class table shows that S3 Standard-IA, the cheaper class for rarely read data, bills each object as at least 128 KB and for at least 30 days. A 1 KB paste there costs as much as a 128 KB one. Keep small pastes in S3 Standard.</p>\n<h4 id=\"the-analytics-database\">The analytics database</h4>\n<p>The MapReduce output goes into a separate <strong>analytics database</strong>. A data warehouse such as Amazon Redshift or Google BigQuery is built for the large scans and grouping queries that reports run, which would slow the serving database. See <a href=\"/learn/batch-processing#data-warehouses-snowflake-google-bigquery-amazon-redshift\">data warehouses</a>.</p>\n<h4 id=\"if-writes-outgrow-one-primary\">If writes outgrow one primary</h4>\n<p>Nothing in the assumptions forces this, but an interviewer may ask about 100 times the traffic. Start with <strong><a href=\"/learn/relational-databases#sql-tuning\">SQL tuning</a></strong>: every query on an index, small rows, pooled connections. Then:</p>\n<ul>\n<li><strong><a href=\"/learn/relational-databases#federation-one-database-per-feature\">Federation</a></strong> splits the database by function, for example pastes on one database and abuse reports on another.</li>\n<li><strong><a href=\"/learn/relational-databases#sharding-splitting-one-table-across-databases\">Sharding</a></strong> splits the <code>pastes</code> table by a hash of <code>shortlink</code>. Every read knows the code, and random codes spread evenly.</li>\n<li><strong><a href=\"/learn/relational-databases#denormalization\">Denormalization</a></strong> copies data to avoid joins. This schema has none, so it helps little here.</li>\n</ul>\n<p>Moving the table to a NoSQL key-value store is another option. It has the right shape and scales writes by adding nodes. The cost is the cleanup job's range query on <code>expires_at</code>, which most key-value stores don't support directly, though some offer built-in per-item expiry instead.</p>","headings":[{"id":"step-4-scale-the-design","title":"Step 4: Scale the design","depth":3},{"id":"finding-the-bottlenecks","title":"Finding the bottlenecks","depth":4},{"id":"dns-and-a-cdn-for-raw-pastes","title":"DNS and a CDN for raw pastes","depth":4},{"id":"a-load-balancer-and-more-web-servers","title":"A load balancer and more web servers","depth":4},{"id":"separate-write-api-and-read-api-servers","title":"Separate Write API and Read API servers","depth":4},{"id":"a-memory-cache-for-hot-pastes","title":"A memory cache for hot pastes","depth":4},{"id":"read-replicas-and-failover-for-the-database","title":"Read replicas and failover for the database","depth":4},{"id":"object-store-growth","title":"Object store growth","depth":4},{"id":"the-analytics-database","title":"The analytics database","depth":4},{"id":"if-writes-outgrow-one-primary","title":"If writes outgrow one primary","depth":4}]},{"id":"trade-offs-and-follow-up-questions","title":"Trade-offs and follow-up questions","html":"<h3 id=\"trade-offs-and-follow-up-questions\">Trade-offs and follow-up questions</h3>\n<p><strong>Text in the object store or in the database?</strong> The object store brings durability for free, a small database, and raw pastes a CDN can serve. It costs a second, slower read on cache misses and two writes with no shared transaction. At 360 GB over three years, the database could hold the text too. I'd use the object store with a memory cache in front, and mention keeping pastes under a few kilobytes inline. See <a href=\"/learn/design-pastebin#where-the-text-lives\">where the text lives</a>.</p>\n<p><strong>Random codes or a counter?</strong> Random codes are unguessable and need no coordination, but inserts sometimes collide, about once per 9,800 creates at the worst point. A counter never collides but needs unique numbers across servers, and sequential codes let anyone list every paste unless they're scrambled. Unlisted sharing is the main use, so I'd pick random codes. See <a href=\"/learn/design-url-shortener#generating-the-short-code\">generating the short code</a>.</p>\n<p><strong>SQL or NoSQL for the metadata?</strong> Reads are lookups by key, which suits key-value stores. Cleanup is a range query on expiry, which suits SQL. At 4 writes a second and 98 GB, one relational primary with replicas is enough. See <a href=\"/learn/nosql-databases#sql-or-nosql-making-the-choice\">SQL or NoSQL</a>.</p>\n<p><strong>How do expired pastes go away?</strong> Only the read-path check is exact. A cleanup job, lifecycle rules and per-item expiry in a store all reclaim space late, by minutes or, for S3 lifecycle, by days. I'd use the read check for correctness, the job for rows and files, and lifecycle rules as a backstop.</p>\n<p><strong>How long should caches keep a paste?</strong> Longer TTLs mean less load; shorter ones make removals visible sooner. Expiry is known in advance, so TTLs can follow it exactly. Abuse takedowns aren't, so they need explicit cache deletes and CDN invalidation. I'd cap the default at about an hour. See <a href=\"/learn/dns-and-cdn#cache-invalidation-strategies\">cache invalidation</a>.</p>\n<p><strong>Batch or streaming analytics?</strong> Batch is cheap, leaves the read path alone and is easy to re-run. Streaming gives live counts for the price of a log cluster and duplicate handling. With no real-time requirement, I'd use batch. See <a href=\"/learn/batch-processing\">batch processing</a>.</p>\n<p><strong>What stops abuse?</strong> Anonymous creation needs <a href=\"/learn/rate-limiting\">rate limiting</a> by IP address. Pastes are untrusted text shown in browsers, so serve raw pastes as <code>text/plain</code> with <code>X-Content-Type-Options: nosniff</code>, escape text in the HTML page, and ideally serve raw pastes from a separate domain so a crafted paste can't run script alongside your cookies. See <a href=\"/learn/security#cross-site-scripting-xss\">cross-site scripting</a>.</p>","headings":[{"id":"trade-offs-and-follow-up-questions","title":"Trade-offs and follow-up questions","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>The product team now expects an average paste of 10 KB instead of 1 KB. Redo the storage estimate. Does it change where the text should live?</summary>\n<p>The metadata is unchanged at about 98 GB over three years. The text grows tenfold: 10 KB × 10 million = 100 GB a month, and 100 GB × 36 = 3.6 TB over three years.</p>\n<p>That strengthens the case for the object store. With text in the database, it would hold about 3.7 TB, so backups, replicas and failover would move almost forty times more data than the metadata needs. The inline option weakens too, since most pastes would now be above a 2 kB threshold. Read bandwidth rises from about 0.4 to about 3.2 megabits a second, which is still small.</p>\n</details>\n<details>\n<summary>A 300 KB paste goes viral and its raw URL gets 5,000 requests a second. What breaks in the first design, and what handles it in the scaled one?</summary>\n<p>In the first design, every request hits the single Read API, which reads the database and then the object store. At 100–200 ms per object read, <a href=\"/learn/performance-and-latency#littles-law\">Little's Law</a> puts 5,000 × 0.1 to 0.2 s = 500 to 1,000 requests in flight at once, each holding a connection and memory. Outgoing traffic is 5,000 × 300 KB = 1.5 GB a second, about 12 gigabits a second, more than a typical server's network card.</p>\n<p>In the scaled design, the CDN answers nearly all of it from the edge. Your servers see about one request per edge location each time the cached copy expires, and those hit the memory cache. The remaining risk is the moment the copy expires everywhere at once, which request coalescing at the CDN or a lock in the cache layer handles.</p>\n</details>\n<details>\n<summary>A user sets a 10-minute expiry. Thirty minutes later a colleague can still open the raw URL, but the HTML page says the paste has expired. What's the likely bug?</summary>\n<p>The HTML page goes through the Read API, which checks <code>expires_at</code> on every request, so the data is right. Something in front of the raw URL is serving a stale copy, most likely the CDN. The raw response was cached with a fixed <code>max-age</code> such as an hour, or with no <code>Cache-Control</code> header, in which case CloudFront keeps it for its default TTL, 24 hours unless changed.</p>\n<p>The fix is to set <code>max-age</code> on raw responses to the smaller of the default and the seconds left, and to check that the CDN's minimum TTL isn't raising it. A lifecycle rule wouldn't help, because rules count in whole days and run late.</p>\n</details>\n<details>\n<summary>Monthly view counts fell 60% in the month the CDN launched, yet the CDN's own reports show traffic grew. What happened?</summary>\n<p>The analytics job counts from web server access logs. Since launch, most raw-paste views have been answered at the edge and never reached a web server, so they're missing from the job's input.</p>\n<p>Ship the CDN's request logs to the object store and teach the mapper both formats. Then avoid counting twice: a CDN miss appears in both the CDN log and the web server log. Count from the CDN log for URLs the CDN serves and from the web server log for the rest.</p>\n</details>\n<details>\n<summary>A Write API server crashes after uploading the text but before inserting the row. What's left behind, and does anyone notice?</summary>\n<p>The object store holds a file under a random key that no row points to. The client saw an error or timeout and probably retries, which uploads a new file and inserts a row. Nobody has a code for the first file, so nobody can reach it, and it only costs a little storage. The orphan sweep lists objects over a day old, checks which keys appear in <code>paste_path</code>, and deletes the rest.</p>\n<p>With the opposite order, the crash would leave a row pointing to a missing file, and anyone with the code would get an error. That's why the text goes first, under a key independent of the code.</p>\n</details>\n<details>\n<summary>Extend the design with a \"burn after reading\" option, where a paste can be viewed exactly once. What changes?</summary>\n<p>The read must claim the paste atomically, so two people opening the link together can't both see it. A single <code>DELETE FROM pastes WHERE shortlink = %s RETURNING paste_path</code>, or a conditional update that sets a \"burned\" flag, succeeds for exactly one request. It must run on the primary, because replicas lag.</p>\n<p>Every cache must be skipped: the Read API doesn't cache these pastes, and responses carry <code>Cache-Control: no-store</code> so neither the CDN nor the browser keeps a copy. There's also a product trap. Chat apps and email clients often fetch a link to build a preview before anyone clicks, which would burn the paste. A common fix is a \"click to reveal\" page, burning the paste only on that second request.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html\" target=\"_blank\" rel=\"noopener noreferrer\">Data protection in Amazon S3</a>: S3's durability and availability design targets and how objects are spread across Availability Zones.</li>\n<li><a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon S3 data consistency model</a>: the read-after-write guarantee that makes \"upload the text, then insert the row\" safe.</li>\n<li><a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html\" target=\"_blank\" rel=\"noopener noreferrer\">Best practices: optimizing Amazon S3 performance</a>: per-prefix request rates and typical small-object latency.</li>\n<li><a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html\" target=\"_blank\" rel=\"noopener noreferrer\">Expiring objects</a>, <a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html\" target=\"_blank\" rel=\"noopener noreferrer\">Lifecycle configuration elements</a> and <a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html\" target=\"_blank\" rel=\"noopener noreferrer\">Setting an S3 Lifecycle configuration</a>: how lifecycle expiry rounds to midnight UTC, why it can run days late, and when billing stops.</li>\n<li><a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html\" target=\"_blank\" rel=\"noopener noreferrer\">Understanding and managing Amazon S3 storage classes</a>: the table of minimum billable object sizes, which is why small pastes belong in S3 Standard.</li>\n<li><a href=\"https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html\" target=\"_blank\" rel=\"noopener noreferrer\">Manage how long content stays in the cache (expiration)</a>: how CloudFront combines <code>Cache-Control</code> headers with its own TTL settings.</li>\n<li><a href=\"https://www.postgresql.org/docs/current/storage-toast.html\" target=\"_blank\" rel=\"noopener noreferrer\">PostgreSQL: TOAST</a>: how PostgreSQL moves large values out of the main table, a built-in version of the inline-or-external choice.</li>\n<li><a href=\"https://www.usenix.org/legacy/events/osdi04/tech/full_papers/dean/dean_html/index.html\" target=\"_blank\" rel=\"noopener noreferrer\">MapReduce: Simplified Data Processing on Large Clusters</a>: Dean and Ghemawat's 2004 paper. Its examples include counting URL access frequency from request logs, the job in this chapter.</li>\n<li><a href=\"https://pastebin.com/faq\" target=\"_blank\" rel=\"noopener noreferrer\">Pastebin FAQ</a>: the real service's size limits, visibility options and policy on pastes that never expire.</li>\n<li><a href=\"https://gist.github.com/jboner/2841832\" target=\"_blank\" rel=\"noopener noreferrer\">Latency numbers every programmer should know</a>: Jonas Bonér's table of rough timings, including the memory, SSD and disk figures used here.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}