System Design Handbook · llms.txt
System Design Handbook
A free, open-source course in designing large-scale systems, in 77 chapters: how to run a system design interview, the core trade-offs, building blocks from DNS to databases and caches, deeper concepts such as consistent hashing and consensus, worked system design and object-oriented design questions, and reference material. Each chapter lists its sources under Further reading.
AI Infrastructure Engineer Handbook: Inference from kernels to clusters: how models run on GPUs, how serving engines schedule them, and how to make them fast and cheap. Each chapter has a lab.
C++ for AI Infrastructure Engineers: From your first CMake build to CUDA kernels, for engineers who know Python: memory, ownership, performance, concurrency and the GPU, ending with an LLM inference engine you write yourself.
Start here
How to use the handbook, how to run an interview answer, how systems grow, and how to estimate.
- How to use this handbook: System design is the work of deciding how a piece of software is split into parts, where its data lives, and how those parts talk to each other so the whole thing stays fast,…
- The system design interview, step by step: In a system design interview, you're given a short, open prompt such as "Design a URL shortener" or "Design the Twitter timeline," and about 45 minutes to talk through how you'd…
- Scalability: If you're new to system design, start here. This chapter covers the basic ideas behind every large website, in the order a growing site usually needs them: vertical and horizontal…
- Back-of-the-envelope estimation: Before you draw boxes, you need a rough idea of how big the problem is. Is it 50 requests per second or 500,000? Gigabytes or petabytes? The answer decides whether one database…
Core trade-offs
The tensions every design balances: speed, scale, availability and consistency.
- Performance and latency: The classic definitions of these ideas come from three short sources: Werner Vogels's 2006 blog post A Word on Scalability, Jonas Bonér's slide deck Scalability, Availability &…
- CAP and consistency: Good short introductions to CAP include Robert Greiner's blog post CAP Theorem: Revisited (2014), Kaushik Sathupadi's A plain English introduction to CAP theorem, Henry Robinson's…
- Availability: We'll keep using Crumb, the recipe site from the Scalability chapter. By now Crumb has several app servers behind a load balancer, a cache, and a database with a copy on a second…
Building blocks
The components a request passes through, from DNS to the database.
- DNS and CDNs: We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb runs several app servers behind a load balancer in one data center in Virginia. Recipe photos…
- Load balancers: We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb runs several identical app servers behind a load balancer. Sessions live in Redis, photos live…
- Reverse proxies, API gateways and microservices: We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb has several app servers behind a load balancer, a cache, a database with read replicas, and…
- Relational databases and scaling SQL: We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb stores its data in a relational database with four main tables: users, recipes, comments and…
- NoSQL and choosing a database: We'll keep using Crumb, the recipe site from the Scalability chapter. Crumb's core data lives in a relational database (say PostgreSQL) with read replicas. Crumb now wants four…
- Database internals: This chapter builds on two earlier ones. Relational databases covers ACID transactions, isolation levels, replication, federation, the basics of sharding and SQL tuning. NoSQL and…
- Caching: We'll keep using Crumb, the recipe site from the Scalability chapter. Crumb already runs several app servers behind a load balancer, with a Redis cache in front of its database.…
- Queues and asynchronous work: We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb has several slow jobs: resizing uploaded photos, sending a weekly newsletter, working out which…
- Network protocols and APIs: We'll keep using Crumb, the recipe site from the Scalability chapter. Crumb now has a website, a mobile app, app servers behind a load balancer, a memcached layer and a database.…
- Security: Security is a broad topic. Unless you have considerable experience, a security background, or are applying for a position that requires knowledge of security, you probably won't…
Deeper concepts
The techniques large systems lean on, from consistent hashing to consensus.
- Consistent hashing: Once data or cache entries are spread across several machines, every request has to answer one question: which machine holds this key? The obvious answer, hashing the key and…
- Rate limiting: A rate limiter caps how many requests a client can make in a period of time and turns away the rest. It protects shared capacity from one noisy client, blunts abuse like…
- Unique ID generation: Every row, message, order and event needs an identifier. On a single database you let it hand out auto-increment numbers and never think about it again. Once you shard, run…
- Message queues and event streams: The queues and asynchronous work chapter explains why queues exist: put slow work on a queue and let workers pick it up later. Interviewers then ask the next questions. What…
- Timeouts, retries and idempotency: Once a request crosses a network, three outcomes are possible: it succeeds, it fails, or you don't know. The third case is the hard one. The server may never have received the…
- Distributed transactions and sagas: Inside one relational database, a transaction makes several writes succeed or fail together. The relational databases chapter covers the ACID properties that make this work. Split…
- Consensus, leader election and coordination: Many designs quietly assume that some decision is made exactly once: which database node is the primary, which worker owns a shard, who holds a lock, whether a config change…
- Real-time updates: polling, WebSockets and server-sent events: Plain HTTP is request/response: the client asks, the server answers, and the server has no way to speak first. Plenty of features need the server to speak first. Examples include…
- Monitoring, logging and tracing: A system you can't see into is a system you can't run. When checkout gets slow at 2 a.m., someone needs to answer three questions fast: is it actually broken for users, where is…
- Batch processing and distributed computing: Most system design questions are about online systems: a user sends a request, a service reads an index or a cache, and a response comes back in tens of milliseconds. Latency and…
System design questions
Worked designs with use cases, constraints, estimates, diagrams and scaling discussion.
- Design a text-sharing service like Pastebin: A text-sharing service lets someone paste a block of text into a web form and get back a short link, such as https://paste.example/r7k2x9a. Anyone who opens the link sees the…
- Design a URL shortener (like TinyURL or Bitly): A URL shortener takes a long web address, such as https://en.wikipedia.org/wiki/Representational_state_transfer#Architectural_constraints, and gives back a short alias such as…
- Design the Twitter timeline and search: Twitter is a service where people post short public messages called tweets. When you follow someone, their tweets show up in your home timeline, the feed of tweets from everyone…
- Design a social media news feed (like Facebook, Instagram, or Twitter): A news feed collects recent posts from the people and pages a user follows, orders or ranks them, and shows them as one scrolling list. When the user opens the app, they expect a…
- Design a web crawler: A web crawler is a program that downloads web pages, finds the links on them, and then downloads the pages those links point to. For example, the crawler fetches…
- Design a web search engine: A web search engine takes a few words and, in a couple of hundred milliseconds, returns the ten most useful pages out of tens of billions. Behind that are three systems that run…
- Design search autocomplete (typeahead): Autocomplete suggests whole queries while you type a prefix: type "how to bo" and you see "how to boil eggs", "how to boost wifi signal" and a few more. It saves keystrokes,…
- Design a key-value cache for a search engine's query results: When you type cheap flights to paris into a search engine, a lot of work happens before the results page comes back. The engine looks up every word in its index, finds the pages…
- Design a distributed key-value store: At its simplest, a key-value store is a hash table: put a key, get it back. This question asks you to build one that spans dozens of machines, keeps working when some of them die,…
- Design the data structures for a social network: A social network stores people and the friendships between them. When you open a stranger's profile on a professional network, the site often shows how you're connected: "You →…
- Design Amazon's sales rank by category: Open a product page on a large online store and you will often see a line such as "#3 in Kitchen Scales". Browse a category and you can sort it by "Best Sellers". Both come from a…
- Design a real-time gaming leaderboard: Online games, competitive programming sites and fitness apps use leaderboards to keep people coming back. Players expect their score and rank to change as soon as a match ends,…
- Design a personal finance tracker like Mint: A personal finance tracker pulls every purchase from your bank and card accounts into one place. Say you link your checking account and two credit cards. The next morning the app…
- Design a payment system and gateway: A payment system takes money from a customer, records where it went, and eventually pays it out to a merchant. Most of the interview designs in this course can tolerate a little…
- Design a notification system: Almost every product sends notifications: a one-time login code, "your order shipped", "your friend commented", "presale starts in an hour". Each one is a small message, but the…
- Design a chat system: A chat system lets people exchange short messages in near real time, one to one or in groups, across several devices each. It looks simple until you list what users take for…
- Design a collaborative document editor (like Google Docs or Figma): A collaborative editor lets several people change the same document at the same time and see each other's changes within a fraction of a second. There is no save button and no…
- Design a file sync service (like Dropbox or Google Drive): A file sync service keeps a folder identical across a user's laptop, desktop and phone, and across everyone it's shared with. Save a file on one machine and, a few seconds later,…
- Design a video streaming platform: A video platform lets creators upload videos and lets viewers watch them on anything from a phone on a train to a living-room TV. The upload side is a batch pipeline that turns…
- Design a personalized recommendation system: Open the home screen of YouTube, Netflix, Spotify or TikTok and you get a ranked list built for you, from a catalog of hundreds of millions of items, using your long-term taste,…
- Design a system that scales to millions of users on AWS: Picture a website where people post short entries, about 1 KB of text each, and read each other's. On launch day it has one user: you. Three years later it has 10 million users,…
Object-oriented design questions
Class design exercises with working Python solutions.
- Design a hash map: A hash map (also called a hash table) stores key-value pairs and finds the value for a key in about the same time whether it holds ten pairs or ten million. A contacts app that…
- Design an LRU cache: A cache keeps copies of recent answers in fast memory so that repeated requests don't have to redo slow work. A search engine is a good example. Computing the results for "weather…
- Design a call center: Picture a support line with three levels of staff. Operators answer the phone. When an operator can't solve a caller's problem, they hand the call to a supervisor. When a…
- Design a deck of cards: Lay a standard pack of playing cards on the table: 52 cards, four suits (hearts, diamonds, clubs and spades), and in each suit an ace, the numbers 2 to 10, and a jack, queen and…
- Design a parking lot: Picture a three-storey garage next to a train station. Motorcycles, cars and the occasional bus drive in at the entrance. The garage has to decide where each one goes, refuse it…
- Design a chat server: Ana wants to message her friend Ben. First she has to find him and send a friend request, and Ben has to accept it. Then the two of them can talk in a private chat. Later Ben…
AI systems
How language models are served, from GPUs and inference engines to retrieval, agents, evaluation and security.
- How LLM inference works: Inference is the work a model server does for each request. It takes the prompt, runs a trained model, and returns the reply. Training happens once, in a separate and much larger…
- GPUs and accelerators: The hardware under a model decides how large a model you can serve, how fast one user gets tokens, how many users one machine can carry, and what each token costs. This chapter…
- Inference serving engines: A model server can answer one user with a few lines of code: load the weights, call the model, return the text. That design breaks down when many users share the same GPUs. Each…
- Estimating AI workloads: A web estimate asks how many requests a second you need and how many bytes they move. An AI estimate adds two questions. Does the model fit in GPU memory, with room left for the…
- Running inference at scale: A model server on one machine answers quickly while its requests fit in its GPUs. A product needs many servers, and a fleet of GPU servers differs from the web fleet in the load…
- LLM gateways: A model provider sells access to a language model through an HTTP API. A gateway sits between your applications and those providers. Each application calls one internal endpoint,…
- Embeddings and vector search: A keyword search finds the documents that contain the query's words, so a search for "get my money back" misses a page titled "Refund policy" unless that page also says "money…
- Retrieval-augmented generation: A plain language model answers from what it learned in training. That knowledge is fixed when training ends, the model can't look anything up while it writes, and it doesn't say…
- Agents and tool use: A model call takes a prompt and returns text. It cannot look up an order, send an email or issue a refund, because it has no connection to those systems. An agent adds that…
- Evaluating and observing AI systems: A web service gets two checks on every request. The status code says whether the call worked, and tests say whether the code does what its author meant. A language model call…
- Security for AI systems: A parameterized SQL query is safe because the database can tell the query text from the values. A language model cannot make that distinction. It reads one stream of tokens, the…
- Training and fine-tuning infrastructure: Serving handles requests as they arrive. Training is a batch job that runs for weeks, spreads across thousands of GPUs and has to survive hardware failures. The Llama 3 paper's…
AI system design questions
Worked designs for products built on language and image models, in the same four steps.
- Design a chat assistant (like ChatGPT or Claude.ai): A chat assistant answers each message with a reply that appears word by word. The first words should arrive within about a second, and the rest keep coming while the reader…
- Design a model inference platform (serve many LLMs to many customers): A model inference platform takes requests for language models and returns their replies. Customers name a model in one API call, and the platform decides which GPUs run it. Hosted…
- Design an LLM gateway (one API in front of many model providers for a whole company): A company with 5,000 engineers has about 200 internal apps that call language models: a support bot, a code review assistant, a contract summarizer, nightly classification jobs…
- Design a question-answering system over company documents (RAG): An employee asks an internal assistant, "What is the limit for a client dinner in Berlin?" The answer should come from the finance wiki, cite the page, reflect yesterday's policy…
- Design an AI coding assistant (inline completions and an agent mode in the editor): An AI coding assistant lives in the developer's editor and has two jobs. The first is inline completion: while the developer types, the editor shows grey "ghost text" after the…
- Design a platform for running AI agents (long-running, tool-using tasks for many customers): A support-software company wants every new ticket labelled and answered in draft before a person opens it. A research firm wants a report on twenty competitors every Monday. A…
- Design a vector database (a managed service for storing and searching embeddings): A developer creates an index called support-docs with 1,536 dimensions and cosine similarity, then upserts 200,000 vectors with metadata such as {"lang": "de", "product":…
- Design an image generation service (text to image, like Midjourney or DALL·E): An image generation service turns a prompt such as "a lighthouse in a storm, ink wash style" into a grid of four pictures. It also edits uploaded images, repaints masked regions…
Reference
Numbers to memorise, more practice questions, and real-world architectures.
- Numbers to know: Suppose an interviewer asks: "How long does it take to read 1 GB from an SSD?" You don't need a benchmark to answer. If you remember that reading 1 MB in sequence from an SSD…
- Practice questions: Reading about system design teaches you the parts. Answering questions out loud teaches you to use them. This chapter is a bank of questions to practise on, with pointers to the…
- Real-world architectures: The rest of this handbook teaches ideas one at a time: caching, replication, sharding, queues, consensus. Real systems use many of them at once, and the interesting part is how…
About
The flashcards, what is being written next, contributing, credits and the license.
- About this handbook: The System Design Handbook is a free, open-source course in system design. It teaches how large software systems are built: how they stay fast, how they survive failures, and how…