# System Design Handbook > A free, open-source course in designing large-scale systems, in 77 chapters: how to run a system design interview, the core trade-offs, building blocks from DNS to databases and caches, deeper concepts such as consistent hashing and consensus, worked system design and object-oriented design questions, and reference material. Each chapter lists its sources under Further reading. Each chapter is available as Markdown at `/learn/.md`, and the whole course is in one file at [llms-full.txt](https://system.tangentbyte.com/llms-full.txt). ## Start here - [How to use this handbook](https://system.tangentbyte.com/learn/how-to-use-this-handbook.md): System design is the work of deciding how a piece of software is split into parts, where its data lives, and how those parts talk to each other so the whole thing stays fast,… - [The system design interview, step by step](https://system.tangentbyte.com/learn/system-design-interview.md): In a system design interview, you're given a short, open prompt such as "Design a URL shortener" or "Design the Twitter timeline," and about 45 minutes to talk through how you'd… - [Scalability](https://system.tangentbyte.com/learn/scalability.md): If you're new to system design, start here. This chapter covers the basic ideas behind every large website, in the order a growing site usually needs them: vertical and horizontal… - [Back-of-the-envelope estimation](https://system.tangentbyte.com/learn/estimation.md): Before you draw boxes, you need a rough idea of how big the problem is. Is it 50 requests per second or 500,000? Gigabytes or petabytes? The answer decides whether one database… ## Core trade-offs - [Performance and latency](https://system.tangentbyte.com/learn/performance-and-latency.md): The classic definitions of these ideas come from three short sources: Werner Vogels's 2006 blog post A Word on Scalability, Jonas Bonér's slide deck Scalability, Availability &… - [CAP and consistency](https://system.tangentbyte.com/learn/cap-and-consistency.md): Good short introductions to CAP include Robert Greiner's blog post CAP Theorem: Revisited (2014), Kaushik Sathupadi's A plain English introduction to CAP theorem, Henry Robinson's… - [Availability](https://system.tangentbyte.com/learn/availability.md): We'll keep using Crumb, the recipe site from the Scalability chapter. By now Crumb has several app servers behind a load balancer, a cache, and a database with a copy on a second… ## Building blocks - [DNS and CDNs](https://system.tangentbyte.com/learn/dns-and-cdn.md): We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb runs several app servers behind a load balancer in one data center in Virginia. Recipe photos… - [Load balancers](https://system.tangentbyte.com/learn/load-balancing.md): We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb runs several identical app servers behind a load balancer. Sessions live in Redis, photos live… - [Reverse proxies, API gateways and microservices](https://system.tangentbyte.com/learn/reverse-proxy-and-api-gateway.md): We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb has several app servers behind a load balancer, a cache, a database with read replicas, and… - [Relational databases and scaling SQL](https://system.tangentbyte.com/learn/relational-databases.md): We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb stores its data in a relational database with four main tables: users, recipes, comments and… - [NoSQL and choosing a database](https://system.tangentbyte.com/learn/nosql-databases.md): We'll keep using Crumb, the recipe site from the Scalability chapter. Crumb's core data lives in a relational database (say PostgreSQL) with read replicas. Crumb now wants four… - [Database internals](https://system.tangentbyte.com/learn/database-internals.md): This chapter builds on two earlier ones. Relational databases covers ACID transactions, isolation levels, replication, federation, the basics of sharding and SQL tuning. NoSQL and… - [Caching](https://system.tangentbyte.com/learn/cache.md): We'll keep using Crumb, the recipe site from the Scalability chapter. Crumb already runs several app servers behind a load balancer, with a Redis cache in front of its database.… - [Queues and asynchronous work](https://system.tangentbyte.com/learn/queues-and-async.md): We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb has several slow jobs: resizing uploaded photos, sending a weekly newsletter, working out which… - [Network protocols and APIs](https://system.tangentbyte.com/learn/protocols-and-apis.md): We'll keep using Crumb, the recipe site from the Scalability chapter. Crumb now has a website, a mobile app, app servers behind a load balancer, a memcached layer and a database.… - [Security](https://system.tangentbyte.com/learn/security.md): Security is a broad topic. Unless you have considerable experience, a security background, or are applying for a position that requires knowledge of security, you probably won't… ## Deeper concepts - [Consistent hashing](https://system.tangentbyte.com/learn/consistent-hashing.md): Once data or cache entries are spread across several machines, every request has to answer one question: which machine holds this key? The obvious answer, hashing the key and… - [Rate limiting](https://system.tangentbyte.com/learn/rate-limiting.md): A rate limiter caps how many requests a client can make in a period of time and turns away the rest. It protects shared capacity from one noisy client, blunts abuse like… - [Unique ID generation](https://system.tangentbyte.com/learn/unique-ids.md): Every row, message, order and event needs an identifier. On a single database you let it hand out auto-increment numbers and never think about it again. Once you shard, run… - [Message queues and event streams](https://system.tangentbyte.com/learn/event-streaming.md): The queues and asynchronous work chapter explains why queues exist: put slow work on a queue and let workers pick it up later. Interviewers then ask the next questions. What… - [Timeouts, retries and idempotency](https://system.tangentbyte.com/learn/reliability-patterns.md): Once a request crosses a network, three outcomes are possible: it succeeds, it fails, or you don't know. The third case is the hard one. The server may never have received the… - [Distributed transactions and sagas](https://system.tangentbyte.com/learn/distributed-transactions.md): Inside one relational database, a transaction makes several writes succeed or fail together. The relational databases chapter covers the ACID properties that make this work. Split… - [Consensus, leader election and coordination](https://system.tangentbyte.com/learn/consensus.md): Many designs quietly assume that some decision is made exactly once: which database node is the primary, which worker owns a shard, who holds a lock, whether a config change… - [Real-time updates: polling, WebSockets and server-sent events](https://system.tangentbyte.com/learn/realtime.md): Plain HTTP is request/response: the client asks, the server answers, and the server has no way to speak first. Plenty of features need the server to speak first. Examples include… - [Monitoring, logging and tracing](https://system.tangentbyte.com/learn/observability.md): A system you can't see into is a system you can't run. When checkout gets slow at 2 a.m., someone needs to answer three questions fast: is it actually broken for users, where is… - [Batch processing and distributed computing](https://system.tangentbyte.com/learn/batch-processing.md): Most system design questions are about online systems: a user sends a request, a service reads an index or a cache, and a response comes back in tens of milliseconds. Latency and… ## System design questions - [Design a text-sharing service like Pastebin](https://system.tangentbyte.com/learn/design-pastebin.md): A text-sharing service lets someone paste a block of text into a web form and get back a short link, such as https://paste.example/r7k2x9a. Anyone who opens the link sees the… - [Design a URL shortener (like TinyURL or Bitly)](https://system.tangentbyte.com/learn/design-url-shortener.md): A URL shortener takes a long web address, such as https://en.wikipedia.org/wiki/Representational_state_transfer#Architectural_constraints, and gives back a short alias such as… - [Design the Twitter timeline and search](https://system.tangentbyte.com/learn/design-twitter.md): Twitter is a service where people post short public messages called tweets. When you follow someone, their tweets show up in your home timeline, the feed of tweets from everyone… - [Design a social media news feed (like Facebook, Instagram, or Twitter)](https://system.tangentbyte.com/learn/design-social-news-feed.md): A news feed collects recent posts from the people and pages a user follows, orders or ranks them, and shows them as one scrolling list. When the user opens the app, they expect a… - [Design a web crawler](https://system.tangentbyte.com/learn/design-web-crawler.md): A web crawler is a program that downloads web pages, finds the links on them, and then downloads the pages those links point to. For example, the crawler fetches… - [Design a web search engine](https://system.tangentbyte.com/learn/design-search-engine.md): A web search engine takes a few words and, in a couple of hundred milliseconds, returns the ten most useful pages out of tens of billions. Behind that are three systems that run… - [Design search autocomplete (typeahead)](https://system.tangentbyte.com/learn/design-autocomplete.md): Autocomplete suggests whole queries while you type a prefix: type "how to bo" and you see "how to boil eggs", "how to boost wifi signal" and a few more. It saves keystrokes,… - [Design a key-value cache for a search engine's query results](https://system.tangentbyte.com/learn/design-query-cache.md): When you type cheap flights to paris into a search engine, a lot of work happens before the results page comes back. The engine looks up every word in its index, finds the pages… - [Design a distributed key-value store](https://system.tangentbyte.com/learn/design-key-value-store.md): At its simplest, a key-value store is a hash table: put a key, get it back. This question asks you to build one that spans dozens of machines, keeps working when some of them die,… - [Design the data structures for a social network](https://system.tangentbyte.com/learn/design-social-graph.md): A social network stores people and the friendships between them. When you open a stranger's profile on a professional network, the site often shows how you're connected: "You →… - [Design Amazon's sales rank by category](https://system.tangentbyte.com/learn/design-sales-rank.md): Open a product page on a large online store and you will often see a line such as "#3 in Kitchen Scales". Browse a category and you can sort it by "Best Sellers". Both come from a… - [Design a real-time gaming leaderboard](https://system.tangentbyte.com/learn/design-gaming-leaderboard.md): Online games, competitive programming sites and fitness apps use leaderboards to keep people coming back. Players expect their score and rank to change as soon as a match ends,… - [Design a personal finance tracker like Mint](https://system.tangentbyte.com/learn/design-mint.md): A personal finance tracker pulls every purchase from your bank and card accounts into one place. Say you link your checking account and two credit cards. The next morning the app… - [Design a payment system and gateway](https://system.tangentbyte.com/learn/design-payment-system.md): A payment system takes money from a customer, records where it went, and eventually pays it out to a merchant. Most of the interview designs in this course can tolerate a little… - [Design a notification system](https://system.tangentbyte.com/learn/design-notification-system.md): Almost every product sends notifications: a one-time login code, "your order shipped", "your friend commented", "presale starts in an hour". Each one is a small message, but the… - [Design a chat system](https://system.tangentbyte.com/learn/design-chat-system.md): A chat system lets people exchange short messages in near real time, one to one or in groups, across several devices each. It looks simple until you list what users take for… - [Design a collaborative document editor (like Google Docs or Figma)](https://system.tangentbyte.com/learn/design-collaborative-document.md): A collaborative editor lets several people change the same document at the same time and see each other's changes within a fraction of a second. There is no save button and no… - [Design a file sync service (like Dropbox or Google Drive)](https://system.tangentbyte.com/learn/design-file-sync.md): A file sync service keeps a folder identical across a user's laptop, desktop and phone, and across everyone it's shared with. Save a file on one machine and, a few seconds later,… - [Design a video streaming platform](https://system.tangentbyte.com/learn/design-video-streaming.md): A video platform lets creators upload videos and lets viewers watch them on anything from a phone on a train to a living-room TV. The upload side is a batch pipeline that turns… - [Design a personalized recommendation system](https://system.tangentbyte.com/learn/design-recommendation-system.md): Open the home screen of YouTube, Netflix, Spotify or TikTok and you get a ranked list built for you, from a catalog of hundreds of millions of items, using your long-term taste,… - [Design a system that scales to millions of users on AWS](https://system.tangentbyte.com/learn/design-scaling-aws.md): Picture a website where people post short entries, about 1 KB of text each, and read each other's. On launch day it has one user: you. Three years later it has 10 million users,… ## Object-oriented design questions - [Design a hash map](https://system.tangentbyte.com/learn/ood-hash-table.md): A hash map (also called a hash table) stores key-value pairs and finds the value for a key in about the same time whether it holds ten pairs or ten million. A contacts app that… - [Design an LRU cache](https://system.tangentbyte.com/learn/ood-lru-cache.md): A cache keeps copies of recent answers in fast memory so that repeated requests don't have to redo slow work. A search engine is a good example. Computing the results for "weather… - [Design a call center](https://system.tangentbyte.com/learn/ood-call-center.md): Picture a support line with three levels of staff. Operators answer the phone. When an operator can't solve a caller's problem, they hand the call to a supervisor. When a… - [Design a deck of cards](https://system.tangentbyte.com/learn/ood-deck-of-cards.md): Lay a standard pack of playing cards on the table: 52 cards, four suits (hearts, diamonds, clubs and spades), and in each suit an ace, the numbers 2 to 10, and a jack, queen and… - [Design a parking lot](https://system.tangentbyte.com/learn/ood-parking-lot.md): Picture a three-storey garage next to a train station. Motorcycles, cars and the occasional bus drive in at the entrance. The garage has to decide where each one goes, refuse it… - [Design a chat server](https://system.tangentbyte.com/learn/ood-online-chat.md): Ana wants to message her friend Ben. First she has to find him and send a friend request, and Ben has to accept it. Then the two of them can talk in a private chat. Later Ben… ## AI systems - [How LLM inference works](https://system.tangentbyte.com/learn/llm-inference.md): Inference is the work a model server does for each request. It takes the prompt, runs a trained model, and returns the reply. Training happens once, in a separate and much larger… - [GPUs and accelerators](https://system.tangentbyte.com/learn/gpus-for-ai.md): The hardware under a model decides how large a model you can serve, how fast one user gets tokens, how many users one machine can carry, and what each token costs. This chapter… - [Inference serving engines](https://system.tangentbyte.com/learn/inference-engines.md): A model server can answer one user with a few lines of code: load the weights, call the model, return the text. That design breaks down when many users share the same GPUs. Each… - [Estimating AI workloads](https://system.tangentbyte.com/learn/ai-estimation.md): A web estimate asks how many requests a second you need and how many bytes they move. An AI estimate adds two questions. Does the model fit in GPU memory, with room left for the… - [Running inference at scale](https://system.tangentbyte.com/learn/inference-fleets.md): A model server on one machine answers quickly while its requests fit in its GPUs. A product needs many servers, and a fleet of GPU servers differs from the web fleet in the load… - [LLM gateways](https://system.tangentbyte.com/learn/llm-gateways.md): A model provider sells access to a language model through an HTTP API. A gateway sits between your applications and those providers. Each application calls one internal endpoint,… - [Embeddings and vector search](https://system.tangentbyte.com/learn/vector-search.md): A keyword search finds the documents that contain the query's words, so a search for "get my money back" misses a page titled "Refund policy" unless that page also says "money… - [Retrieval-augmented generation](https://system.tangentbyte.com/learn/rag.md): A plain language model answers from what it learned in training. That knowledge is fixed when training ends, the model can't look anything up while it writes, and it doesn't say… - [Agents and tool use](https://system.tangentbyte.com/learn/ai-agents.md): A model call takes a prompt and returns text. It cannot look up an order, send an email or issue a refund, because it has no connection to those systems. An agent adds that… - [Evaluating and observing AI systems](https://system.tangentbyte.com/learn/ai-evals-and-observability.md): A web service gets two checks on every request. The status code says whether the call worked, and tests say whether the code does what its author meant. A language model call… - [Security for AI systems](https://system.tangentbyte.com/learn/ai-security.md): A parameterized SQL query is safe because the database can tell the query text from the values. A language model cannot make that distinction. It reads one stream of tokens, the… - [Training and fine-tuning infrastructure](https://system.tangentbyte.com/learn/training-infrastructure.md): Serving handles requests as they arrive. Training is a batch job that runs for weeks, spreads across thousands of GPUs and has to survive hardware failures. The Llama 3 paper's… ## AI system design questions - [Design a chat assistant (like ChatGPT or Claude.ai)](https://system.tangentbyte.com/learn/design-chat-assistant.md): A chat assistant answers each message with a reply that appears word by word. The first words should arrive within about a second, and the rest keep coming while the reader… - [Design a model inference platform (serve many LLMs to many customers)](https://system.tangentbyte.com/learn/design-inference-platform.md): A model inference platform takes requests for language models and returns their replies. Customers name a model in one API call, and the platform decides which GPUs run it. Hosted… - [Design an LLM gateway (one API in front of many model providers for a whole company)](https://system.tangentbyte.com/learn/design-llm-gateway.md): A company with 5,000 engineers has about 200 internal apps that call language models: a support bot, a code review assistant, a contract summarizer, nightly classification jobs… - [Design a question-answering system over company documents (RAG)](https://system.tangentbyte.com/learn/design-rag-qa.md): An employee asks an internal assistant, "What is the limit for a client dinner in Berlin?" The answer should come from the finance wiki, cite the page, reflect yesterday's policy… - [Design an AI coding assistant (inline completions and an agent mode in the editor)](https://system.tangentbyte.com/learn/design-coding-assistant.md): An AI coding assistant lives in the developer's editor and has two jobs. The first is inline completion: while the developer types, the editor shows grey "ghost text" after the… - [Design a platform for running AI agents (long-running, tool-using tasks for many customers)](https://system.tangentbyte.com/learn/design-agent-platform.md): A support-software company wants every new ticket labelled and answered in draft before a person opens it. A research firm wants a report on twenty competitors every Monday. A… - [Design a vector database (a managed service for storing and searching embeddings)](https://system.tangentbyte.com/learn/design-vector-database.md): A developer creates an index called support-docs with 1,536 dimensions and cosine similarity, then upserts 200,000 vectors with metadata such as {"lang": "de", "product":… - [Design an image generation service (text to image, like Midjourney or DALL·E)](https://system.tangentbyte.com/learn/design-image-generation.md): An image generation service turns a prompt such as "a lighthouse in a storm, ink wash style" into a grid of four pictures. It also edits uploaded images, repaints masked regions… ## Reference - [Numbers to know](https://system.tangentbyte.com/learn/numbers-to-know.md): Suppose an interviewer asks: "How long does it take to read 1 GB from an SSD?" You don't need a benchmark to answer. If you remember that reading 1 MB in sequence from an SSD… - [Practice questions](https://system.tangentbyte.com/learn/practice-questions.md): Reading about system design teaches you the parts. Answering questions out loud teaches you to use them. This chapter is a bank of questions to practise on, with pointers to the… - [Real-world architectures](https://system.tangentbyte.com/learn/real-world-architectures.md): The rest of this handbook teaches ideas one at a time: caching, replication, sharding, queues, consensus. Real systems use many of them at once, and the interesting part is how… ## Optional - [About this handbook](https://system.tangentbyte.com/learn/about.md): The System Design Handbook is a free, open-source course in system design. It teaches how large software systems are built: how they stay fast, how they survive failures, and how… ## AI Infrastructure Engineer Handbook: Orientation - [The inference engineer](https://system.tangentbyte.com/learn/inference-engineer-role.md): Training produces a model once. Serving it runs every day, for every user, and that is where most of the GPU bill goes. An inference engineer makes the serving fast, cheap and… - [Models and machines](https://system.tangentbyte.com/learn/models-and-machines.md): Numbers only make sense against a fixed model and a fixed machine. This handbook uses three models and three machines throughout, so a figure in Part 5 can be checked against one… ## AI Infrastructure Engineer Handbook: The model as a workload - [The transformer, seen by the hardware](https://system.tangentbyte.com/learn/transformer-as-workload.md): To a GPU, a transformer is a list of matrix multiplies with some element-wise work in between. Which of those operations takes the time depends on how many tokens go through at… - [The arithmetic of inference](https://system.tangentbyte.com/learn/inference-arithmetic.md): Before touching a profiler, an inference engineer should be able to say roughly how fast a model will run: how long one decode step takes, how many users fit, how long a long… - [Mixture-of-experts models](https://system.tangentbyte.com/learn/mixture-of-experts.md): A mixture-of-experts (MoE) model replaces each layer's MLP with many smaller MLPs, called experts, and a router that sends each token to a few of them. The model stores far more… ## AI Infrastructure Engineer Handbook: GPUs and kernels - [How a GPU executes code](https://system.tangentbyte.com/learn/gpu-execution.md): Part 1 treated a GPU as two numbers, peak FLOPs and memory bandwidth, and that was enough to predict how fast a model runs. Writing or fixing a kernel needs a closer look. This… - [Writing kernels in Triton](https://system.tangentbyte.com/learn/triton-kernels.md): Triton is a language for writing GPU kernels in Python. Instead of describing what each of thousands of threads does, a Triton kernel describes what one program does to a block of… - [CUDA for reading and debugging](https://system.tangentbyte.com/learn/cuda-basics.md): Most inference engineers write more Triton than CUDA, but they read CUDA every week. The fastest kernels in vLLM, SGLang, FlashAttention and TensorRT-LLM are CUDA C++, and when… - [FlashAttention from the ground up](https://system.tangentbyte.com/learn/flash-attention.md): Attention is the one part of a transformer whose cost grows with the square of the sequence. Computed the obvious way, it writes an N × N matrix of scores to GPU memory, reads it… - [Profiling and benchmarking](https://system.tangentbyte.com/learn/profiling.md): Every claim in this part of the handbook rests on a measurement, and a measurement is only as good as its method. This chapter covers both halves of that work. Benchmarking asks… ## AI Infrastructure Engineer Handbook: The runtime layer - [PyTorch execution for inference](https://system.tangentbyte.com/learn/pytorch-execution.md): Chapter 2.5 profiled one decode step of Llama 3.2 1B on an H100 and found the GPU idle for most of it: hundreds of tiny kernels, each launched separately by Python, and a GPU that… - [Native code and Python bindings](https://system.tangentbyte.com/learn/native-code.md): Every major inference engine is a Python program on top of native code. Python holds the parts that change often, such as the scheduler's policy, the API server and the model… - [Compilers and graph runtimes](https://system.tangentbyte.com/learn/graph-runtimes.md): Chapter 3.1 compiled a model just in time: torch.compile traced the decode step inside the serving process, on its first call, and the process kept the result in memory. The other… ## AI Infrastructure Engineer Handbook: Compression - [Number formats](https://system.tangentbyte.com/learn/number-formats.md): Every weight, activation and cache entry a model touches is stored in some number format, and the format decides three things at once: how many bytes the hardware must move, which… - [Quantization math](https://system.tangentbyte.com/learn/quantization-math.md): Chapter 4.1 showed what the small number formats can hold. This chapter covers how a model's weights and activations are mapped onto them: the scale and zero-point that turn… - [Post-training quantization algorithms](https://system.tangentbyte.com/learn/ptq-algorithms.md): Rounding every weight to its nearest grid point, the method chapter 4.2 used, is called round-to-nearest (RTN). It works well at 8 bits and poorly at 4: on SmolLM2-135M, INT4 with… - [Measuring quality loss](https://system.tangentbyte.com/learn/quality-eval.md): A quantized model is only worth serving if it still does its job, and "the perplexity went up by 2%" does not say whether it does. This chapter covers how to measure what… - [Pruning and distillation](https://system.tangentbyte.com/learn/other-compression.md): Quantization keeps every weight and stores it in fewer bits. The two other ways to make a model cheaper remove work instead. Pruning sets weights to zero, either anywhere or in a… ## AI Infrastructure Engineer Handbook: Serving engines - [Build a mini inference engine](https://system.tangentbyte.com/learn/mini-inference-engine.md): Inference serving engines described what an engine does: it batches requests at every step, keeps their KV caches in fixed-size blocks, splits long prompts into chunks, and… - [Inside vLLM](https://system.tangentbyte.com/learn/inside-vllm.md): Chapter 5.1 built an engine in about 650 lines. vLLM is the same loop with three years of production requirements attached: the vllm/v1 folder alone holds about 186,000 lines of… - [SGLang and prefix caching](https://system.tangentbyte.com/learn/sglang-prefix-caching.md): Chapter 5.2 ended with one measurement of prefix caching: 63 requests that shared a 3,000-token prefix finished 12.7 times sooner when vLLM could reuse the prefix's keys and… - [Speculative decoding](https://system.tangentbyte.com/learn/speculative-decoding.md): Every technique in this part so far has made the server do the same work more efficiently. Speculative decoding is different in kind: it makes the server do more work in order to… - [Structured output and sampling](https://system.tangentbyte.com/learn/structured-output.md): The forward pass ends with a vector of logits, one score for every token in the vocabulary, and the engine still has to turn that vector into one token. Two pieces of an engine do… ## C++ for AI Infrastructure Engineers: The ground rules - [The toolchain and the compilation model](https://system.tangentbyte.com/learn/cpp-toolchain.md): Sooner or later every AI infrastructure engineer types cmake --build on someone else's project and gets a wall of red text: llama.cpp with a new backend switched on, a PyTorch… - [C++ syntax for Python programmers](https://system.tangentbyte.com/learn/cpp-syntax.md): You already know what a loop is, what a function is and how an if works, so this chapter does not teach programming again. It teaches the differences, and there are two kinds. The… - [Memory: stack, heap and lifetime](https://system.tangentbyte.com/learn/cpp-memory.md): Here is a number that surprises most Python programmers. A Python list of a million floats, built with something like [float(i) for i in range(1_000_000)], takes about 32 MB on a… - [Pointers, references and const](https://system.tangentbyte.com/learn/cpp-pointers.md): Open any CUDA kernel, any BLAS routine or any PyTorch custom operator, and the first line looks something like this: - [Undefined behavior](https://system.tangentbyte.com/learn/cpp-undefined-behavior.md): Here is a bug that every ML systems engineer meets sooner or later. A custom kernel computes the mean of each row of an activation matrix. It compiles without a warning, it runs… - [Project A: a log analyzer in Python and C++](https://system.tangentbyte.com/learn/cpp-log-analyzer.md): It is the middle of the night, and the inference service you run has an alert: users say chat responses feel slow. The dashboards are down, but every server writes one line per… ## C++ for AI Infrastructure Engineers: Ownership and abstraction - [Classes and RAII](https://system.tangentbyte.com/learn/cpp-raii.md): Imagine a model server that loads LoRA adapters from disk when a request asks for one. The loader opens the adapter file, reads a small header, and if the adapter was trained for… - [Copy and move semantics](https://system.tangentbyte.com/learn/cpp-copy-move.md): Here is a bug that costs nothing to write and a great deal to run. A small C++ inference engine walks through a model's layers for every token it generates: