{"id":"gpus-for-ai","title":"GPUs and accelerators","sections":[{"id":"overview","title":"Overview","html":"<p>The hardware under a model decides how large a model you can serve, how fast one user gets tokens, how many users one machine can carry, and what each token costs. This chapter covers the parts of a GPU or other accelerator that matter for serving, how to read the numbers vendors publish, and how to choose hardware. If prefill and decode are new terms, read <a href=\"/learn/llm-inference\">how LLM inference works</a> first.</p>\n<p>The running example is Llama 3.1 70B, a model with 70 billion parameters stored in BF16 (16-bit floating point, 2 bytes per number). Its shape comes from Table 3 of the Llama 3 paper: 80 layers, a model dimension of 8,192, 64 attention heads, and 8 key/value heads. Each head therefore has dimension 8,192 ÷ 64 = 128 (<a href=\"https://arxiv.org/html/2407.21783\" target=\"_blank\" rel=\"noopener noreferrer\">Llama 3 paper</a>). Product figures are as of October 2026, and each table names its sources.</p>","headings":[]},{"id":"what-a-gpu-is-good-at","title":"What a GPU is good at","html":"<h3 id=\"what-a-gpu-is-good-at\">What a GPU is good at</h3>\n<p>A graphics processing unit (GPU) runs the same arithmetic on many numbers at once. NVIDIA's H100 SXM, the data-center GPU this chapter uses most, has 132 streaming multiprocessors (SMs). Each SM has 128 FP32 CUDA cores and four fourth-generation Tensor Cores, so the chip has 16,896 CUDA cores and 528 Tensor Cores (<a href=\"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/\" target=\"_blank\" rel=\"noopener noreferrer\">Hopper architecture article</a>).</p>\n<p>A Tensor Core performs matrix multiply-accumulate operations, multiplying blocks of two matrices and adding the result to a third. Serving a transformer is mostly this work. Each parameter takes part in one multiplication and one addition per token, or 2 floating-point operations (FLOPs). For the 70B model, one token costs 140 GFLOP.</p>\n<p>A GPU is fast only with enough independent work, such as many tokens or requests, to keep its Tensor Cores busy. Arithmetic is also only half the story, since every multiplication needs weights from memory.</p>","headings":[{"id":"what-a-gpu-is-good-at","title":"What a GPU is good at","depth":3}]},{"id":"gpu-memory","title":"GPU memory","html":"<h3 id=\"gpu-memory\">GPU memory</h3>\n<p>The memory that matters most is high-bandwidth memory (HBM), a stack of DRAM chips beside the GPU die, linked by a very wide bus. The H100 SXM has 80 GB of HBM3 (<a href=\"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/\" target=\"_blank\" rel=\"noopener noreferrer\">Hopper article</a>), read at 3.35 TB/s (<a href=\"https://www.nvidia.com/en-us/data-center/h100/\" target=\"_blank\" rel=\"noopener noreferrer\">H100 page</a>). The H200 keeps the same Tensor Core figures and raises memory to 141 GB of HBM3e at 4.8 TB/s (<a href=\"https://www.nvidia.com/en-us/data-center/h200/\" target=\"_blank\" rel=\"noopener noreferrer\">H200 page</a>).</p>\n<p>On-chip memory is far smaller and faster to reach. The H100 has a 50 MB L2 cache, and each SM has up to 228 KB of shared memory (<a href=\"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/\" target=\"_blank\" rel=\"noopener noreferrer\">Hopper article</a>). Across 132 SMs, shared memory totals about 30 MB, so the chip holds roughly 80 MB on chip, about one thousandth of its HBM. Reusing data on chip avoids trips to HBM. FlashAttention, for example, tiles attention so that it reads and writes HBM less often (<a href=\"https://arxiv.org/abs/2205.14135\" target=\"_blank\" rel=\"noopener noreferrer\">arXiv 2205.14135</a>).</p>\n<p>Host memory, the system DRAM attached to the CPU, is the third level, reached over PCIe. NVIDIA lists the H100's PCIe Gen5 link at 128 GB/s in total, or 64 GB/s each way (<a href=\"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/\" target=\"_blank\" rel=\"noopener noreferrer\">Hopper article</a>). That is about 50 times slower than HBM (3,350 ÷ 64 ≈ 52). Weights or cache that spill into host memory slow every step that reads them.</p>\n<figure class=\"diagram\" data-text=\"GPU die 132 SMs, 50 MB L2 -> HBM3, on package 80 GB (3.35 TB/s); GPU die 132 SMs, 50 MB L2 <-> Host memory in the server (PCIe, 64 GB/s each way)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 480 265\" width=\"480\" role=\"img\" aria-label=\"The H100&#x27;s memory. Bandwidth drops as the pool gets bigger.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"88.7\" x2=\"327\" y2=\"55.7\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"153\" y1=\"129.4\" x2=\"327\" y2=\"194.6\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"76\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"98.5\">GPU die</tspan><tspan x=\"80\" y=\"114.5\">132 SMs, 50 MB L2</tspan></text><path class=\"dg-node dg-store\" d=\"M332,18 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,18 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"41.5\">HBM3, on package</tspan><tspan x=\"400\" y=\"57.5\">80 GB</tspan></text><path class=\"dg-node dg-store\" d=\"M332,198 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,198 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"221.5\">Host memory</tspan><tspan x=\"400\" y=\"237.5\">in the server</tspan></text><rect class=\"dg-edge-bg\" x=\"206.1\" y=\"63.2\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"239\" y=\"76.2\" text-anchor=\"middle\">3.35 TB/s</text><rect class=\"dg-edge-bg\" x=\"166.8\" y=\"153\" width=\"146.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"166\" text-anchor=\"middle\">PCIe, 64 GB/s each way</text></svg></div><figcaption>The H100's memory. Bandwidth drops as the pool gets bigger.</figcaption></figure>\n<p>The table gives the NVIDIA numbers this chapter uses, each with its source.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>GPU</th><th>Launch</th><th>Memory</th><th>Bandwidth</th><th>Dense BF16 TFLOPS (sparse)</th><th>NVLink per GPU</th><th>Sources</th></tr></thead><tbody><tr><td>A100 80 GB SXM</td><td>Announced May 14, 2020</td><td>80 GB HBM2e</td><td>2,039 GB/s</td><td>312 (624)</td><td>600 GB/s</td><td><a href=\"https://www.nvidia.com/en-us/data-center/a100/\">A100 page</a>, <a href=\"https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/\">Ampere article</a></td></tr><tr><td>H100 SXM</td><td>Announced March 2022, available from Q3 2022</td><td>80 GB HBM3</td><td>3.35 TB/s</td><td>989 (1,979)</td><td>900 GB/s</td><td><a href=\"https://www.nvidia.com/en-us/data-center/h100/\">H100 page</a>, <a href=\"https://nvidianews.nvidia.com/news/nvidia-announces-hopper-architecture-the-next-generation-of-accelerated-computing\">2022 announcement</a></td></tr><tr><td>H200 SXM</td><td>Announced Nov 13, 2023, available from Q2 2024</td><td>141 GB HBM3e</td><td>4.8 TB/s</td><td>989 (1,979)</td><td>900 GB/s</td><td><a href=\"https://www.nvidia.com/en-us/data-center/h200/\">H200 page</a>, <a href=\"https://nvidianews.nvidia.com/news/nvidia-supercharges-hopper-the-worlds-leading-ai-computing-platform\">2023 announcement</a></td></tr><tr><td>B200 (per GPU)</td><td>Announced March 18, 2024</td><td>180 GB HBM3e</td><td>8 TB/s</td><td>2,250 (4,500)</td><td>1.8 TB/s</td><td><a href=\"https://www.nvidia.com/en-us/data-center/dgx-b200/\">DGX B200 page</a>, <a href=\"https://www.nvidia.com/en-us/data-center/hgx/\">HGX page</a>, <a href=\"https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing\">2024 announcement</a></td></tr></tbody></table></div>\n<p>The figures in parentheses assume 2:4 sparsity, the label NVIDIA's pages use, so each dense figure is half the sparse one. The H100 page prints only the sparse 1,979 teraFLOPS, and 989 is half of it. The Hopper article's 2022 preliminary table rounded dense BF16 to 1,000, and the H200 page still carries a \"preliminary specifications\" footnote. The B200 pages print system totals, so the per-GPU values divide them by eight: 1,440 GB ÷ 8 = 180 GB, 64 TB/s ÷ 8 = 8 TB/s, and the HGX page's 36 PFLOPS of BF16 with sparsity ÷ 8 = 4.5 PFLOPS per GPU, or 2.25 PFLOPS dense.</p>","headings":[{"id":"gpu-memory","title":"GPU memory","depth":3}]},{"id":"compute-versus-memory-bandwidth","title":"Compute versus memory bandwidth","html":"<h3 id=\"compute-versus-memory-bandwidth\">Compute versus memory bandwidth</h3>\n<p>Two numbers decide how long a step takes: how much arithmetic it does and how many bytes it reads. Their ratio is the <strong>arithmetic intensity</strong>, in FLOPs per byte.</p>\n<p>The <strong>roofline model</strong> caps a workload at the lower of two lines: peak compute, or memory bandwidth times arithmetic intensity. As a formula, attainable performance = min(peak compute, bandwidth × intensity) (<a href=\"https://en.wikipedia.org/wiki/Roofline_model\" target=\"_blank\" rel=\"noopener noreferrer\">Wikipedia's roofline article</a>, which summarizes Williams, Waterman and Patterson's 2009 paper). The point where the lines meet is the <strong>ridge point</strong>. Below it, a workload is <strong>memory-bound</strong>, and bandwidth sets its speed. Above it, the workload is <strong>compute-bound</strong>.</p>\n<p>For the H100 SXM, dense BF16 peak is 989 teraFLOPS and bandwidth is 3.35 TB/s. The ridge point is 989 × 10^12 ÷ 3.35 × 10^12 ≈ 295 FLOPs per byte.</p>\n<p><strong>Decode.</strong> Each decode step reads all 140 GB of weights once and does 140 GFLOP for each sequence in the batch. For a batch of B sequences, the intensity is B × 140 GFLOP ÷ 140 GB = B FLOPs per byte, because the two FLOPs and two bytes per parameter cancel. On the H100:</p>\n<ul>\n<li>At batch 1, the intensity is 1 FLOP per byte. The roofline caps the chip at 3.35 TFLOP/s, which is 0.34% of its 989 TFLOP/s peak.</li>\n<li>Ignoring the KV cache, decode reaches the ridge point at a batch of about 295 sequences.</li>\n</ul>\n<p>The KV cache changes the picture. Each sequence also reads its own cache: 2,000 × 327,680 bytes ≈ 655 MB per sequence at 2,000 tokens. The intensity becomes B × 140 ÷ (140 + 0.655 × B), which is 49 at batch 64 and approaches 140 ÷ 0.655 ≈ 214 as B grows. That is below the ridge point of 295, so one H100 at peak compute stays memory-bound at every batch size for this model at 2,000-token contexts. A kernel that reaches only half of peak has an effective ridge point near 148, so large batches would then be compute-bound.</p>\n<p>This sketch computes those numbers:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Roofline numbers for one H100 SXM, using the dense BF16 peak and HBM bandwidth.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">PEAK_FLOPS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 989e12</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">       # FLOP/s, dense BF16 (the spec page's 1,979 assumes sparsity)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BANDWIDTH</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3.35e12</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">       # bytes/s, HBM3</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">PARAMS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 70e9</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">             # Llama 3.1 70B</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">WEIGHT_BYTES</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">          # BF16</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">KV_PER_TOKEN</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 327_680</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # bytes of KV cache per token (running example)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CONTEXT</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2_000</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">           # tokens per sequence</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> ridge_point</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"FLOPs per byte where a step stops being memory-bound.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PEAK_FLOPS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> BANDWIDTH</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> intensity</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(batch: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, context: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"FLOPs per byte read during one decode step.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    flops </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> batch </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PARAMS</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                  # 2 FLOPs per parameter per token</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    weight_bytes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PARAMS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> WEIGHT_BYTES</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">        # all weights are read once per step</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    kv_bytes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> batch </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> context </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> KV_PER_TOKEN</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   # each sequence reads its own cache</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> flops </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (weight_bytes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> kv_bytes)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> attainable</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(i: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Roofline: the lower of the compute ceiling and bandwidth times intensity.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">PEAK_FLOPS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BANDWIDTH</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"ridge point: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">ridge_point()</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> FLOPs per byte\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">295</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">296</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> intensity(b)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        kind </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"compute\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ridge_point() </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"memory\"</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"batch </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">b</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:3d</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:6.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> FLOPs/byte, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">kind</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">-bound, \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">              f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">attainable(i) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e12</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:6.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> TFLOP/s\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"batch 64, 2,000-token contexts: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">intensity(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">64</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CONTEXT</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"limit as batch grows, 2,000-token contexts: \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">          f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PARAMS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CONTEXT</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> KV_PER_TOKEN</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>ridge point: 295.2 FLOPs per byte</span></span>\n<span class=\"line\"><span>batch   1:    1.0 FLOPs/byte, memory-bound,    3.4 TFLOP/s</span></span>\n<span class=\"line\"><span>batch 295:  295.0 FLOPs/byte, memory-bound,  988.2 TFLOP/s</span></span>\n<span class=\"line\"><span>batch 296:  296.0 FLOPs/byte, compute-bound,  989.0 TFLOP/s</span></span>\n<span class=\"line\"><span>batch 64, 2,000-token contexts: 49.2</span></span>\n<span class=\"line\"><span>limit as batch grows, 2,000-token contexts: 213.6</span></span></code></pre>\n<p><strong>Prefill.</strong> Prefill sends the whole prompt through in one pass, so the weights are read once for all its tokens. A 2,000-token prompt needs 280 TFLOP for that one read, an intensity of 2,000. On four H100s at dense peak (3,956 TFLOPS), the prompt takes about 71 ms, or about 142 ms at half of peak. This is the split <a href=\"/learn/llm-inference#prefill-and-decode\">the inference chapter</a> describes: prefill is limited by arithmetic, and decode by memory.</p>\n<p>Two consequences follow:</p>\n<ul>\n<li><strong>Decode speed follows bandwidth.</strong> The H200's 4.8 TB/s is 1.43 times the H100's 3.35 TB/s, so at the same GPU count the decode ceiling rises by that factor.</li>\n<li><strong>Prefill speed follows dense FLOPS.</strong> The H200 has the same Tensor Core figures as the H100, so its prefill is no faster.</li>\n</ul>","headings":[{"id":"compute-versus-memory-bandwidth","title":"Compute versus memory bandwidth","depth":3}]},{"id":"number-formats-and-quantization","title":"Number formats and quantization","html":"<h3 id=\"number-formats-and-quantization\">Number formats and quantization</h3>\n<p>A number format sets how many bits store each value. The exponent bits set the range, and the mantissa (fraction) bits set the precision.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Format</th><th>Bytes</th><th>Sign / exponent / mantissa bits</th><th>Source</th></tr></thead><tbody><tr><td>FP32</td><td>4</td><td>1 / 8 / 23</td><td>Derived from the BF16 row</td></tr><tr><td>BF16</td><td>2</td><td>1 / 8 / 7</td><td><a href=\"https://docs.cloud.google.com/tpu/docs/bfloat16\">Google, bfloat16</a></td></tr><tr><td>FP16</td><td>2</td><td>1 / 5 / 10</td><td><a href=\"https://en.wikipedia.org/wiki/Half-precision_floating-point_format\">Wikipedia, binary16</a></td></tr><tr><td>FP8 E4M3</td><td>1</td><td>1 / 4 / 3</td><td><a href=\"https://arxiv.org/abs/2209.05433\">FP8 formats paper</a></td></tr><tr><td>FP8 E5M2</td><td>1</td><td>1 / 5 / 2</td><td><a href=\"https://arxiv.org/abs/2209.05433\">FP8 formats paper</a></td></tr><tr><td>INT8</td><td>1</td><td>integer, 256 values</td><td>arithmetic</td></tr><tr><td>INT4</td><td>0.5</td><td>integer, 16 values</td><td>arithmetic</td></tr></tbody></table></div>\n<p>Google's bfloat16 description gives BF16 one sign bit, eight exponent bits and seven mantissa bits, and says it \"has the same exponent size as FP32,\" so FP32 has 8 exponent bits and 23 mantissa bits. Google's blog gives the reason for the trade: \"neural networks are far more sensitive to the size of the exponent than that of the mantissa.\" The next sketch rounds FP32 values to BF16 and prints the largest value each format holds:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Number formats: how BF16 keeps FP32's range, and how much precision it gives up.</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> struct</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> to_bf16</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(x: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Round an FP32 value to BF16 by keeping the top 16 bits (round to nearest even).\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> struct.unpack(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\">I\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, struct.pack(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\">f\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, x))[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> 0x</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">7FFF</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ((bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 16</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)     </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># rounding; NaN and Inf are not handled</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    kept </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 16</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;&#x3C;</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 16</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">               # drop the low 16 mantissa bits</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> struct.unpack(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\">f\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, struct.pack(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\">I\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, kept))[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> max_finite</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(exp_bits: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, man_bits: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Largest finite value of an IEEE-style format with these field widths.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    bias </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> **</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (exp_bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    top </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> **</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> exp_bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> -</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> bias          </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the all-ones exponent is reserved</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> -</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> **</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> -</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">man_bits) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> **</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> top</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3.14159</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1000.7</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">x</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:>9</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> stored as BF16: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">to_bf16(x)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.6g</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"FP16 largest value: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">max_finite(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"BF16 largest value: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">max_finite(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">7</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.4e</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"FP32 largest value: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">max_finite(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">23</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.4e</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>  3.14159 stored as BF16: 3.14062</span></span>\n<span class=\"line\"><span>      0.1 stored as BF16: 0.100098</span></span>\n<span class=\"line\"><span>   1000.7 stored as BF16: 1000</span></span>\n<span class=\"line\"><span>FP16 largest value: 65504.0</span></span>\n<span class=\"line\"><span>BF16 largest value: 3.3895e+38</span></span>\n<span class=\"line\"><span>FP32 largest value: 3.4028e+38</span></span></code></pre>\n<p>BF16 keeps only about two decimal digits, but its largest value, 3.39 × 10^38, is close to FP32's 3.40 × 10^38. FP16 tops out at 65,504.</p>\n<p><strong>Quantization</strong> stores each weight in fewer bits. The 70B weights take 140 GB in BF16, 70 GB in FP8 or INT8, and 35 GB in INT4. Decode reads every weight each step, so halving the bytes halves the time of a memory-bound step. Quantizing also rounds each weight to the nearest level the format can express. The next sketch quantizes 100,000 random weights with one scale, then adds one outlier, 0.9, which widens the grid for every weight:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># What quantization does to weights: fewer levels, and one outlier stretches the grid.</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> random</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> quantize_dequantize</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(values: list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], bits: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Symmetric round-to-nearest quantization with one scale for the whole tensor.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    qmax </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> **</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                    # 127 for INT8, 7 for INT4</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    scale </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">abs</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(v) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> values) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> qmax    </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the largest value sets the grid</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">qmax, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(qmax, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">round</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(v </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> scale))) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> scale </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> values]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> mean_abs_error</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(values: list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], bits: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    restored </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> quantize_dequantize(values, bits)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">abs</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(a </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> b) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> a, b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> zip</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(values, restored)) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(values)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    random.seed(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    weights </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [random.gauss(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.02</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> _ </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">100_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)]   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># typical spread</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    with_outlier </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> weights </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]                                </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># one large value</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"INT</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">bits</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> **</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> symmetric levels): \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">              f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"error </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">mean_abs_error(weights, bits)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.6f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">              f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"with one outlier </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">mean_abs_error(with_outlier, bits)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.6f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>INT8 (255 symmetric levels): error 0.000167, with one outlier 0.001776</span></span>\n<span class=\"line\"><span>INT4 (15 symmetric levels): error 0.003032, with one outlier 0.015937</span></span></code></pre>\n<p>The outlier makes the error about 10 times larger at INT8 and about 5 times larger at INT4. Published results matter more than a toy:</p>\n<ul>\n<li>GPTQ quantized a 175-billion-parameter GPT model to 3 or 4 bits per weight in about four GPU hours, with \"negligible accuracy degradation relative to the uncompressed baseline.\" Its end-to-end speedups over FP16 were about 3.25 times on an A100 and 4.5 times on an A6000 (<a href=\"https://arxiv.org/abs/2210.17323\" target=\"_blank\" rel=\"noopener noreferrer\">arXiv 2210.17323</a>).</li>\n<li>LLM.int8() keeps the few outlier feature dimensions of large models in 16 bits and runs over 99.9% of values in 8 bits. The authors report halved inference memory with no performance degradation up to 175 billion parameters (<a href=\"https://arxiv.org/abs/2208.07339\" target=\"_blank\" rel=\"noopener noreferrer\">arXiv 2208.07339</a>).</li>\n<li>A 2024 study found \"non-negligible degradation\" in low-bit Llama 3 models, especially at ultra-low bit widths (<a href=\"https://arxiv.org/abs/2404.14047\" target=\"_blank\" rel=\"noopener noreferrer\">arXiv 2404.14047</a>). Test on your own tasks.</li>\n</ul>\n<p>Sparsity is a separate speedup. NVIDIA's pages print a sparse peak, double the dense peak, for 2:4 sparse weights, where two of every four values are zero. Keeping accuracy needs pruning and retraining. In NVIDIA's own benchmark of a sparse ResNeXt-101 model on an A100, the gain was up to 20% at large batch sizes, far short of 2 times (<a href=\"https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA's sparsity article</a>), so treat the sparse peak as a ceiling. The A100 has no FP8 row, since FP8 arrived with the H100.</p>","headings":[{"id":"number-formats-and-quantization","title":"Number formats and quantization","depth":3}]},{"id":"connecting-gpus","title":"Connecting GPUs","html":"<h3 id=\"connecting-gpus\">Connecting GPUs</h3>\n<p>Several GPUs need links between them. The table gives each link's speed per GPU.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Link</th><th>Connects</th><th>Speed</th><th>Source</th></tr></thead><tbody><tr><td>PCIe Gen5 (H100)</td><td>GPU to host memory</td><td>128 GB/s total, 64 GB/s each way</td><td>Hopper article</td></tr><tr><td>NVLink, A100</td><td>GPUs in one node</td><td>600 GB/s</td><td><a href=\"https://www.nvidia.com/en-us/data-center/a100/\">A100 page</a></td></tr><tr><td>NVLink, H100 and H200</td><td>GPUs in one node</td><td>900 GB/s total, 450 GB/s each way</td><td><a href=\"https://www.nvidia.com/en-us/data-center/h100/\">H100 page</a>, <a href=\"https://www.nvidia.com/en-us/data-center/h200/\">H200 page</a></td></tr><tr><td>NVLink, B200</td><td>GPUs in one node</td><td>1.8 TB/s</td><td><a href=\"https://www.nvidia.com/en-us/data-center/hgx/\">HGX page</a></td></tr><tr><td>InfiniBand adapter</td><td>Nodes</td><td>up to 400 Gb/s per port (50 GB/s)</td><td><a href=\"https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html\">DGX H100 guide</a></td></tr><tr><td>NVLink domain, GB200 NVL72</td><td>72 GPUs in one rack</td><td>130 TB/s aggregate</td><td><a href=\"https://www.nvidia.com/en-us/data-center/gb200-nvl72/\">GB200 NVL72 page</a></td></tr></tbody></table></div>\n<p>A DGX H100 node has eight H100 GPUs, 640 GB of GPU memory, and 900 GB/s of GPU-to-GPU bandwidth. Each GPU has its own InfiniBand adapter, up to 400 Gb/s or 50 GB/s, so NVLink inside the node is 18 times faster than the network path out (900 ÷ 50).</p>\n<figure class=\"diagram\" data-text=\"Node 1 8 H100 GPUs NVLink 900 GB/s -> InfiniBand fabric 400 Gb/s per GPU (50 GB/s); InfiniBand fabric 400 Gb/s per GPU -> Node 2 8 H100 GPUs NVLink 900 GB/s (50 GB/s)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 480 265\" width=\"480\" role=\"img\" aria-label=\"Inside a node, GPUs use NVLink. Between nodes, they use InfiniBand.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"112.9\" y1=\"79\" x2=\"208\" y2=\"186\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"270.2\" y1=\"188\" x2=\"365.3\" y2=\"81\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"8\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"30.5\">Node 1</tspan><tspan x=\"80\" y=\"46.5\">8 H100 GPUs</tspan><tspan x=\"80\" y=\"62.5\">NVLink 900 GB/s</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"8\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"30.5\">Node 2</tspan><tspan x=\"400\" y=\"46.5\">8 H100 GPUs</tspan><tspan x=\"400\" y=\"62.5\">NVLink 900 GB/s</tspan></text><path class=\"dg-node dg-store\" d=\"M172,198 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M172,198 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"221.5\">InfiniBand fabric</tspan><tspan x=\"240\" y=\"237.5\">400 Gb/s per GPU</tspan></text><rect class=\"dg-edge-bg\" x=\"133.7\" y=\"123.5\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"160.4\" y=\"136.5\" text-anchor=\"middle\">50 GB/s</text><rect class=\"dg-edge-bg\" x=\"291.1\" y=\"125.5\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"317.8\" y=\"138.5\" text-anchor=\"middle\">50 GB/s</text></svg></div><figcaption>Inside a node, GPUs use NVLink. Between nodes, they use InfiniBand.</figcaption></figure>\n<p>Take the 70B model split across four GPUs with tensor parallelism (described below), serving 64 sequences in one step. Assume, as a layout assumption, two combines of activations per layer. Each combine moves 64 tokens × 8,192 values × 2 bytes = 1 MiB. A ring all-reduce moves 2 × (4 − 1) ÷ 4 = 1.5 times that per GPU, about 1.57 MB. Over 80 layers with two combines each, that is about 252 MB per GPU per step. NVLink, at 450 GB/s each way, moves it in about 0.56 ms, roughly 4% of the 13.6 ms step. PCIe, at 64 GB/s each way, takes about 3.9 ms, roughly 29%. These figures ignore latency and overlap, but they show why tensor parallelism belongs inside a node.</p>","headings":[{"id":"connecting-gpus","title":"Connecting GPUs","depth":3}]},{"id":"splitting-a-model-across-gpus","title":"Splitting a model across GPUs","html":"<h3 id=\"splitting-a-model-across-gpus\">Splitting a model across GPUs</h3>\n<p>A model too big for one GPU must be spread over several. Replicas copy the whole model, and the other three split it.</p>\n<p><strong>Replicas</strong> run complete copies, each on its own GPUs, and send each request to one copy. Replicas in different groups never talk while a token is generated, so they add throughput cheaply, as long as the whole model and its cache fit in one copy.</p>\n<p><strong>Tensor parallelism (TP)</strong> splits the matrices inside each layer across GPUs, which Megatron-LM calls an \"intra-layer model parallel approach\" (<a href=\"https://arxiv.org/abs/1909.08053\" target=\"_blank\" rel=\"noopener noreferrer\">arXiv 1909.08053</a>). Each GPU reads a share of every layer, so weight traffic per GPU falls roughly with the GPU count. The cost is one combine per layer, which needs fast links. vLLM recommends it for a model too large for one GPU but small enough for one node, and warns that multi-node tensor parallelism \"requires fast internode communication, preferably through high-speed network adapters such as InfiniBand\" (<a href=\"https://docs.vllm.ai/en/latest/serving/parallelism_scaling/\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM docs</a>).</p>\n<p><strong>Pipeline parallelism (PP)</strong> gives each GPU a run of consecutive layers. GPipe pipelines \"different sub-sequences of layers on separate accelerators\" (<a href=\"https://arxiv.org/abs/1811.06965\" target=\"_blank\" rel=\"noopener noreferrer\">arXiv 1811.06965</a>). Only one activation tensor crosses each stage boundary per token, far less traffic than a combine per layer. The cost is latency: with one sequence, the stages take turns. vLLM suggests it across nodes, and for GPUs without NVLink.</p>\n<p><strong>Expert parallelism (EP)</strong> applies to mixture-of-experts models, which store many expert networks but send each token through only a few. The Switch Transformer paper calls the result \"a sparsely-activated model ... with outrageous numbers of parameters but a constant computational cost\" (<a href=\"https://arxiv.org/abs/2101.03961\" target=\"_blank\" rel=\"noopener noreferrer\">arXiv 2101.03961</a>). Experts can sit on different GPUs, and vLLM's documentation describes this layout for MoE layers.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Strategy</th><th>What it splits</th><th>Traffic between GPUs</th><th>Main cost</th></tr></thead><tbody><tr><td>Replicas</td><td>Nothing, since each copy is whole</td><td>None while generating</td><td>Each copy needs the whole model</td></tr><tr><td>Tensor</td><td>Matrices inside each layer</td><td>One combine per layer</td><td>Needs fast links</td></tr><tr><td>Pipeline</td><td>Consecutive layers</td><td>One activation hop per stage</td><td>Stages take turns for one sequence</td></tr><tr><td>Expert</td><td>Experts in MoE layers</td><td>Tokens sent to their expert</td><td>Only for MoE models</td></tr></tbody></table></div>\n<p>For the running example, four H100s with tensor parallelism hold 35 GB of weights each, leaving 288 − 140 = 148 GB of cache across the group, about 450,000 tokens. Llama 3.1 405B is harder. Its BF16 weights take 405 × 10^9 × 2 = 810 GB, more than one 8-GPU node holds (640 GB). Following vLLM's multi-node rule, use a tensor size of 8 inside each node and a pipeline size of 2 across two nodes. Each of the 16 GPUs then holds 50.6 GB of weights and has about 21 GB left for cache (0.9 × 80 − 50.6 ≈ 21.4).</p>","headings":[{"id":"splitting-a-model-across-gpus","title":"Splitting a model across GPUs","depth":3}]},{"id":"other-accelerators","title":"Other accelerators","html":"<h3 id=\"other-accelerators\">Other accelerators</h3>\n<p>Google's TPUs, AWS's Trainium and Inferentia chips, and AMD's Instinct line are the other families a team meets. Each has its own software stack, and the serving engine has to run on it.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Chip</th><th>Status</th><th>Memory per chip</th><th>Bandwidth per chip</th><th>Compute per chip</th><th>Chip-to-chip link</th><th>Source</th></tr></thead><tbody><tr><td>TPU v6e (Trillium)</td><td>Generally available Dec 16, 2024</td><td>32 GB HBM</td><td>1,638 GB/s</td><td>918 TFLOPS BF16, 1,836 TOPS INT8</td><td>800 GB/s ICI, pods of 256</td><td><a href=\"https://docs.cloud.google.com/tpu/docs/v6e\">TPU v6e docs</a>, <a href=\"https://docs.cloud.google.com/tpu/docs/release-notes\">release notes</a></td></tr><tr><td>TPU7x (Ironwood)</td><td>Generally available Mar 31, 2026</td><td>192 GiB HBM</td><td>7,380 GB/s</td><td>2,307 TFLOPS BF16, 4,614 TFLOPS FP8</td><td>1,200 GB/s ICI, pods of 9,216</td><td><a href=\"https://docs.cloud.google.com/tpu/docs/tpu7x\">TPU7x docs</a>, <a href=\"https://docs.cloud.google.com/tpu/docs/release-notes\">release notes</a></td></tr><tr><td>AWS Trainium2</td><td>Generally available Dec 3, 2024</td><td>96 GB</td><td>about 2.9 TB/s (46 TB/s ÷ 16 chips)</td><td>up to 20.8 PFLOPS FP8 per 16-chip instance</td><td>NeuronLink, not quantified</td><td><a href=\"https://press.aboutamazon.com/2024/12/aws-trainium2-instances-now-generally-available\">AWS press release</a>, <a href=\"https://aws.amazon.com/ec2/instance-types/trn2/\">Trn2 page</a></td></tr><tr><td>AWS Trainium3</td><td>No date on AWS's page</td><td>144 GB HBM3e</td><td>4.9 TB/s</td><td>not given per chip</td><td>NeuronLink, not quantified</td><td><a href=\"https://aws.amazon.com/ai/machine-learning/trainium/\">Trainium page</a></td></tr><tr><td>AWS Inferentia2</td><td>Generally available Apr 13, 2023</td><td>32 GB HBM</td><td>about 0.8 TB/s (9.8 TB/s ÷ 12 chips)</td><td>up to 2.3 PFLOPS per 12-chip instance, precision not stated</td><td>not printed</td><td><a href=\"https://aws.amazon.com/ec2/instance-types/inf2/\">Inf2 page</a>, <a href=\"https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-ec2-inf2-instances-generative-ai-generally-available\">What's New</a></td></tr><tr><td>AMD Instinct MI300X</td><td>Launched Dec 6, 2023</td><td>192 GB HBM3</td><td>5.3 TB/s</td><td>1,307 TFLOPS BF16 (2,615 with sparsity)</td><td>8 Infinity Fabric links, 128 GB/s each</td><td><a href=\"https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html\">AMD MI300X page</a></td></tr></tbody></table></div>\n<p>Google's documentation says TPUs multiply BF16 values by default (<a href=\"https://docs.cloud.google.com/tpu/docs/bfloat16\" target=\"_blank\" rel=\"noopener noreferrer\">TPU bfloat16 docs</a>). AWS's Trn2 instance combines 16 Trainium2 chips over NeuronLink. AMD's MI300X has more memory and bandwidth per chip than an H100, so a 70B model in BF16 fits on one with room for cache (192 − 140 = 52 GB before the reserve). AMD's newer MI355X has 288 GiB of memory, according to AMD's ROCm documentation.</p>","headings":[{"id":"other-accelerators","title":"Other accelerators","depth":3}]},{"id":"choosing-hardware","title":"Choosing hardware","html":"<h3 id=\"choosing-hardware\">Choosing hardware</h3>\n<p>Start from the model and the load, then compare each candidate on four numbers:</p>\n<ol>\n<li><strong>Memory per GPU</strong> sets how many GPUs one replica needs: the weights, plus cache for your users and context, plus about 10% for activations and runtime.</li>\n<li><strong>Bandwidth per GPU</strong> sets the decode ceiling, since a memory-bound step takes bytes read ÷ bandwidth.</li>\n<li><strong>Dense FLOPS at your precision</strong> set prefill speed and the batch size where compute becomes the limit.</li>\n<li><strong>Interconnect</strong> decides how far tensor parallelism can stretch.</li>\n</ol>\n<p>Then price the result. Cost per token is GPUs × price per GPU-hour, divided by tokens per hour. The price below is an assumption: $3.00 per H100-hour. Real prices vary widely by provider and commitment.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Decode step time and cost for the running example: 70B in BF16 on 4 x H100.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">GPUS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 4</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BANDWIDTH</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3.35e12</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                  # bytes/s per H100</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">PRICE_PER_GPU_HOUR</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3.00</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">            # an assumption, not a quote</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">WEIGHTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 140e9</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                      # 70B parameters x 2 bytes</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">KV_64_USERS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 64</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2_000</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 327_680</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   # 64 sequences of 2,000 tokens each</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> step_seconds</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(bytes_read: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Decode is memory-bound, so a step takes bytes read / aggregate bandwidth.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> bytes_read </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">GPUS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> BANDWIDTH</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> dollars_per_million_tokens</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(batch: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, bytes_read: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    tokens_per_second </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> batch </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> step_seconds(bytes_read)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    cost_per_second </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> GPUS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PRICE_PER_GPU_HOUR</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3600</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cost_per_second </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tokens_per_second </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e6</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> batch, bytes_read </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ((</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">WEIGHTS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">64</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">WEIGHTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> KV_64_USERS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        ms </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> step_seconds(bytes_read) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1000</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        tps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> batch </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> step_seconds(bytes_read)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        cost </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> dollars_per_million_tokens(batch, bytes_read)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"batch </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">batch</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:2d</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: step </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">ms</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:5.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> ms, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">tps</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:7.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> tokens/s, \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">              f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"$</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">cost</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> per million tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>batch  1: step 10.45 ms,    95.7 tokens/s, $34.83 per million tokens</span></span>\n<span class=\"line\"><span>batch 64: step 13.58 ms,  4713.6 tokens/s, $0.71 per million tokens</span></span></code></pre>\n<p>One user, at about 96 tokens a second, pays about $35 per million tokens. Sixty-four users in the same step each get about 74 tokens a second, about 4,700 in total, and pay $0.71 per million tokens. The weights are read once per step however many users share them, so the cost per token falls by about 49 times. These are ceilings at full utilization. They ignore prefill, idle time and scheduling overhead, so a real service costs more. At 30% average utilization, $0.71 becomes about $2.37.</p>\n<p>The same reasoning compares generations. The H200 raises the decode ceiling by 1.43 times over the H100 at the same GPU count, with no prefill gain. Its 141 GB lets two GPUs hold the BF16 weights with about 114 GB of cache left (2 × 141 × 0.9 − 140). A two-GPU H200 replica reaches about 69 tokens a second at batch 1, against 96 for the four-GPU H100 replica, or about 34 per GPU against 24. Price per GPU-hour decides which wins.</p>\n<p>As of October 2026, NVIDIA's HGX page lists B200 and B300 as shipping and Vera Rubin as ramping into full production (<a href=\"https://www.nvidia.com/en-us/data-center/hgx/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA HGX</a>). The tables above will age. Check current spec pages before buying, and compare TFLOPS only at the same precision and sparsity.</p>","headings":[{"id":"choosing-hardware","title":"Choosing hardware","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>How fast can one user decode from an 8B model in BF16 on one H100 at batch 1, using the bandwidth ceiling?</summary>\n<p>The weights take 8 × 10^9 × 2 bytes = 16 GB, and each token reads all of them. A step takes at least 16 GB ÷ 3,350 GB/s ≈ 4.8 ms, so the ceiling is about 209 tokens a second. Real speed is lower.</p>\n</details>\n<details>\n<summary>A team wants 70B weights in FP8 on one GPU, with a BF16 cache and a 10% reserve. Compare an H100 (80 GB) and an H200 (141 GB) for cache room.</summary>\n<p>The weights take 70 GB. On an H100, 80 − 70 − 8 = 2 GB remains, about 6,000 tokens, which is effectively no room. On an H200, 141 − 70 − 14.1 ≈ 57 GB remains, about 174,000 tokens, or roughly 87 sequences of 2,000 tokens. Two H100s would leave about 37 GB per GPU for cache.</p>\n</details>\n<details>\n<summary>A service moved from H100 to H200 and expects about 43% faster decoding, but sees 5%. Name two causes to check.</summary>\n<p>The 43% assumes decode is memory-bound and that the extra bandwidth is used. Check how much time goes to prefill, which has the same dense BF16 figure on both chips, and whether the new memory is used. If the cache pool or batch limit still reflects 80 GB, batches stay small and scheduling or host overhead can dominate.</p>\n</details>\n<details>\n<summary>You serve Llama 3.1 405B across two 8-GPU H100 nodes linked by InfiniBand. Choose the tensor and pipeline sizes and explain why.</summary>\n<p>Use a tensor size of 8 inside each node and a pipeline size of 2 across them. Tensor parallelism combines results after every layer, so it belongs on NVLink. Pipeline parallelism sends one activation hop per stage boundary per token, which InfiniBand can carry. The 810 GB of weights over 16 GPUs leave about 21 GB per GPU for cache.</p>\n</details>\n<details>\n<summary>You quantize to INT4. Throughput doubles, but a quality eval drops 3 points. What do you check before shipping?</summary>\n<p>Check per-task results, because an average can hide a drop in one task. Find which layers or outlier features are hurt, since LLM.int8() keeps outliers in 16 bits for this reason. Try INT8 or FP8 first, since the Llama 3 study reports most of its loss at ultra-low bit widths. Then measure on your own eval set.</p>\n</details>\n<details>\n<summary>Four H100s at $3.00 per GPU-hour serve 64 users in one step, about 74 tokens a second each. What is the cost per million tokens at full utilization and at 30%? Why does one user cost so much more?</summary>\n<p>At full utilization it is $0.71 per million tokens, and at 30% about $2.37. One user at about 96 tokens a second pays for the same weight reads for each token: $34.83 per million tokens at the ceiling, or about $116 at 30%. The batch spreads those reads across everyone in it.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://www.nvidia.com/en-us/data-center/h100/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA H100</a>, <a href=\"https://www.nvidia.com/en-us/data-center/h200/\" target=\"_blank\" rel=\"noopener noreferrer\">H200</a> and <a href=\"https://www.nvidia.com/en-us/data-center/a100/\" target=\"_blank\" rel=\"noopener noreferrer\">A100</a> spec pages: the spec tables and sparsity footnotes. The H200 page marks its figures preliminary.</li>\n<li><a href=\"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA Hopper architecture</a> and <a href=\"https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/\" target=\"_blank\" rel=\"noopener noreferrer\">Ampere architecture</a> articles: SM counts, cache sizes, link speeds, the A100 date and the 2:4 sparsity definition.</li>\n<li><a href=\"https://nvidianews.nvidia.com/news/nvidia-announces-hopper-architecture-the-next-generation-of-accelerated-computing\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA's Hopper</a>, <a href=\"https://nvidianews.nvidia.com/news/nvidia-supercharges-hopper-the-worlds-leading-ai-computing-platform\" target=\"_blank\" rel=\"noopener noreferrer\">H200</a> and <a href=\"https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing\" target=\"_blank\" rel=\"noopener noreferrer\">Blackwell</a> announcements: launch dates.</li>\n<li><a href=\"https://www.nvidia.com/en-us/data-center/hgx/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA HGX</a>, <a href=\"https://www.nvidia.com/en-us/data-center/dgx-b200/\" target=\"_blank\" rel=\"noopener noreferrer\">DGX B200</a>, <a href=\"https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html\" target=\"_blank\" rel=\"noopener noreferrer\">DGX H100 guide</a> and <a href=\"https://www.nvidia.com/en-us/data-center/gb200-nvl72/\" target=\"_blank\" rel=\"noopener noreferrer\">GB200 NVL72</a>: node totals, link speeds and the 72-GPU NVLink domain.</li>\n<li><a href=\"https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA's 2:4 sparsity article</a>: the pattern, the retraining need and a measured gain well below 2 times.</li>\n<li><a href=\"https://docs.cloud.google.com/tpu/docs/v6e\" target=\"_blank\" rel=\"noopener noreferrer\">TPU v6e</a>, <a href=\"https://docs.cloud.google.com/tpu/docs/tpu7x\" target=\"_blank\" rel=\"noopener noreferrer\">TPU7x</a> docs, <a href=\"https://docs.cloud.google.com/tpu/docs/release-notes\" target=\"_blank\" rel=\"noopener noreferrer\">release notes</a> and the <a href=\"https://cloud.google.com/blog/products/compute/ironwood-tpus-and-new-axion-based-vms-for-your-ai-workloads\" target=\"_blank\" rel=\"noopener noreferrer\">Ironwood post</a>: per-chip specs and availability dates.</li>\n<li><a href=\"https://docs.cloud.google.com/tpu/docs/bfloat16\" target=\"_blank\" rel=\"noopener noreferrer\">Google's bfloat16 docs</a> and <a href=\"https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus\" target=\"_blank\" rel=\"noopener noreferrer\">bfloat16 post</a>: the BF16 layout and why exponent bits matter more.</li>\n<li><a href=\"https://press.aboutamazon.com/2024/12/aws-trainium2-instances-now-generally-available\" target=\"_blank\" rel=\"noopener noreferrer\">AWS Trainium2 availability</a>, <a href=\"https://aws.amazon.com/ai/machine-learning/trainium/\" target=\"_blank\" rel=\"noopener noreferrer\">Trainium</a>, <a href=\"https://aws.amazon.com/ec2/instance-types/trn2/\" target=\"_blank\" rel=\"noopener noreferrer\">Trn2</a> and <a href=\"https://aws.amazon.com/ec2/instance-types/inf2/\" target=\"_blank\" rel=\"noopener noreferrer\">Inf2</a>: AWS chip specs and dates.</li>\n<li><a href=\"https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html\" target=\"_blank\" rel=\"noopener noreferrer\">AMD Instinct MI300X</a> and <a href=\"https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html\" target=\"_blank\" rel=\"noopener noreferrer\">AMD ROCm GPU architecture specs</a>: AMD's specifications and memory sizes.</li>\n<li><a href=\"https://en.wikipedia.org/wiki/Roofline_model\" target=\"_blank\" rel=\"noopener noreferrer\">Roofline model</a> and <a href=\"https://en.wikipedia.org/wiki/Half-precision_floating-point_format\" target=\"_blank\" rel=\"noopener noreferrer\">half-precision format</a> (Wikipedia): secondary summaries of the roofline formula and the FP16 layout.</li>\n<li><a href=\"https://docs.vllm.ai/en/latest/serving/parallelism_scaling/\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM parallelism and scaling</a>: tensor and pipeline sizes and the multi-node rule.</li>\n<li><a href=\"https://arxiv.org/abs/1909.08053\" target=\"_blank\" rel=\"noopener noreferrer\">Megatron-LM</a>, <a href=\"https://arxiv.org/abs/1811.06965\" target=\"_blank\" rel=\"noopener noreferrer\">GPipe</a> and <a href=\"https://arxiv.org/abs/2101.03961\" target=\"_blank\" rel=\"noopener noreferrer\">Switch Transformer</a>: the splitting strategies in the original papers.</li>\n<li><a href=\"https://arxiv.org/abs/2210.17323\" target=\"_blank\" rel=\"noopener noreferrer\">GPTQ</a>, <a href=\"https://arxiv.org/abs/2208.07339\" target=\"_blank\" rel=\"noopener noreferrer\">LLM.int8()</a>, <a href=\"https://arxiv.org/abs/2404.14047\" target=\"_blank\" rel=\"noopener noreferrer\">a low-bit Llama 3 study</a> and <a href=\"https://arxiv.org/abs/2209.05433\" target=\"_blank\" rel=\"noopener noreferrer\">FP8 formats</a>: quantization results and the FP8 layouts.</li>\n<li><a href=\"https://arxiv.org/abs/2205.14135\" target=\"_blank\" rel=\"noopener noreferrer\">FlashAttention</a>: tiling to cut trips between HBM and on-chip SRAM.</li>\n<li><a href=\"https://arxiv.org/html/2407.21783\" target=\"_blank\" rel=\"noopener noreferrer\">The Llama 3 paper</a>: Table 3, the model shapes behind the running example.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}