System Design Handbook · llms.txt
Part 1 treated a GPU as two numbers, peak FLOPs and memory bandwidth, and that was enough to predict how fast a model runs. Writing or fixing a kernel needs a closer look. This chapter explains how a GPU runs thousands of threads at once, why it groups them into warps, how its memory is layered from registers to HBM, and why the order in which threads read memory can change a kernel's speed by ten times. Every measurement in the chapter was taken with the lab's scripts on a rented H100 or A10, or on an M2 MacBook, and every specification is from NVIDIA's documentation.
Before this chapter, read GPUs and accelerators for what a GPU is for, and the step-time model for how the roofline predicts an operation's time.
A kernel is a function that runs on the GPU. When a program launches one, it asks for a grid of thread blocks, and every thread in every block runs the same function on different data. A kernel that adds two vectors of a million numbers might use a grid of 4,096 blocks of 256 threads, one thread per number.
The hardware runs blocks on streaming multiprocessors (SMs). The H100 in this chapter's lab reported 132 of them. The GPU assigns each block to one SM, where the block stays until it finishes, so threads in a block can share fast on-chip memory and wait for each other. An SM can hold several blocks at once, and a grid usually has many more blocks than the GPU has SMs, so blocks queue for a free SM.
Inside an SM, each block is split into warps of 32 threads. The warp is the unit the hardware schedules: one instruction is issued for all 32 threads of a warp together, each applying it to its own data. NVIDIA calls this single-instruction, multiple-thread (SIMT) execution. When the threads of a warp take different branches of an if, the warp runs both paths one after the other, with the threads of the other path switched off, so branches that split warps cost time.
A read from HBM takes hundreds of clock cycles to come back. A CPU core hides that delay with large caches and by reordering instructions. A GPU hides it with numbers instead: each SM keeps many warps resident at once and, every cycle, issues an instruction from a warp that is ready. While one warp waits for memory, others compute. The CUDA programming guide explains that each warp's execution context, its program counter and registers, stays on the chip for the warp's whole life, so switching between warps costs nothing.
That only works if enough warps are resident. Occupancy is the number of active warps on an SM divided by the maximum the SM supports, as NVIDIA's CUDA Best Practices Guide defines it. Four resources limit how many blocks fit on one SM at a time:
| Resource per SM | H100 (compute capability 9.0) | L4 (compute capability 8.9) |
|---|---|---|
| Warp slots | 64 (2,048 threads) | 48 (1,536 threads) |
| Block slots | 32 | 24 |
| 32-bit registers | 65,536 | 65,536 |
| Shared memory | 228 KB, at most 227 KB per block | 100 KB, at most 99 KB per block |
The figures come from NVIDIA's Hopper and Ada tuning guides, and the lab's probe printed the same register and shared memory sizes on a real H100 and a real L4. Registers are usually the limit that bites. A block of 256 threads that uses 64 registers per thread needs 16,384 registers, so at most four such blocks fit on an SM, which is 32 of the H100's 64 warp slots and 50% occupancy.
High occupancy is not a goal in itself. The Best Practices Guide states that higher occupancy does not always mean higher performance. Memory-bound kernels such as element-wise operations need many warps in flight to keep the memory system busy. Matmul kernels often run best at low occupancy, because each warp keeps many independent loads and tensor-core operations in flight, and the registers that limit occupancy are what hold its tile of the output.
The widget below computes occupancy for one kernel configuration. Try these settings:
A GPU's memory comes in layers. Each layer down is larger and slower, and a fast kernel keeps the data it reuses as high in the stack as it can.
| Layer | H100 SXM | L4 | Who controls it |
|---|---|---|---|
| Registers | 64K × 4 bytes = 256 KB per SM | 256 KB per SM | The compiler, per thread |
| L1 cache and shared memory | 256 KB per SM combined, up to 228 KB as shared memory | 128 KB per SM combined, up to 100 KB as shared memory | Shared memory is managed by the kernel; L1 by the hardware |
| L2 cache | 50 MB for the whole GPU | 48 MB for the whole GPU | The hardware |
| HBM or GDDR | 80 GB of HBM3 at 3.35 TB/s | 24 GB of GDDR6 at 300 GB/s | The program, through allocations |
The H100 rows come from the Hopper tuning guide and NVIDIA's H100 specifications, and the L4 rows from the Ada tuning guide, NVIDIA's L4 specifications and the Ada architecture whitepaper, whose L4 table lists 58 SMs and 49,152 KB of L2. The probe confirmed both L2 sizes on real hardware: 52,428,800 bytes on the H100, exactly 50 MiB, and 50,331,648 bytes on the L4, exactly 48 MiB.
Shared memory is the layer that kernel writers manage by hand. It is fast scratch space on the SM, visible to all threads of one block. A matmul kernel loads a tile of each input from HBM into shared memory once and then lets every thread in the block reuse it many times. FlashAttention, covered in chapter 2.4, is built around exactly that idea.
The lab measured how copy bandwidth changes as the working set grows past the L2 cache on an H100:
== 2. Copy bandwidth by working set (read + write counted)
2 MiB working set: 479 GB/s (fits in L2)
8 MiB working set: 1901 GB/s (fits in L2)
16 MiB working set: 3657 GB/s (fits in L2)
32 MiB working set: 3736 GB/s (fits in L2)
64 MiB working set: 2642 GB/s (beyond L2)
128 MiB working set: 2823 GB/s (beyond L2)
512 MiB working set: 2992 GB/s (beyond L2)
2048 MiB working set: 3021 GB/s (beyond L2)
Three things stand out. The 16 and 32 MiB copies, which fit in L2, run faster than any copy from HBM, at about 3.7 TB/s. Copies far larger than L2 settle at 3.0 TB/s, 90% of the 3.35 TB/s on the spec sheet, which is the realistic ceiling for a memory-bound kernel on this GPU. The smallest copies are slow for a different reason: a 2 MiB copy at L2 speed finishes in about half a microsecond, which is shorter than the fixed cost of launching a kernel, so launch overhead dominates. The measurements show it: the 2 MiB and 8 MiB copies both take about 4.4 µs each, although one moves four times as much data. Part 3 returns to that overhead, because a decode step at batch 1 is made of many small kernels.
When the 32 threads of a warp each load a value, the hardware does not fetch 32 separate values. On every GPU of compute capability 6.0 or newer, it combines the warp's loads into transactions of 32-byte sectors and fetches each sector it needs once. The CUDA Best Practices Guide calls this coalescing, and it gives the rules that matter:
The widget below counts sectors for one warp. Try the patterns in order:
The lab measured the same effect on real hardware. It copies 16 million float32 values to a contiguous output, reading them with a growing stride, and reports the useful data moved per second on an H100:
== 3. Reading 16M useful float32 values with different access patterns
stride 1 ( 4 bytes apart): 2717 GB/s of useful data
stride 2 ( 8 bytes apart): 1817 GB/s of useful data
stride 4 ( 16 bytes apart): 1195 GB/s of useful data
stride 8 ( 32 bytes apart): 676 GB/s of useful data
stride 16 ( 64 bytes apart): 358 GB/s of useful data
stride 32 (128 bytes apart): 240 GB/s of useful data
random gather: 199 GB/s of useful data (index reads not counted)
The useful bandwidth falls with each doubling of the stride, as the sector rule predicts, but by less than half per step, because the output side of the copy stays contiguous at every stride. At stride 2 each element costs 8 bytes of read sectors plus 4 bytes of write, against 4 plus 4 at stride 1, which predicts two thirds of the bandwidth; the measurement is 1,817 against 2,717 GB/s. Random reads deliver 199 GB/s, 7% of the contiguous rate, so a kernel that gathers scattered values gives up more than 90% of the GPU's bandwidth. The fall continues beyond a stride of 8, where each value already has its own sector, which suggests the memory system fetches in larger units than one sector. CUDA exposes such a unit as a device limit, cudaLimitMaxL2FetchGranularity, that can be set between 0 and 128 bytes.
Coalescing explains several design choices in inference engines. Weights are stored so that the threads of a warp read consecutive elements of a row. Paged KV caches keep each block of keys contiguous, so that attention kernels read whole sectors even though the blocks themselves are scattered. When a kernel has to read a matrix by columns, it first loads a tile into shared memory with coalesced reads and then reads the columns from there.
Shared memory has its own access rule. It is divided into 32 banks, each 4 bytes wide, and consecutive 4-byte words sit in consecutive banks. All 32 threads of a warp can read in one pass if each reads a different bank. If several threads read different words in the same bank, the CUDA Best Practices Guide explains that the accesses are serialised: a two-way conflict takes two passes, and a 32-way conflict takes 32. Threads that read the same word do not conflict, because the value is broadcast to all of them.
The classic case is a 32 × 32 tile of float32 values stored row by row. Reading a row is conflict-free. Reading a column puts every element in the same bank, because consecutive rows are 32 words apart, so the warp needs 32 passes. The standard fix is to pad each row to 33 words. Elements of a column are then 33 words apart and fall in 32 different banks.
Switch the widget above to "Shared memory" to see this. "Stride 32" shows the column read with its 32-way conflict, and "Stride 33" shows the padded version with none.
Most of an H100's arithmetic comes from tensor cores, units that multiply small matrices in one instruction rather than one number at a time. The ordinary CUDA cores still do element-wise work and FP32 arithmetic, but the figures on the spec sheet that matter for inference are tensor-core figures. The lab ran an 8,192 × 8,192 × 8,192 matmul in each number format on an H100:
== 4. Matmul throughput, 8192 x 8192 x 8192
FP32 (no tensor cores): 52.0 TFLOP/s
TF32 (tensor cores): 417.2 TFLOP/s
BF16 (tensor cores): 803.4 TFLOP/s
FP8 E4M3 (tensor cores): 1410.8 TFLOP/s
| Format | Measured | NVIDIA's dense peak | Share of peak |
|---|---|---|---|
| FP32 without tensor cores | 52.0 TFLOP/s | 67 | 78% |
| TF32 on tensor cores | 417 | 495 | 84% |
| BF16 on tensor cores | 803 | 989 | 81% |
| FP8 on tensor cores | 1,411 | 1,979 | 71% |
NVIDIA's H100 specification page lists the tensor-core figures with sparsity, which doubles them, so the dense peaks in the table are half of the printed numbers. Moving from FP32 to TF32 on tensor cores raises the peak more than sevenfold, each further step from TF32 to BF16 to FP8 doubles it, and a large matmul in PyTorch reaches about 70% to 85% of it. Part 1's roofline estimates used the full peak, so real compute-bound times are about a quarter longer.
The share of peak also depends on the size of the problem. The lab's matmul_sizes.py ran square matmuls of growing size on the same H100:
NVIDIA H100 80GB HBM3, 132 SMs, PyTorch 2.15.0+cu132
N BF16 TFLOP/s FP8 TFLOP/s
1024 104.7 96.8
2048 496.2 577.9
4096 786.6 1330.1
8192 801.4 1454.7
16384 752.1 1418.7
A 1,024 × 1,024 matmul reaches about a tenth of the BF16 peak and a twentieth of the FP8 peak, because it has too few output tiles to give all 132 SMs enough work, and the fixed cost of a launch is large next to its runtime. From 4,096 up, BF16 holds at 76% to 81% of its peak and FP8 at 67% to 74%. Making the problem larger does not close the remaining gap, so it comes from the kernel and the hardware rather than from a lack of work. This matters for inference: a decode step multiplies weights by only a few rows of activations, so its matmuls are far smaller in one dimension than these squares, which is one more reason decode reaches a small share of the peak.
Interviewers expect you to recognise the newer hardware features by name and say what each one is for. These are the ones that the fastest inference kernels rely on:
Tier: [Mac] for part A, [1 GPU] for part B. Part A runs on your laptop in under a minute. Part B needs an NVIDIA GPU for a few seconds of work, which costs less than a cent on Modal.
The scripts are ready to download:
Part A, access patterns on your Mac. Measure copy bandwidth for working sets from 2 MiB to 1 GiB, then the useful bandwidth of reading every element, every second, fourth and so on up to every 32nd, and random elements. The part is done when you can say at what working-set size your machine's caches stop helping, and how much a stride of 8 costs compared with a stride of 1.
Part B, an NVIDIA GPU against its spec sheet. Print the device's SM count, registers, shared memory and L2 size. Measure copy bandwidth across working sets that fit in L2 and that do not, useful bandwidth for strided and random reads, and matmul throughput in FP32, TF32, BF16 and FP8. Put each measurement next to the figure on NVIDIA's specification page. The part is done when you have a table of measured against specified numbers and a sentence explaining each gap. Run it with modal run modal_run.py --script gpu_probe.py --gpu 'H100!', or with python gpu_probe.py on any machine that has an NVIDIA GPU.
Part A. This is the output of access_patterns.py on an M2 MacBook with an 8-core GPU and 8 GB of memory, using PyTorch 2.15's MPS backend:
device: mps, PyTorch 2.15.0
1. Copy bandwidth by working set (bytes read + bytes written)
2 MiB: 251.8 GB/s
8 MiB: 87.7 GB/s
32 MiB: 84.8 GB/s
128 MiB: 90.1 GB/s
512 MiB: 88.5 GB/s
1024 MiB: 86.8 GB/s
2. Reading 8M useful float32 values with different access patterns
stride 1 ( 4 bytes apart): 70.2 GB/s of useful data
stride 2 ( 8 bytes apart): 50.1 GB/s of useful data
stride 4 ( 16 bytes apart): 30.8 GB/s of useful data
stride 8 ( 32 bytes apart): 17.9 GB/s of useful data
stride 16 ( 64 bytes apart): 10.5 GB/s of useful data
stride 32 (128 bytes apart): 10.5 GB/s of useful data
random gather: 2.4 GB/s of useful data (index reads not counted)
Four things in this output are worth noticing:
Part B. This is the output of gpu_probe.py on a Modal H100 SXM, with PyTorch 2.15 and CUDA 13.2. It used 4 seconds of GPU time, about half a cent:
== 1. NVIDIA H100 80GB HBM3, compute capability 9.0, PyTorch 2.15.0+cu132, CUDA 13.2
SMs 132
warp size 32
max threads per SM 2,048
registers per SM 65,536
shared memory per SM (bytes) 233,472
shared memory per block, opt-in (bytes) 232,448
L2 cache (bytes) 52,428,800
global memory (bytes) 85,009,367,040
Sections 2, 3 and 4 of the output are printed in full earlier in this chapter. The shared memory figures are 228 KB and 227 KB, matching the Hopper tuning guide, and the L2 is exactly 50 MiB. The table of measured against specified numbers comes out as follows:
| Quantity | Measured | Specified | Share |
|---|---|---|---|
| HBM bandwidth (2 GiB copy) | 3.02 TB/s | 3.35 TB/s | 90% |
| FP32 matmul | 52.0 TFLOP/s | 67 | 78% |
| TF32 matmul | 417 | 495 | 84% |
| BF16 matmul | 803 | 989 | 81% |
| FP8 matmul | 1,411 | 1,979 | 71% |
No real kernel reaches a spec sheet's peak, and these shares are typical for a well-tuned library. The size sweep in the chapter shows that a larger problem does not raise them: FP8 levels off at about 70% of its peak and BF16 at about 80%. The BF16 rate even dips at 16,384. Chapter 2.5 measured the reason: under a long run of large matmuls, the H100 reached its 700 W power limit and lowered its SM clock from 1,980 MHz to about 1,400 MHz.
The same script on a Modal A10, an older Ampere GPU with 24 GB of GDDR6, gives a useful contrast for about a fifth of a cent. It reported 72 SMs and an L2 cache of only 6 MiB, so even the 8 MiB working set no longer fits in L2. Large copies ran at 485 GB/s, 81% of the 600 GB/s NVIDIA specifies, and BF16 matmuls at 82 TFLOP/s, 66% of the 125 TFLOP/s dense peak. The A10 has no FP8 tensor cores, and the script says so instead of measuring them. The strided reads fell from 478 GB/s at stride 1 to 113 GB/s at stride 8, about a sixth of the H100's rates, and random reads to 43 GB/s. Beyond stride 8 the A10's curve almost flattens, at 104 and 89 GB/s, as the 32-byte sector rule predicts, while the H100's keeps falling. That suggests the H100's HBM is fetched in larger units than the A10's GDDR6.
The L4, the cheap GPU this handbook planned around, was unavailable on Modal when this part's labs were first run, which is why the A10 appears instead. When one became available, the same script took 9 seconds on it, under a cent. It confirmed the L4 figures in the tables above, which came from NVIDIA's documents: 58 SMs, 1,536 threads per SM, 100 KB of shared memory per SM with 99 KB per block, and an L2 of 50,331,648 bytes, exactly 48 MiB. That L2 is eight times the A10's, so the 16 and 32 MiB working sets copied at about 1.9 TB/s, while those beyond it fell to 231 GB/s, 77% of the 300 GB/s NVIDIA specifies. BF16 matmuls ran at 59 TFLOP/s and FP8 at 112 TFLOP/s, under half of the dense peaks of 121 and 242 TFLOP/s. The L4 is a 72 W card, so it most likely cannot hold its peak clock through a long run of dense matmuls, the limit chapter 2.5 measured on the H100; the probe does not record the clock, so that explanation is untested here. Strided reads fell from 232 GB/s at stride 1 to 57 GB/s at stride 8, about a quarter, as on the other two GPUs.
One block needs 256 × 64 = 16,384 registers, and an SM has 65,536, so four blocks fit. Four blocks of 8 warps are 32 warps, half of the 64 warp slots, so the occupancy is 50%. Warp slots, block slots and shared memory would all allow more, so registers are the limit.
The values span bytes 12 to 139, which touch sectors 0 to 4, so five sectors or 160 bytes are fetched for 128 useful bytes. The useful share is 80%. This is the sequential but misaligned access of the CUDA Best Practices Guide, which it says requests five 32-byte segments.
The H100 has 50 MB of L2 cache. A 32 MiB working set stays in L2 after the first copy, so later copies are served from the cache rather than HBM. A 2 GiB working set cannot fit, so every copy reads and writes HBM, which delivers about 90% of its specified 3.35 TB/s.
Shared memory has 32 banks of 4-byte words, and consecutive rows of the tile are 32 words apart, so every element of a column falls in the same bank and the reads are serialised. Padding each row to 33 words puts consecutive column elements 33 words apart, which places them in 32 different banks, so the warp reads the column in one pass.
No. NVIDIA's figure includes sparsity, which doubles the dense rate, so the dense BF16 peak is about 989 TFLOP/s. The measured 803 is 81% of that, which is normal for a large matmul in a well-tuned library. Estimates should use the dense peak and assume a real kernel reaches about 70% to 85% of it.
Previous: Mixture-of-experts models · Next: Writing kernels in Triton