System Design Handbook · llms.txt

AI systems

GPUs and accelerators

The hardware under a model decides how large a model you can serve, how fast one user gets tokens, how many users one machine can carry, and what each token costs. This chapter covers the parts of a GPU or other accelerator that matter for serving, how to read the numbers vendors publish, and how to choose hardware. If prefill and decode are new terms, read how LLM inference works first.

The running example is Llama 3.1 70B, a model with 70 billion parameters stored in BF16 (16-bit floating point, 2 bytes per number). Its shape comes from Table 3 of the Llama 3 paper: 80 layers, a model dimension of 8,192, 64 attention heads, and 8 key/value heads. Each head therefore has dimension 8,192 ÷ 64 = 128 (Llama 3 paper). Product figures are as of October 2026, and each table names its sources.

What a GPU is good at

A graphics processing unit (GPU) runs the same arithmetic on many numbers at once. NVIDIA's H100 SXM, the data-center GPU this chapter uses most, has 132 streaming multiprocessors (SMs). Each SM has 128 FP32 CUDA cores and four fourth-generation Tensor Cores, so the chip has 16,896 CUDA cores and 528 Tensor Cores (Hopper architecture article).

A Tensor Core performs matrix multiply-accumulate operations, multiplying blocks of two matrices and adding the result to a third. Serving a transformer is mostly this work. Each parameter takes part in one multiplication and one addition per token, or 2 floating-point operations (FLOPs). For the 70B model, one token costs 140 GFLOP.

A GPU is fast only with enough independent work, such as many tokens or requests, to keep its Tensor Cores busy. Arithmetic is also only half the story, since every multiplication needs weights from memory.

GPU memory

The memory that matters most is high-bandwidth memory (HBM), a stack of DRAM chips beside the GPU die, linked by a very wide bus. The H100 SXM has 80 GB of HBM3 (Hopper article), read at 3.35 TB/s (H100 page). The H200 keeps the same Tensor Core figures and raises memory to 141 GB of HBM3e at 4.8 TB/s (H200 page).

On-chip memory is far smaller and faster to reach. The H100 has a 50 MB L2 cache, and each SM has up to 228 KB of shared memory (Hopper article). Across 132 SMs, shared memory totals about 30 MB, so the chip holds roughly 80 MB on chip, about one thousandth of its HBM. Reusing data on chip avoids trips to HBM. FlashAttention, for example, tiles attention so that it reads and writes HBM less often (arXiv 2205.14135).

Host memory, the system DRAM attached to the CPU, is the third level, reached over PCIe. NVIDIA lists the H100's PCIe Gen5 link at 128 GB/s in total, or 64 GB/s each way (Hopper article). That is about 50 times slower than HBM (3,350 ÷ 64 ≈ 52). Weights or cache that spill into host memory slow every step that reads them.

GPU die132 SMs, 50 MB L2HBM3, on package80 GBHost memoryin the server3.35 TB/sPCIe, 64 GB/s each way
The H100's memory. Bandwidth drops as the pool gets bigger.

The table gives the NVIDIA numbers this chapter uses, each with its source.

GPULaunchMemoryBandwidthDense BF16 TFLOPS (sparse)NVLink per GPUSources
A100 80 GB SXMAnnounced May 14, 202080 GB HBM2e2,039 GB/s312 (624)600 GB/sA100 page, Ampere article
H100 SXMAnnounced March 2022, available from Q3 202280 GB HBM33.35 TB/s989 (1,979)900 GB/sH100 page, 2022 announcement
H200 SXMAnnounced Nov 13, 2023, available from Q2 2024141 GB HBM3e4.8 TB/s989 (1,979)900 GB/sH200 page, 2023 announcement
B200 (per GPU)Announced March 18, 2024180 GB HBM3e8 TB/s2,250 (4,500)1.8 TB/sDGX B200 page, HGX page, 2024 announcement

The figures in parentheses assume 2:4 sparsity, the label NVIDIA's pages use, so each dense figure is half the sparse one. The H100 page prints only the sparse 1,979 teraFLOPS, and 989 is half of it. The Hopper article's 2022 preliminary table rounded dense BF16 to 1,000, and the H200 page still carries a "preliminary specifications" footnote. The B200 pages print system totals, so the per-GPU values divide them by eight: 1,440 GB ÷ 8 = 180 GB, 64 TB/s ÷ 8 = 8 TB/s, and the HGX page's 36 PFLOPS of BF16 with sparsity ÷ 8 = 4.5 PFLOPS per GPU, or 2.25 PFLOPS dense.

Compute versus memory bandwidth

Two numbers decide how long a step takes: how much arithmetic it does and how many bytes it reads. Their ratio is the arithmetic intensity, in FLOPs per byte.

The roofline model caps a workload at the lower of two lines: peak compute, or memory bandwidth times arithmetic intensity. As a formula, attainable performance = min(peak compute, bandwidth × intensity) (Wikipedia's roofline article, which summarizes Williams, Waterman and Patterson's 2009 paper). The point where the lines meet is the ridge point. Below it, a workload is memory-bound, and bandwidth sets its speed. Above it, the workload is compute-bound.

For the H100 SXM, dense BF16 peak is 989 teraFLOPS and bandwidth is 3.35 TB/s. The ridge point is 989 × 10^12 ÷ 3.35 × 10^12 ≈ 295 FLOPs per byte.

Decode. Each decode step reads all 140 GB of weights once and does 140 GFLOP for each sequence in the batch. For a batch of B sequences, the intensity is B × 140 GFLOP ÷ 140 GB = B FLOPs per byte, because the two FLOPs and two bytes per parameter cancel. On the H100:

  • At batch 1, the intensity is 1 FLOP per byte. The roofline caps the chip at 3.35 TFLOP/s, which is 0.34% of its 989 TFLOP/s peak.
  • Ignoring the KV cache, decode reaches the ridge point at a batch of about 295 sequences.

The KV cache changes the picture. Each sequence also reads its own cache: 2,000 × 327,680 bytes ≈ 655 MB per sequence at 2,000 tokens. The intensity becomes B × 140 ÷ (140 + 0.655 × B), which is 49 at batch 64 and approaches 140 ÷ 0.655 ≈ 214 as B grows. That is below the ridge point of 295, so one H100 at peak compute stays memory-bound at every batch size for this model at 2,000-token contexts. A kernel that reaches only half of peak has an effective ridge point near 148, so large batches would then be compute-bound.

This sketch computes those numbers:

# Roofline numbers for one H100 SXM, using the dense BF16 peak and HBM bandwidth.
PEAK_FLOPS = 989e12       # FLOP/s, dense BF16 (the spec page's 1,979 assumes sparsity)
BANDWIDTH = 3.35e12       # bytes/s, HBM3
PARAMS = 70e9             # Llama 3.1 70B
WEIGHT_BYTES = 2          # BF16
KV_PER_TOKEN = 327_680    # bytes of KV cache per token (running example)
CONTEXT = 2_000           # tokens per sequence


def ridge_point() -> float:
    """FLOPs per byte where a step stops being memory-bound."""
    return PEAK_FLOPS / BANDWIDTH


def intensity(batch: int, context: int = 0) -> float:
    """FLOPs per byte read during one decode step."""
    flops = batch * 2 * PARAMS                  # 2 FLOPs per parameter per token
    weight_bytes = PARAMS * WEIGHT_BYTES        # all weights are read once per step
    kv_bytes = batch * context * KV_PER_TOKEN   # each sequence reads its own cache
    return flops / (weight_bytes + kv_bytes)


def attainable(i: float) -> float:
    """Roofline: the lower of the compute ceiling and bandwidth times intensity."""
    return min(PEAK_FLOPS, BANDWIDTH * i)


if __name__ == "__main__":
    print(f"ridge point: {ridge_point():.1f} FLOPs per byte")
    for b in (1, 295, 296):
        i = intensity(b)
        kind = "compute" if i >= ridge_point() else "memory"
        print(f"batch {b:3d}: {i:6.1f} FLOPs/byte, {kind}-bound, "
              f"{attainable(i) / 1e12:6.1f} TFLOP/s")
    print(f"batch 64, 2,000-token contexts: {intensity(64, CONTEXT):.1f}")
    print(f"limit as batch grows, 2,000-token contexts: "
          f"{2 * PARAMS / (CONTEXT * KV_PER_TOKEN):.1f}")
ridge point: 295.2 FLOPs per byte
batch   1:    1.0 FLOPs/byte, memory-bound,    3.4 TFLOP/s
batch 295:  295.0 FLOPs/byte, memory-bound,  988.2 TFLOP/s
batch 296:  296.0 FLOPs/byte, compute-bound,  989.0 TFLOP/s
batch 64, 2,000-token contexts: 49.2
limit as batch grows, 2,000-token contexts: 213.6

Prefill. Prefill sends the whole prompt through in one pass, so the weights are read once for all its tokens. A 2,000-token prompt needs 280 TFLOP for that one read, an intensity of 2,000. On four H100s at dense peak (3,956 TFLOPS), the prompt takes about 71 ms, or about 142 ms at half of peak. This is the split the inference chapter describes: prefill is limited by arithmetic, and decode by memory.

Two consequences follow:

  • Decode speed follows bandwidth. The H200's 4.8 TB/s is 1.43 times the H100's 3.35 TB/s, so at the same GPU count the decode ceiling rises by that factor.
  • Prefill speed follows dense FLOPS. The H200 has the same Tensor Core figures as the H100, so its prefill is no faster.

Number formats and quantization

A number format sets how many bits store each value. The exponent bits set the range, and the mantissa (fraction) bits set the precision.

FormatBytesSign / exponent / mantissa bitsSource
FP3241 / 8 / 23Derived from the BF16 row
BF1621 / 8 / 7Google, bfloat16
FP1621 / 5 / 10Wikipedia, binary16
FP8 E4M311 / 4 / 3FP8 formats paper
FP8 E5M211 / 5 / 2FP8 formats paper
INT81integer, 256 valuesarithmetic
INT40.5integer, 16 valuesarithmetic

Google's bfloat16 description gives BF16 one sign bit, eight exponent bits and seven mantissa bits, and says it "has the same exponent size as FP32," so FP32 has 8 exponent bits and 23 mantissa bits. Google's blog gives the reason for the trade: "neural networks are far more sensitive to the size of the exponent than that of the mantissa." The next sketch rounds FP32 values to BF16 and prints the largest value each format holds:

# Number formats: how BF16 keeps FP32's range, and how much precision it gives up.
import struct


def to_bf16(x: float) -> float:
    """Round an FP32 value to BF16 by keeping the top 16 bits (round to nearest even)."""
    bits = struct.unpack(">I", struct.pack(">f", x))[0]
    bits += 0x7FFF + ((bits >> 16) & 1)     # rounding; NaN and Inf are not handled
    kept = (bits >> 16) << 16               # drop the low 16 mantissa bits
    return struct.unpack(">f", struct.pack(">I", kept))[0]


def max_finite(exp_bits: int, man_bits: int) -> float:
    """Largest finite value of an IEEE-style format with these field widths."""
    bias = 2 ** (exp_bits - 1) - 1
    top = 2 ** exp_bits - 2 - bias          # the all-ones exponent is reserved
    return (2 - 2.0 ** -man_bits) * 2.0 ** top


if __name__ == "__main__":
    for x in (3.14159, 0.1, 1000.7):
        print(f"{x:>9} stored as BF16: {to_bf16(x):.6g}")
    print(f"FP16 largest value: {max_finite(5, 10):.1f}")
    print(f"BF16 largest value: {max_finite(8, 7):.4e}")
    print(f"FP32 largest value: {max_finite(8, 23):.4e}")
  3.14159 stored as BF16: 3.14062
      0.1 stored as BF16: 0.100098
   1000.7 stored as BF16: 1000
FP16 largest value: 65504.0
BF16 largest value: 3.3895e+38
FP32 largest value: 3.4028e+38

BF16 keeps only about two decimal digits, but its largest value, 3.39 × 10^38, is close to FP32's 3.40 × 10^38. FP16 tops out at 65,504.

Quantization stores each weight in fewer bits. The 70B weights take 140 GB in BF16, 70 GB in FP8 or INT8, and 35 GB in INT4. Decode reads every weight each step, so halving the bytes halves the time of a memory-bound step. Quantizing also rounds each weight to the nearest level the format can express. The next sketch quantizes 100,000 random weights with one scale, then adds one outlier, 0.9, which widens the grid for every weight:

# What quantization does to weights: fewer levels, and one outlier stretches the grid.
import random


def quantize_dequantize(values: list[float], bits: int) -> list[float]:
    """Symmetric round-to-nearest quantization with one scale for the whole tensor."""
    qmax = 2 ** (bits - 1) - 1                    # 127 for INT8, 7 for INT4
    scale = max(abs(v) for v in values) / qmax    # the largest value sets the grid
    return [max(-qmax, min(qmax, round(v / scale))) * scale for v in values]


def mean_abs_error(values: list[float], bits: int) -> float:
    restored = quantize_dequantize(values, bits)
    return sum(abs(a - b) for a, b in zip(values, restored)) / len(values)


if __name__ == "__main__":
    random.seed(0)
    weights = [random.gauss(0.0, 0.02) for _ in range(100_000)]   # typical spread
    with_outlier = weights + [0.9]                                # one large value
    for bits in (8, 4):
        print(f"INT{bits} ({2 ** bits - 1} symmetric levels): "
              f"error {mean_abs_error(weights, bits):.6f}, "
              f"with one outlier {mean_abs_error(with_outlier, bits):.6f}")
INT8 (255 symmetric levels): error 0.000167, with one outlier 0.001776
INT4 (15 symmetric levels): error 0.003032, with one outlier 0.015937

The outlier makes the error about 10 times larger at INT8 and about 5 times larger at INT4. Published results matter more than a toy:

  • GPTQ quantized a 175-billion-parameter GPT model to 3 or 4 bits per weight in about four GPU hours, with "negligible accuracy degradation relative to the uncompressed baseline." Its end-to-end speedups over FP16 were about 3.25 times on an A100 and 4.5 times on an A6000 (arXiv 2210.17323).
  • LLM.int8() keeps the few outlier feature dimensions of large models in 16 bits and runs over 99.9% of values in 8 bits. The authors report halved inference memory with no performance degradation up to 175 billion parameters (arXiv 2208.07339).
  • A 2024 study found "non-negligible degradation" in low-bit Llama 3 models, especially at ultra-low bit widths (arXiv 2404.14047). Test on your own tasks.

Sparsity is a separate speedup. NVIDIA's pages print a sparse peak, double the dense peak, for 2:4 sparse weights, where two of every four values are zero. Keeping accuracy needs pruning and retraining. In NVIDIA's own benchmark of a sparse ResNeXt-101 model on an A100, the gain was up to 20% at large batch sizes, far short of 2 times (NVIDIA's sparsity article), so treat the sparse peak as a ceiling. The A100 has no FP8 row, since FP8 arrived with the H100.

Connecting GPUs

Several GPUs need links between them. The table gives each link's speed per GPU.

LinkConnectsSpeedSource
PCIe Gen5 (H100)GPU to host memory128 GB/s total, 64 GB/s each wayHopper article
NVLink, A100GPUs in one node600 GB/sA100 page
NVLink, H100 and H200GPUs in one node900 GB/s total, 450 GB/s each wayH100 page, H200 page
NVLink, B200GPUs in one node1.8 TB/sHGX page
InfiniBand adapterNodesup to 400 Gb/s per port (50 GB/s)DGX H100 guide
NVLink domain, GB200 NVL7272 GPUs in one rack130 TB/s aggregateGB200 NVL72 page

A DGX H100 node has eight H100 GPUs, 640 GB of GPU memory, and 900 GB/s of GPU-to-GPU bandwidth. Each GPU has its own InfiniBand adapter, up to 400 Gb/s or 50 GB/s, so NVLink inside the node is 18 times faster than the network path out (900 ÷ 50).

Node 18 H100 GPUsNVLink 900 GB/sNode 28 H100 GPUsNVLink 900 GB/sInfiniBand fabric400 Gb/s per GPU50 GB/s50 GB/s
Inside a node, GPUs use NVLink. Between nodes, they use InfiniBand.

Take the 70B model split across four GPUs with tensor parallelism (described below), serving 64 sequences in one step. Assume, as a layout assumption, two combines of activations per layer. Each combine moves 64 tokens × 8,192 values × 2 bytes = 1 MiB. A ring all-reduce moves 2 × (4 − 1) ÷ 4 = 1.5 times that per GPU, about 1.57 MB. Over 80 layers with two combines each, that is about 252 MB per GPU per step. NVLink, at 450 GB/s each way, moves it in about 0.56 ms, roughly 4% of the 13.6 ms step. PCIe, at 64 GB/s each way, takes about 3.9 ms, roughly 29%. These figures ignore latency and overlap, but they show why tensor parallelism belongs inside a node.

Splitting a model across GPUs

A model too big for one GPU must be spread over several. Replicas copy the whole model, and the other three split it.

Replicas run complete copies, each on its own GPUs, and send each request to one copy. Replicas in different groups never talk while a token is generated, so they add throughput cheaply, as long as the whole model and its cache fit in one copy.

Tensor parallelism (TP) splits the matrices inside each layer across GPUs, which Megatron-LM calls an "intra-layer model parallel approach" (arXiv 1909.08053). Each GPU reads a share of every layer, so weight traffic per GPU falls roughly with the GPU count. The cost is one combine per layer, which needs fast links. vLLM recommends it for a model too large for one GPU but small enough for one node, and warns that multi-node tensor parallelism "requires fast internode communication, preferably through high-speed network adapters such as InfiniBand" (vLLM docs).

Pipeline parallelism (PP) gives each GPU a run of consecutive layers. GPipe pipelines "different sub-sequences of layers on separate accelerators" (arXiv 1811.06965). Only one activation tensor crosses each stage boundary per token, far less traffic than a combine per layer. The cost is latency: with one sequence, the stages take turns. vLLM suggests it across nodes, and for GPUs without NVLink.

Expert parallelism (EP) applies to mixture-of-experts models, which store many expert networks but send each token through only a few. The Switch Transformer paper calls the result "a sparsely-activated model ... with outrageous numbers of parameters but a constant computational cost" (arXiv 2101.03961). Experts can sit on different GPUs, and vLLM's documentation describes this layout for MoE layers.

StrategyWhat it splitsTraffic between GPUsMain cost
ReplicasNothing, since each copy is wholeNone while generatingEach copy needs the whole model
TensorMatrices inside each layerOne combine per layerNeeds fast links
PipelineConsecutive layersOne activation hop per stageStages take turns for one sequence
ExpertExperts in MoE layersTokens sent to their expertOnly for MoE models

For the running example, four H100s with tensor parallelism hold 35 GB of weights each, leaving 288 − 140 = 148 GB of cache across the group, about 450,000 tokens. Llama 3.1 405B is harder. Its BF16 weights take 405 × 10^9 × 2 = 810 GB, more than one 8-GPU node holds (640 GB). Following vLLM's multi-node rule, use a tensor size of 8 inside each node and a pipeline size of 2 across two nodes. Each of the 16 GPUs then holds 50.6 GB of weights and has about 21 GB left for cache (0.9 × 80 − 50.6 ≈ 21.4).

Other accelerators

Google's TPUs, AWS's Trainium and Inferentia chips, and AMD's Instinct line are the other families a team meets. Each has its own software stack, and the serving engine has to run on it.

ChipStatusMemory per chipBandwidth per chipCompute per chipChip-to-chip linkSource
TPU v6e (Trillium)Generally available Dec 16, 202432 GB HBM1,638 GB/s918 TFLOPS BF16, 1,836 TOPS INT8800 GB/s ICI, pods of 256TPU v6e docs, release notes
TPU7x (Ironwood)Generally available Mar 31, 2026192 GiB HBM7,380 GB/s2,307 TFLOPS BF16, 4,614 TFLOPS FP81,200 GB/s ICI, pods of 9,216TPU7x docs, release notes
AWS Trainium2Generally available Dec 3, 202496 GBabout 2.9 TB/s (46 TB/s ÷ 16 chips)up to 20.8 PFLOPS FP8 per 16-chip instanceNeuronLink, not quantifiedAWS press release, Trn2 page
AWS Trainium3No date on AWS's page144 GB HBM3e4.9 TB/snot given per chipNeuronLink, not quantifiedTrainium page
AWS Inferentia2Generally available Apr 13, 202332 GB HBMabout 0.8 TB/s (9.8 TB/s ÷ 12 chips)up to 2.3 PFLOPS per 12-chip instance, precision not statednot printedInf2 page, What's New
AMD Instinct MI300XLaunched Dec 6, 2023192 GB HBM35.3 TB/s1,307 TFLOPS BF16 (2,615 with sparsity)8 Infinity Fabric links, 128 GB/s eachAMD MI300X page

Google's documentation says TPUs multiply BF16 values by default (TPU bfloat16 docs). AWS's Trn2 instance combines 16 Trainium2 chips over NeuronLink. AMD's MI300X has more memory and bandwidth per chip than an H100, so a 70B model in BF16 fits on one with room for cache (192 − 140 = 52 GB before the reserve). AMD's newer MI355X has 288 GiB of memory, according to AMD's ROCm documentation.

Choosing hardware

Start from the model and the load, then compare each candidate on four numbers:

  1. Memory per GPU sets how many GPUs one replica needs: the weights, plus cache for your users and context, plus about 10% for activations and runtime.
  2. Bandwidth per GPU sets the decode ceiling, since a memory-bound step takes bytes read ÷ bandwidth.
  3. Dense FLOPS at your precision set prefill speed and the batch size where compute becomes the limit.
  4. Interconnect decides how far tensor parallelism can stretch.

Then price the result. Cost per token is GPUs × price per GPU-hour, divided by tokens per hour. The price below is an assumption: $3.00 per H100-hour. Real prices vary widely by provider and commitment.

# Decode step time and cost for the running example: 70B in BF16 on 4 x H100.
GPUS = 4
BANDWIDTH = 3.35e12                  # bytes/s per H100
PRICE_PER_GPU_HOUR = 3.00            # an assumption, not a quote
WEIGHTS = 140e9                      # 70B parameters x 2 bytes
KV_64_USERS = 64 * 2_000 * 327_680   # 64 sequences of 2,000 tokens each


def step_seconds(bytes_read: float) -> float:
    """Decode is memory-bound, so a step takes bytes read / aggregate bandwidth."""
    return bytes_read / (GPUS * BANDWIDTH)


def dollars_per_million_tokens(batch: int, bytes_read: float) -> float:
    tokens_per_second = batch / step_seconds(bytes_read)
    cost_per_second = GPUS * PRICE_PER_GPU_HOUR / 3600
    return cost_per_second / tokens_per_second * 1e6


if __name__ == "__main__":
    for batch, bytes_read in ((1, WEIGHTS), (64, WEIGHTS + KV_64_USERS)):
        ms = step_seconds(bytes_read) * 1000
        tps = batch / step_seconds(bytes_read)
        cost = dollars_per_million_tokens(batch, bytes_read)
        print(f"batch {batch:2d}: step {ms:5.2f} ms, {tps:7.1f} tokens/s, "
              f"${cost:.2f} per million tokens")
batch  1: step 10.45 ms,    95.7 tokens/s, $34.83 per million tokens
batch 64: step 13.58 ms,  4713.6 tokens/s, $0.71 per million tokens

One user, at about 96 tokens a second, pays about $35 per million tokens. Sixty-four users in the same step each get about 74 tokens a second, about 4,700 in total, and pay $0.71 per million tokens. The weights are read once per step however many users share them, so the cost per token falls by about 49 times. These are ceilings at full utilization. They ignore prefill, idle time and scheduling overhead, so a real service costs more. At 30% average utilization, $0.71 becomes about $2.37.

The same reasoning compares generations. The H200 raises the decode ceiling by 1.43 times over the H100 at the same GPU count, with no prefill gain. Its 141 GB lets two GPUs hold the BF16 weights with about 114 GB of cache left (2 × 141 × 0.9 − 140). A two-GPU H200 replica reaches about 69 tokens a second at batch 1, against 96 for the four-GPU H100 replica, or about 34 per GPU against 24. Price per GPU-hour decides which wins.

As of October 2026, NVIDIA's HGX page lists B200 and B300 as shipping and Vera Rubin as ramping into full production (NVIDIA HGX). The tables above will age. Check current spec pages before buying, and compare TFLOPS only at the same precision and sparsity.

Check yourself

How fast can one user decode from an 8B model in BF16 on one H100 at batch 1, using the bandwidth ceiling?

The weights take 8 × 10^9 × 2 bytes = 16 GB, and each token reads all of them. A step takes at least 16 GB ÷ 3,350 GB/s ≈ 4.8 ms, so the ceiling is about 209 tokens a second. Real speed is lower.

A team wants 70B weights in FP8 on one GPU, with a BF16 cache and a 10% reserve. Compare an H100 (80 GB) and an H200 (141 GB) for cache room.

The weights take 70 GB. On an H100, 80 − 70 − 8 = 2 GB remains, about 6,000 tokens, which is effectively no room. On an H200, 141 − 70 − 14.1 ≈ 57 GB remains, about 174,000 tokens, or roughly 87 sequences of 2,000 tokens. Two H100s would leave about 37 GB per GPU for cache.

A service moved from H100 to H200 and expects about 43% faster decoding, but sees 5%. Name two causes to check.

The 43% assumes decode is memory-bound and that the extra bandwidth is used. Check how much time goes to prefill, which has the same dense BF16 figure on both chips, and whether the new memory is used. If the cache pool or batch limit still reflects 80 GB, batches stay small and scheduling or host overhead can dominate.

You serve Llama 3.1 405B across two 8-GPU H100 nodes linked by InfiniBand. Choose the tensor and pipeline sizes and explain why.

Use a tensor size of 8 inside each node and a pipeline size of 2 across them. Tensor parallelism combines results after every layer, so it belongs on NVLink. Pipeline parallelism sends one activation hop per stage boundary per token, which InfiniBand can carry. The 810 GB of weights over 16 GPUs leave about 21 GB per GPU for cache.

You quantize to INT4. Throughput doubles, but a quality eval drops 3 points. What do you check before shipping?

Check per-task results, because an average can hide a drop in one task. Find which layers or outlier features are hurt, since LLM.int8() keeps outliers in 16 bits for this reason. Try INT8 or FP8 first, since the Llama 3 study reports most of its loss at ultra-low bit widths. Then measure on your own eval set.

Four H100s at $3.00 per GPU-hour serve 64 users in one step, about 74 tokens a second each. What is the cost per million tokens at full utilization and at 30%? Why does one user cost so much more?

At full utilization it is $0.71 per million tokens, and at 30% about $2.37. One user at about 96 tokens a second pays for the same weight reads for each token: $34.83 per million tokens at the ceiling, or about $116 at 30%. The batch spreads those reads across everyone in it.

Further reading

Previous: How LLM inference works · Next: Inference serving engines