System Design Handbook · llms.txt
A mixture-of-experts (MoE) model replaces each layer's MLP with many smaller MLPs, called experts, and a router that sends each token to a few of them. The model stores far more parameters than any one token uses, so it can know more while computing as much as a much smaller dense model. That split between parameters stored and parameters used changes almost every serving number from the previous chapter. This chapter explains how routing works, works out what it does to memory, bytes per decode step and batch size, and measures a real router on your own machine.
Before this chapter, read counting parameters for how Qwen3-30B-A3B's 30.5B stored and 3.0B active parameters are counted, and a decode step, part by part for the step-time model this chapter extends.
In an MoE transformer, attention is unchanged. Only the MLP after it is different: instead of one gate, up and down projection, the layer holds E experts with those three projections each, plus a router. For each token, the layer does four things:
Models differ in how they compute the weights, but the common forms give the same result. Qwen3's code takes a softmax over all experts, keeps the top k and divides them by their sum, which its config calls norm_topk_prob. Granite's code takes the top k raw scores first and applies a softmax to those k. Both produce the same weights, because dividing softmax values by their sum cancels every term that involves the experts left out.
The idea is older than the transformer era of language models. Shazeer and others introduced the sparsely gated MoE layer in 2017, GShard (2020) used top-2 routing to scale translation models across thousands of accelerator cores, and the Switch Transformer (2021) showed that sending each token to a single expert works as well and costs less. Mixtral 8x7B (2024) was one of the first widely used open MoE language models, and DeepSeek's and Qwen's models since then use many small experts.
An MoE model has two parameter counts. The total is everything that must be stored, and the active count is what one token multiplies through. The model's name usually gives both, as in "30B-A3B" for 30B total and 3B active.
| Model | Experts per layer | Chosen per token | Shared experts | Total | Active per token |
|---|---|---|---|---|---|
| Mixtral 8x7B | 8 | 2 | 0 | 47B | 13B |
| Granite 3.1 1B-A400M | 32 | 8 | 0 | 1.3B | 400M |
| Qwen3-30B-A3B | 128 | 8 | 0 | 30.5B | 3.3B |
| DeepSeek-V3 | 256 | 8 | 1 | 671B | 37B |
The figures come from the Mixtral paper, the Granite and Qwen3 model cards, and the DeepSeek-V3 paper.
Two design choices separate the newer models from Mixtral, and both come from the DeepSeekMoE paper (2024):
For serving, the two counts set different resources. Memory follows the total. Qwen3-30B-A3B needs 61.1 GB for its BF16 weights, four times what Llama 3.1 8B needs. Compute follows the active count. Each token costs about 6.1 GFLOP, against 15.0 for the 8B model. The bytes read per decode step follow neither count exactly, and the next section shows why.
A dense model reads every weight once per decode step, whatever the batch size, which is why batching made decode cheaper in chapter 1.2. An MoE model reads its attention weights, routers and output head on every step, but it reads an expert only if at least one token in the batch chose it.
If the router spread its choices evenly and independently, one token would skip a given expert with probability 1 − k/E. All B tokens in a batch skip it with probability (1 − k/E) to the power B, so the expected number of experts a layer must read is:
experts read per layer = E × (1 − (1 − k/E)^B)
For Qwen3-30B-A3B, with E = 128 and k = 8, one token reads 8 experts, 4 tokens read about 29, 16 tokens read about 82, and 64 tokens read nearly all 128. The script moe_arithmetic.py puts this formula into the step-time model from chapter 1.2 for one H100, with 2,000-token contexts in BF16:
batch | experts/layer weights read step tok/s | 8B: weights step tok/s
1 | 8.0 6.1 GB 1.87 ms 533 | 15.0 GB 4.56 ms 219
2 | 15.5 9.5 GB 2.95 ms 679 | 15.0 GB 4.63 ms 432
4 | 29.1 15.7 GB 4.91 ms 815 | 15.0 GB 4.79 ms 835
8 | 51.6 25.8 GB 8.18 ms 978 | 15.0 GB 5.10 ms 1568
16 | 82.4 39.8 GB 12.82 ms 1248 | 15.0 GB 5.73 ms 2792
32 | 111.8 53.1 GB 17.73 ms 1805 | 15.0 GB 6.98 ms 4583
64 | 125.9 59.5 GB 21.52 ms 2974 | 15.0 GB 9.49 ms 6747
128 | 128.0 60.4 GB 25.55 ms 5010 | 15.0 GB 14.49 ms 8831
256 | 128.0 60.4 GB 33.07 ms 7742 | 15.0 GB 24.51 ms 10445
512 | 128.0 60.4 GB 48.09 ms 10646 | 15.0 GB 47.83 ms 10704
The table has three lessons.
The DeepSeek-V3 paper describes the same effect at a much larger scale. It reports that during decoding each expert usually sees a batch of at most 256 tokens and that the bottleneck is memory access, not computation. Its decode deployment therefore spreads the experts over 320 GPUs, with one expert on most of them, so that a very large total batch gives each expert enough tokens. That technique is called expert parallelism, and Part 6 of this handbook covers it.
Memory makes the problem worse on one GPU. After its 61.1 GB of BF16 weights, Qwen3-30B-A3B leaves 10.9 GB of an H100's usable 72 GB for the KV cache, which holds 55 sequences of 2,000 tokens. The 8B model leaves room for 213. With FP8 weights, which Qwen publishes as Qwen3-30B-A3B-FP8, the weights take 30.5 GB and about 210 sequences fit. This is why MoE models are almost always served in FP8 or across several GPUs.
The roofline explorer from chapter 1.2 now includes Qwen3-30B-A3B, labelled 30B-A3B. Try these settings:
The formula above assumes an even router, and real routers are not even. The lab script expert_load.py runs IBM's Granite 3.1 1B-A400M, a small MoE model with 32 experts and top-8 routing, over 4,096 tokens of English prose and 4,096 tokens of Python code, and records every routing decision. On an M2 MacBook it found three things.
The load is very uneven. In the average layer, the busiest expert received 3.0 times the even share of tokens. With top-8 routing over 32 experts, the largest possible ratio is 4, which means every single token chose that expert, and the most uneven layer reached exactly 4.00. In that layer one expert behaves like a shared expert that the router selects every time.
Experts specialise only a little by content. The routing for prose and for code differed by 14% on average, measured as the share of routing decisions that would have to move to turn one distribution into the other. The same few experts were the busiest for both texts in most layers.
Fewer experts are read than the even formula predicts. Because the router favours some experts, a batch of 4 tokens read 18.4 experts per layer instead of 21.9, and a batch of 16 read 27.0 instead of 31.7. Uneven routing therefore saves some bytes per step, but it costs time in another way.
Imbalance matters most when experts sit on different GPUs. A layer finishes only when its busiest GPU finishes, so a GPU that holds two popular experts slows the whole step. Models are trained to limit this, and engines correct what remains:
--enable-eplb.Use the widget below to see the effect of routing on one layer. Set the router to "Like a trained router", which copies the unevenness measured on Granite above, and compare it with "Even". The calibration comes from Granite's 32 experts. With 128 or 256 experts, the same unevenness makes the busiest expert stand out much further, and nobody has measured those larger routers in this handbook yet, so treat their busiest-expert numbers as an illustration rather than a measurement.
Training systems for MoE models often need fixed tensor shapes, so they give every expert a fixed capacity. The Switch Transformer sets each expert's capacity to the batch's tokens divided by the number of experts, multiplied by a capacity factor. A token that arrives at a full expert is dropped from that expert, and its vector passes to the next layer through the residual connection unchanged. The Switch paper reports that its auxiliary loss kept dropped tokens typically below 1%.
A router that was not trained against a capacity limit drops far more. The lab measured what Switch-style capacity would do to Granite's routing in batches of 256 tokens:
| Capacity factor | Slots per expert | Token-expert pairs dropped |
|---|---|---|
| 1.0 | 64 | 33.2% |
| 1.25 | 80 | 23.3% |
| 1.5 | 96 | 16.0% |
| 2.0 | 128 | 7.4% |
Inference systems avoid dropping because it changes the model's output. DeepSeek-V3 drops no tokens in training or in inference, and vLLM's MoE kernels compute every routed token. In serving, an uneven load therefore costs time rather than quality: the busiest expert takes longer, and everyone waits for it.
The simplest implementation loops over the experts, gathers the tokens that chose each one, runs its MLP and adds the results back. Recent versions of Hugging Face transformers keep this loop as their plain reference implementation, next to a faster one built on PyTorch's grouped matrix multiply. The loop launches separate small matmuls for every expert that has tokens, so with 48 layers and up to 128 experts each, one decode step can launch thousands of small kernels.
Serving engines group the work instead. They sort the (token, expert) pairs by expert, so that each expert's tokens sit in one contiguous block, and then run every expert's matmul in one grouped matmul kernel. They finish by putting the rows back in token order and summing each token's k results. vLLM's moe_align_block_size function does the sorting, padding each expert's block to the kernel's tile size, and its Triton kernel fused_moe_kernel runs the grouped matmul. Part A of the lab below writes both versions in plain PyTorch and checks that they agree.
When experts are spread over GPUs, two communication steps surround that kernel. A dispatch step sends each token's vector to the GPUs that hold its chosen experts, and a combine step brings the results back. Both are all-to-all exchanges, and DeepEP, which vLLM can use as a backend, provides kernels tuned for each phase. Part 6 covers expert parallelism in detail.
Tier: [Mac] for parts A and B, [1 GPU] for part C. Parts A and B run on an 8 GB M2 MacBook in about a minute each. Part C needs a rented 80 GB GPU, which costs well under a dollar on Modal.
The scripts are ready to download, and they also sit in the site's repository at public/labs/infra/:
Try parts A and B yourself before you open the scripts or the reference solution.
Part A, an MoE layer from scratch. Build a tiny Qwen3MoeForCausalLM from transformers with 16 experts and top-4 routing, and take its first layer's MoE block. Write the same layer twice in plain PyTorch: once as a loop over experts, and once in the grouped form that sorts tokens by expert. Both must match the Hugging Face block to float32 rounding on 512 random tokens. Then print how many tokens each expert received, and the FLOPs per token against a dense MLP that runs every expert.
Part B, a real router. Run Granite 3.1 1B-A400M over a few thousand tokens of prose and of code with output_router_logits=True, and record the top-8 experts that every token chose in every layer. From those choices, report four results: the busiest expert's load against the even share per layer, how differently prose and code are routed, the number of experts a decode batch of B tokens touches compared with the even formula, and how many tokens a capacity factor of 1.0 to 2.0 would drop.
Part C, measured against predicted (GPU). Serve Qwen3-30B-A3B and Llama 3.1 8B with vLLM on one H100, measure the decode step time at batch 1 to 16, and put the results next to moe_arithmetic.py's predictions. The GPU folder's README lists what to record and send back.
The lab is done when parts A and B run, when you can explain each number they print, and when your notes say at which batch size Qwen3-30B-A3B stops being faster than the 8B model on one H100, and why.
Part A. The grouped version is the heart of the exercise. It flattens the router's choices into T × k (token, expert) pairs, sorts them by expert with a stable argsort, runs each expert on its contiguous slice, scales each row by its routing weight, and uses index_add_ to sum each token's k rows back into place:
def moe_grouped(x):
weights, experts = route(x)
flat = experts.reshape(-1) # T*K (token, expert) pairs
order = flat.argsort(stable=True) # group the pairs by expert
token = order // K # pair index -> token index
counts = torch.bincount(flat, minlength=E).tolist() # tokens per expert: the group sizes
rows = x[token] # the "permute" step: [T*K, D]
outs, start = [], 0
for e, n in enumerate(counts): # one grouped GEMM in a real engine
outs.append(expert_mlp(e, rows[start:start + n]))
start += n
y = torch.cat(outs) * weights.reshape(-1)[order, None] # scale each row by its routing weight
return torch.zeros_like(x).index_add_(0, token, y), counts # "unpermute" and sum the K rows per token
The full script, with the router, the loop version and the checks, is moe_layer.py. This is its output with PyTorch 2.15 and transformers 5.22 on an M2 CPU:
loop vs HF: max error 3.7e-09 (outputs reach 3.0e-02)
grouped vs HF: max error 3.7e-09
512 tokens x top-4 = 2048 expert slots over 16 experts, 128 per expert if even
tokens per expert: [134, 117, 123, 119, 127, 136, 135, 119, 128, 126, 123, 143, 119, 154, 142, 103]
busiest expert: 154 tokens = 1.20 x the mean; quietest: 103
per token: 0.79 MFLOP active, against 3.15 MFLOP if every expert ran (25%); weights stored: 6.3 MB in FP32
Part B. The key step collects the router's logits from every layer and takes the top k of each. Top-k of the logits gives the same experts as top-k of the probabilities, because softmax keeps the order:
out = model(ids[None, start:start + CHUNK].to(device), output_router_logits=True)
picks.append(torch.stack([r.float().topk(K, dim=-1).indices.cpu() for r in out.router_logits]))
The full script is expert_load.py. It uses English prose from Python's own documentation and Python source code from the standard library, so it needs no dataset download. This is its output on an M2 MacBook with 8 GB of memory, which took 51 seconds including loading the model:
ibm-granite/granite-3.1-1b-a400m-instruct: 24 layers, 32 experts, top-8, 4,096 tokens of each text on mps
1. Load imbalance: the busiest expert's tokens divided by the even share (1.00 = perfectly even)
layer prose code busiest experts, prose | code
0 2.24 2.04 [31, 13, 3] | [31, 13, 9]
3 3.31 3.09 [2, 13, 30] | [2, 13, 26]
6 3.07 2.96 [21, 12, 0] | [21, 13, 0]
9 2.99 3.35 [19, 30, 17] | [30, 17, 19]
12 2.35 2.32 [7, 6, 19] | [6, 13, 7]
15 2.51 2.78 [10, 8, 22] | [10, 22, 18]
18 1.98 2.40 [8, 17, 31] | [9, 13, 22]
21 3.46 3.70 [3, 19, 2] | [19, 13, 29]
prose: mean over layers 3.01, worst layer 4.00; experts never chosen: 0 of 768
code: mean over layers 3.11, worst layer 4.00; experts never chosen: 1 of 768
2. Specialisation: share of routing that differs between prose and code (total variation distance)
mean over layers 0.14, lowest 0.04 (layer 1), highest 0.22 (layer 18)
3. Experts touched per layer by a decode batch of B tokens from different places in the text
B measured even routing
1 8.0 8.0
2 12.8 14.0
4 18.4 21.9
8 23.4 28.8
16 27.0 31.7
32 29.0 32.0
64 30.3 32.0
4. Tokens dropped when each expert takes at most capacity = factor x (tokens x K / E) per 256-token batch
factor 1.00: capacity 64 slots per expert, 33.18% of token-expert pairs dropped
factor 1.25: capacity 80 slots per expert, 23.32% of token-expert pairs dropped
factor 1.50: capacity 96 slots per expert, 15.96% of token-expert pairs dropped
factor 2.00: capacity 128 slots per expert, 7.39% of token-expert pairs dropped
Four things in the output are worth noticing:
Part C. The script gpu/moe_decode_sweep.py loads each model in vLLM once and isolates decode the same way as chapter 1.2's lab: for each batch it subtracts a run that generates 1 token from a run that generates 129, and divides by the 128 decode steps. It also prints a few of vLLM's own start-up lines. Run on Modal with modal run modal_run.py --script moe_decode_sweep.py --gpu 'H100!' --vllm, its output was:
[Qwen3-30B-A3B] Loading weights took 12.61 seconds
[Qwen3-30B-A3B] [backbone] torch.compile took 33.95 s in total
[Qwen3-30B-A3B] Available KV cache memory: 12.18 GiB
[Qwen3-30B-A3B] GPU KV cache size: 133,072 tokens, Maximum concurrency for 2,048 tokens per request: 64.98x
[Meta-Llama-3.1-8B-Instruct] Loading weights took 2.09 seconds
[Meta-Llama-3.1-8B-Instruct] [backbone] torch.compile took 13.26 s in total
[Meta-Llama-3.1-8B-Instruct] Available KV cache memory: 54.44 GiB
[Meta-Llama-3.1-8B-Instruct] GPU KV cache size: 445,968 tokens, Maximum concurrency for 2,048 tokens per request: 217.76x
NVIDIA H100 80GB HBM3, vLLM 0.32.0, PyTorch 2.13.0+cu130; prediction at 3.35 TB/s
BF16, 1000-token prompts, 128 decode steps timed (median of 8 runs)
model batch measured predicted ratio tok/s
Qwen/Qwen3-30B-A3B 1 4.58 ms 1.85 ms 2.48 218
Qwen/Qwen3-30B-A3B 2 5.51 ms 2.89 ms 1.90 363
Qwen/Qwen3-30B-A3B 4 7.81 ms 4.80 ms 1.63 512
Qwen/Qwen3-30B-A3B 8 9.36 ms 7.96 ms 1.18 855
Qwen/Qwen3-30B-A3B 16 12.24 ms 12.38 ms 0.99 1308
unsloth/Meta-Llama-3.1-8B-Instruct 1 6.30 ms 4.52 ms 1.39 159
unsloth/Meta-Llama-3.1-8B-Instruct 2 6.32 ms 4.56 ms 1.39 317
unsloth/Meta-Llama-3.1-8B-Instruct 4 6.47 ms 4.64 ms 1.39 618
unsloth/Meta-Llama-3.1-8B-Instruct 8 6.66 ms 4.81 ms 1.38 1201
unsloth/Meta-Llama-3.1-8B-Instruct 16 7.00 ms 5.14 ms 1.36 2286
The predictions differ slightly from the table earlier in this chapter because these runs use 1,000-token prompts and 128 steps, so the average context is 1,064 tokens instead of 2,000. Four things stand out:
The total sets the memory, because every expert must be in GPU memory in case a token chooses it: 61.1 GB in BF16. The active count sets the FLOPs, because each token runs only its 8 chosen experts plus attention, the routers and the output head: about 6.1 GFLOP per token.
E × (1 − (1 − k/E)^B) = 128 × (1 − (120/128)^4) = 128 × (1 − 0.772) ≈ 29.1 experts. That is nearly four times the 8 experts that one token reads, which is why the MoE model's step time grows quickly with the batch.
Each weight is 2 bytes and does one multiply and one add, 2 FLOPs, for each token that passes through it. With n tokens, every 2 bytes read support 2 × n FLOPs, which is n FLOPs per byte. An expert that receives 4 tokens therefore runs at about 4 FLOPs per byte, far below an H100's ridge point of about 295.
The KV cache only gets what the weights leave over. The MoE model's 61.1 GB of weights leave 10.9 GB of the usable 72 GB, while the 8B model's 16.1 GB leave 55.9 GB. At 98,304 bytes per token, a 2,000-token sequence needs about 197 MB, so 10.9 GB holds 55 of them. FP8 weights halve the weight memory and raise the count to about 210.
The even share is 256 × 8 ÷ 32 = 64 token slots per expert, so the capacity is 1.25 × 64 = 80. A token that arrives at an expert that already holds 80 tokens skips that expert, and its vector reaches the next layer through the residual connection without that expert's contribution.
An auxiliary balancing loss evens out the expert load, but a coefficient large enough to balance well also hurts the model's quality. DeepSeek-V3 instead adds a bias to each expert's routing score, used only to choose experts and not to weight their outputs. After each training step it lowers the bias of experts that were overloaded and raises the bias of those that were underloaded, which balances the load without adding a term to the loss.
moe_align_block_size.py) and the Triton grouped-matmul kernel (fused_moe.py) described above.Previous: The arithmetic of inference · Next: How a GPU executes code