{"id":"sglang-prefix-caching","title":"SGLang and prefix caching","sections":[{"id":"overview","title":"Overview","html":"<p><a href=\"/learn/inside-vllm\">Chapter 5.2</a> ended with one measurement of prefix caching: 63 requests that shared a 3,000-token prefix finished 12.7 times sooner when vLLM could reuse the prefix's keys and values. That experiment was the easy case, with one prefix, sent in advance, on an idle server. This chapter is about the general problem. A server holds the KV of thousands of earlier requests, a new prompt arrives, and the engine must find the longest stored prefix of it, keep that prefix alive while it is in use, decide what to throw away when memory runs out, and, if it is clever, choose which waiting request to run next so that the cache is used well.</p>\n<p>SGLang is the engine that was designed around this problem. Its paper calls the technique RadixAttention: the KV cache of every request, finished or running, is indexed by a radix tree over token ids. vLLM solves the same problem with hashed blocks. The chapter reads SGLang's design against vLLM's, implements both in a few hundred lines to see where they differ, and then measures the two engines on a multi-turn chat workload on an H100.</p>\n<p>Everything about SGLang refers to <strong>SGLang 0.5.23</strong> (commit <code>297e54f</code>, 28 October 2026), and everything about vLLM to 0.32.0, the version of chapter 5.2. Paths for SGLang are under <code>python/sglang/srt/</code>. Before this chapter, read <a href=\"/learn/inside-vllm\">Inside vLLM</a>, whose section on the block pool this chapter builds on.</p>","headings":[]},{"id":"where-prefixes-repeat","title":"Where prefixes repeat","html":"<h3 id=\"where-prefixes-repeat\">Where prefixes repeat</h3>\n<p>The keys and values of a token depend on that token and on every token before it, and on nothing after it. This is a property of causal attention (<a href=\"/learn/transformer-as-workload\">The transformer, seen by the hardware</a>), and it is the whole basis of prefix caching: if two prompts agree on their first n tokens, the KV of those n tokens is identical in both, and the second prompt can use the first one's. The reuse is exact. The model produces the same logits it would have produced by computing everything again.</p>\n<p>The same property sets the limit. Only a prefix can be reused. If two prompts share a 2,000-token document but one of them has a different first sentence, every token of the document sees a different past, and nothing can be shared.</p>\n<p>Four kinds of traffic have long shared prefixes.</p>\n<ul>\n<li><strong>System prompts.</strong> Every request of an application starts with the same instructions, tool definitions and policies, often thousands of tokens of them.</li>\n<li><strong>Multi-turn chat.</strong> Chat APIs are stateless, so each turn resends the whole conversation. Turn 6 of a conversation is turn 5's prompt, turn 5's answer and one new message.</li>\n<li><strong>Agents.</strong> An agent loop is a chat in which the turns are tool calls and their results. Each step resends everything so far, and the history grows by a tool result at a time.</li>\n<li><strong>Few-shot prompts and shared documents.</strong> A batch of questions is asked against the same examples, or many questions against one long document.</li>\n</ul>\n<p>What the cache saves is prefill. In chapter 5.2's trace, a 2,022-token prompt took 85.8 ms to its first token when nearly everything was computed, and a second prompt of 2,021 tokens, whose first 2,000 tokens were found in the cache, took 31.1 ms. Decoding is not affected, so the saving shows in time to first token and in the GPU time that prefill would have taken from other requests, and it grows with the ratio of prompt length to output length. For an agent that sends 20,000 tokens of history to receive a 100-token tool call, prefill is nearly all of the work.</p>\n<p>What the cache costs is memory. The KV of one token of Llama 3.1 8B takes 131,072 bytes, so a cached 2,000-token system prompt holds 262 MB of the KV pool for as long as it stays. Neither engine reserves separate memory for this. Cached tokens live in the same pool as the tokens of running requests, and are given up when running requests need the space.</p>\n<p>Because a hit requires the tokens to match from position 0, small things in a prompt decide whether the cache works at all. A timestamp or a request id at the top of a system prompt changes the first tokens of every request and makes every one of them a miss. So does serializing tool definitions in a different order from one request to the next. The rule for writing prompts follows directly: put what never changes first, what changes per user next, and what changes per request last.</p>","headings":[{"id":"where-prefixes-repeat","title":"Where prefixes repeat","depth":3}]},{"id":"a-radix-tree-over-token-ids","title":"A radix tree over token ids","html":"<h3 id=\"a-radix-tree-over-token-ids\">A radix tree over token ids</h3>\n<p>A trie stores sequences so that sequences with a common prefix share a path from the root. A radix tree is a trie in which a chain of nodes with one child each is merged into a single node, so each node holds a run of tokens. SGLang's tree stores token sequences, and each node stands for the KV of the tokens in its run.</p>\n<figure class=\"diagram\" data-text=\"root -> System prompt 2,000 tokens; root -> Examples 1,200 tokens; System prompt 2,000 tokens -> Chat A, turn 1 180 tokens; System prompt 2,000 tokens -> Chat B, turn 1 150 tokens; Chat A, turn 1 180 tokens -> Chat A, turn 2 210 tokens; Examples 1,200 tokens -> Question 1 40 tokens; Examples 1,200 tokens -> Question 2 35 tokens\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 640 440\" width=\"640\" role=\"img\" aria-label=\"Five requests in a radix tree. Each box is a node: a run of tokens and the KV slots that hold them\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"135.4\" x2=\"167\" y2=\"129.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"104\" y1=\"189\" x2=\"212.4\" y2=\"311\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"75.4\" x2=\"327\" y2=\"69.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"128.6\" x2=\"327\" y2=\"134.6\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"315.4\" x2=\"327\" y2=\"309.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"368.6\" x2=\"327\" y2=\"374.6\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-note\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">root</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"76\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"98.5\">System prompt</tspan><tspan x=\"240\" y=\"114.5\">2,000 tokens</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Chat A, turn 1</tspan><tspan x=\"400\" y=\"54.5\">180 tokens</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Chat A, turn 2</tspan><tspan x=\"560\" y=\"54.5\">210 tokens</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"158.5\">Chat B, turn 1</tspan><tspan x=\"400\" y=\"174.5\">150 tokens</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"316\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"338.5\">Examples</tspan><tspan x=\"240\" y=\"354.5\">1,200 tokens</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"278.5\">Question 1</tspan><tspan x=\"400\" y=\"294.5\">40 tokens</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"376\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"398.5\">Question 2</tspan><tspan x=\"400\" y=\"414.5\">35 tokens</tspan></text></svg></div><figcaption>Five requests in a radix tree. Each box is a node: a run of tokens and the KV slots that hold them</figcaption></figure>\n<p>A node (<code>TreeNode</code> in <code>mem_cache/radix_cache.py</code>) has four things that matter here.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Field</th><th>What it holds</th></tr></thead><tbody><tr><td><code>key</code></td><td>The token ids of this node's run</td></tr><tr><td><code>value</code></td><td>One KV slot index per token of the key: where in the pool that token's keys and values are</td></tr><tr><td><code>children</code></td><td>A dictionary from the first token of each child's key to the child</td></tr><tr><td><code>lock_ref</code>, <code>last_access_time</code></td><td>How many running requests use this node, and when it was last matched</td></tr></tbody></table></div>\n<p>The memory underneath is simpler than vLLM's. With SGLang's defaults on an NVIDIA GPU, the page size is 1: the pool is an array of token slots, the allocator (<code>TokenToKVPoolAllocator</code>) hands out individual slot indices, and a table with one row per running request (<code>req_to_token</code>) maps each position of the request to its slot. Where vLLM's block table has one entry per 16 tokens, SGLang's has one per token. A cache hit therefore has no block boundary to round down to, and a request that reuses a prefix simply copies the prefix's slot indices into its own row.</p>\n<p>Looking up a prompt is a walk from the root. This is the lookup in the Python class, shortened.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> _match_prefix_helper</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, node, key):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    access_time </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> time.monotonic()</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    child_key </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> key.child_key(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.page_size)          </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the first token of what is left</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    value </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> []</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    while</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(key) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> child_key </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> node.children.keys():</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        child </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> node.children[child_key]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        child.last_access_time </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> access_time</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        prefix_len </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> child.key.match(key)              </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># how many tokens of this node's run agree</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> prefix_len </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(child.key):                </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the prompt leaves the run partway through</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            new_node </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">._split_node(child.key, child, prefix_len)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            value.append(new_node.value)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            node </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> new_node</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            break</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        value.append(child.value)                      </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the whole run matched: take its slots, go deeper</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        node </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> child</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        key </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> key[prefix_len:]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(key):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            child_key </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> key.child_key(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.page_size)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> value, node</span></span></code></pre>\n<p>At each node the walk does one dictionary lookup on the next token, then compares the node's run with the prompt. <code>RadixKey.match</code> does the comparison with slices of an integer array, doubling the window until it finds a difference and then halving it to find the exact token, so a 2,000-token run that matches completely is compared in eleven C-level comparisons and not in a Python loop over tokens. The cost of a lookup is proportional to the number of nodes on the path, and a path has one node per point where earlier prompts diverged, which is far fewer than its tokens.</p>\n<p>When the prompt agrees with only the first part of a node's run, the node is <strong>split</strong>: a new node takes the part that matched and becomes the parent of the old node, which keeps the rest. The slot indices are divided the same way and no KV moves. Splitting is what keeps the invariant that makes the tree useful: every point where two stored sequences diverge is a node boundary, so \"the longest cached prefix of this prompt\" is always \"the path to some node\".</p>\n<p>In 0.5.23 the tree a default server uses is no longer this Python class. <code>mem_cache/registry.py</code> builds a <code>UnifiedRadixCache</code>, whose tree is implemented in Rust (<code>rust/sglang-radix-tree/</code>), with a Python implementation of the same contract as the fallback. The structure and the operations are the ones described here: nodes with a key, a value and children, a match that splits, and eviction from the leaves. The Python class is still in the source, is still used for the scheduler's own small tree described below, and is the right file to read first.</p>","headings":[{"id":"a-radix-tree-over-token-ids","title":"A radix tree over token ids","depth":3}]},{"id":"a-requests-life-in-the-tree","title":"A request's life in the tree","html":"<h3 id=\"a-requests-life-in-the-tree\">A request's life in the tree</h3>\n<p>Five moments in a request's life touch the tree. Each has a counterpart in vLLM, and the table after the list puts them side by side.</p>\n<ol>\n<li><strong>Admission: match and lock.</strong> When the scheduler considers a waiting request it matches the prompt against the tree. The match may cover at most the prompt minus its last token (<code>_compute_max_prefix_len</code> in <code>managers/schedule_batch.py</code>), because the last token has to be run to produce the logits of the first output token. The request then locks the node where the match ended: <code>lock_ref</code> is incremented on that node and on every ancestor up to the root. A node with a nonzero <code>lock_ref</code> cannot be evicted, so the prefix stays in place while the request uses it.</li>\n<li><strong>Allocation, with eviction on demand.</strong> The request needs slots for the tokens that were not matched. If the free list is short, the allocator asks the tree to evict enough leaves to cover the shortfall (<code>evict_to_free_tokens</code> in <code>mem_cache/allocator/base.py</code>). Nothing is evicted before that moment. When the scheduler decides whether a request fits, it counts evictable cache as available memory.</li>\n<li><strong>After the prefill: publish the prompt.</strong> As soon as the request's prefill result is processed, <code>checkpoint_kv_cache</code> inserts the prompt into the tree and moves the request's lock to the new node. From then on a second request with the same prefix shares it, while the first is still decoding. A prompt that is prefilled in chunks is published after each chunk.</li>\n<li><strong>At the end: publish the answer.</strong> When the request finishes, the prompt and the generated tokens are inserted, and the lock is released. This is what makes multi-turn chat work: the next turn's prompt begins with this turn's prompt and answer, and both are in the tree. The insert splits the new node at the end of the prompt, so that eviction can later drop the answer and keep the prompt.</li>\n<li><strong>Eviction.</strong> The tree keeps the set of leaves with <code>lock_ref</code> 0. To evict, it puts them in a heap ordered by the eviction policy, removes the first, frees its slots, and if the removed leaf's parent has now become an unlocked leaf, adds the parent to the heap. The default policy, <code>--radix-eviction-policy lru</code>, orders leaves by <code>last_access_time</code>; <code>lfu</code>, <code>slru</code>, <code>priority</code> and <code>tlru</code> are the other choices. Leaves go first because a leaf is a suffix that only one stored sequence needs, and evicting an inner node would orphan everything below it.</li>\n</ol>\n<p>The pool can still fill up with running requests, which no eviction can help. SGLang then <strong>retracts</strong> some of them (<code>retract_decode</code> in <code>managers/schedule_batch.py</code>), which is vLLM's preemption by recomputation under another name. The requests with the fewest output tokens go first, their KV is freed without being put in the tree, and they return to the waiting queue with the output they have so far.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th></th><th>SGLang 0.5.23</th><th>vLLM 0.32.0</th></tr></thead><tbody><tr><td>Index</td><td>A radix tree over token ids</td><td>A hash table from chained block hashes to blocks</td></tr><tr><td>Unit of memory</td><td>1 token (page size 1 by default)</td><td>A block of 16 tokens</td></tr><tr><td>Longest possible hit</td><td>Prompt length minus 1</td><td>The whole blocks within prompt length minus 1</td></tr><tr><td>Lookup cost</td><td>One step per tree node on the path, each a slice comparison</td><td>One SHA-256 per block when the request arrives, then one table lookup per block</td></tr><tr><td>What protects a prefix in use</td><td><code>lock_ref</code> on every node of the path</td><td>A reference count on every block</td></tr><tr><td>When a prompt becomes shareable</td><td>After its prefill step has run</td><td>When its blocks are allocated, before the step runs</td></tr><tr><td>Is the answer cached</td><td>Yes, at the end of the request</td><td>Yes, block by block as the output fills blocks</td></tr><tr><td>What is evicted</td><td>The least recently used unlocked leaf, whole</td><td>The block at the front of the free queue</td></tr><tr><td>When</td><td>When an allocation is short</td><td>When the block is handed out again</td></tr><tr><td>Order within one request</td><td>Answer before prompt, later turns before earlier</td><td>Tail blocks before head blocks</td></tr><tr><td>Pool full of running requests</td><td>Retract the requests with the fewest output tokens</td><td>Preempt the request that started running last</td></tr></tbody></table></div>\n<p>Two of these rows describe real differences in behaviour, and the rest describe two implementations of the same policy.</p>\n<p>The first is granularity. A vLLM hit is rounded down to a multiple of 16 tokens, so each request recomputes up to 15 tokens that a token-level cache would have found. Against prompts of thousands of tokens this is under one percent.</p>\n<p>The second is the moment of publication, and it matters when many requests with the same new prefix arrive together. vLLM enters a block in its table when the block is allocated, so a request scheduled one step after another can already share the first one's blocks, even though the first has not finished computing them. That is safe because the step that fills them is already ahead in the queue. SGLang publishes after the prefill, so requests admitted in the same batch as the first each compute the prefix for themselves, unless the scheduler holds them back, which is the subject of the section on scheduling.</p>\n<p>Everything else in the table is the same idea twice. Both engines keep cached KV in the pool that running requests use, both give it no size setting of its own, both evict in least-recently-used order from the ends of sequences inward, and both protect what a running request reads by counting references. A chained hash is a way of naming a path in a trie without storing the trie: the hash of block k identifies the same thing as the tree node reached after 16k tokens.</p>","headings":[{"id":"a-requests-life-in-the-tree","title":"A request's life in the tree","depth":3}]},{"id":"measuring-the-two-designs-without-a-gpu","title":"Measuring the two designs without a GPU","html":"<h3 id=\"measuring-the-two-designs-without-a-gpu\">Measuring the two designs without a GPU</h3>\n<p>The claim that the two designs behave alike is easy to test, because neither needs a model. <a href=\"/labs/infra/prefix_cache_sim.py\">prefix_cache_sim.py</a> implements both from scratch, in about 120 and 80 lines: <code>RadixCache</code>, with match, split, locks and leaf eviction, and <code>BlockCache</code>, with chained hashes, the free queue and vLLM's rule that freed blocks with a hash go to the back and blocks without one to the front. Both manage a pool of the same number of token slots and are driven by the same traces, with four requests in flight at a time.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>$ python prefix_cache_sim.py</span></span>\n<span class=\"line\"><span>all checks passed</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>Hit rate (cached prompt tokens / prompt tokens), 4 requests in flight, first come first served</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>multi-turn chat: 24 conversations of 8 turns, one 800-token system prompt</span></span>\n<span class=\"line\"><span>  192 requests, 307,374 prompt tokens, 42,183 distinct tokens to hold; a cache that never evicts reaches 95.7%</span></span>\n<span class=\"line\"><span>    pool, tokens   radix tree  hashed blocks  evicted, tree  evicted, blocks</span></span>\n<span class=\"line\"><span>           4,096        51.5%          51.5%        173,785          172,400</span></span>\n<span class=\"line\"><span>           8,192        55.5%          55.6%        158,500          155,792</span></span>\n<span class=\"line\"><span>          16,384        78.8%          79.4%         79,501           74,608</span></span>\n<span class=\"line\"><span>          32,768        95.0%          94.6%         11,610           11,344</span></span>\n<span class=\"line\"><span>          65,536        95.6%          95.2%              0                0</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>few-shot: 200 questions after the same 2,000 tokens of examples</span></span>\n<span class=\"line\"><span>  200 requests, 407,941 prompt tokens, 12,436 distinct tokens to hold; a cache that never evicts reaches 97.6%</span></span>\n<span class=\"line\"><span>    pool, tokens   radix tree  hashed blocks  evicted, tree  evicted, blocks</span></span>\n<span class=\"line\"><span>           4,096        97.6%          97.6%          8,393            6,960</span></span>\n<span class=\"line\"><span>           8,192        97.6%          97.6%          4,256            2,864</span></span>\n<span class=\"line\"><span>          16,384        97.6%          97.6%              0                0</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>12 tenants, each with its own system prompt of 1,000 to 2,000 tokens</span></span>\n<span class=\"line\"><span>  300 requests, 534,941 prompt tokens, 66,234 distinct tokens to hold; a cache that never evicts reaches 93.0%</span></span>\n<span class=\"line\"><span>    pool, tokens   radix tree  hashed blocks  evicted, tree  evicted, blocks</span></span>\n<span class=\"line\"><span>           4,096        25.1%          27.9%        425,798          408,528</span></span>\n<span class=\"line\"><span>           8,192        49.6%          52.8%        291,107          271,280</span></span>\n<span class=\"line\"><span>          16,384        76.7%          78.0%        137,396          128,480</span></span>\n<span class=\"line\"><span>          32,768        92.8%          92.4%         34,780           35,040</span></span></code></pre>\n<p>Three things can be read from these tables.</p>\n<p>The two designs are within a few points of each other everywhere. With a pool large enough to hold everything, the tree is ahead by 0.4 points on the chat workload, 95.6% against 95.2%, which is the 16-token rounding. With a small pool the blocks are slightly ahead, 27.9% against 25.1% for the tenants at 4,096 tokens, because the tree evicts a whole leaf when it needs a few slots, and a leaf can be a system prompt of more than a thousand tokens, while the block cache gives up 16 tokens at a time and keeps the head of what it was evicting.</p>\n<p>The size of the pool matters far more than the index. The chat workload has 42,183 distinct tokens to hold. A pool of 32,768 tokens reaches 95%, and a pool of 4,096 reaches 51.5%, with either design. The hit rate of a real server is decided by how much of the pool its running requests leave free, which is to say by load.</p>\n<p>What the cache needs depends on the shape of the traffic. The few-shot workload has one hot prefix, and any pool that holds it reaches the maximum. The chat workload needs room for the conversations that are idle between turns as well as for the ones that are running: at 8,192 tokens the cache keeps little more than the 800-token system prompt, which is half of an average prompt of 1,600 tokens, and the hit rate is 55%. The tenant workload needs room for twelve system prompts at once, and below that it thrashes.</p>","headings":[{"id":"measuring-the-two-designs-without-a-gpu","title":"Measuring the two designs without a GPU","depth":3}]},{"id":"cache-aware-scheduling","title":"Cache-aware scheduling","html":"<h3 id=\"cache-aware-scheduling\">Cache-aware scheduling</h3>\n<p>A cache only helps a request whose prefix is still in it when the request runs. With a queue of waiting requests, the order in which they are run changes how often that is true, and this is the second idea of the SGLang paper. Suppose 120 requests from six few-shot tasks are waiting, shuffled together, and the pool has room for the examples of one or two tasks. In arrival order, nearly every request belongs to a different task from the one before it, so its examples have just been evicted to make room for someone else's. If the scheduler instead runs all the requests of one task before starting the next, each task's examples are computed once.</p>\n<p>The paper proves the general statement. For a batch of requests, visiting them in depth-first order of the radix tree they form achieves the best possible cache hit rate, provided the cache is at least as large as the longest request, and running the request with the longest cached prefix first produces that order. The simulator's last table is this experiment, with one request in flight.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>Scheduling order: 120 waiting requests from 6 few-shot tasks of 2,000 tokens each, shuffled,</span></span>\n<span class=\"line\"><span>one request in flight</span></span>\n<span class=\"line\"><span>    pool, tokens  first come first served  longest prefix first</span></span>\n<span class=\"line\"><span>           2,560                    21.2%                 93.2%</span></span>\n<span class=\"line\"><span>           4,608                    39.2%                 93.2%</span></span>\n<span class=\"line\"><span>           6,656                    49.8%                 93.2%</span></span>\n<span class=\"line\"><span>          12,800                    87.4%                 93.2%</span></span></code></pre>\n<p>With a pool that holds one task's examples, first-come-first-served finds 21% of prompt tokens in the cache and longest-prefix-first finds 93%, which is the most this queue allows, since each of six tasks must be computed once. The order stops mattering once the pool holds all six tasks.</p>\n<p>In SGLang the order is set by <code>--schedule-policy</code>, and <code>SchedulePolicy.calc_priority</code> (<code>managers/schedule_policy.py</code>) sorts the waiting queue before each round of admissions.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Policy</th><th>Order of the waiting queue</th></tr></thead><tbody><tr><td><code>fcfs</code> (the default)</td><td>Arrival order</td></tr><tr><td><code>lpm</code></td><td>Longest matched prefix first</td></tr><tr><td><code>dfs-weight</code></td><td>Depth-first through the tree, entering the subtree with the most waiting requests first</td></tr><tr><td><code>lof</code>, <code>random</code>, <code>routing-key</code>, <code>hrrn</code>, <code>shortest-prefill-first</code></td><td>Longest output first, random order, and three more specialized orders</td></tr></tbody></table></div>\n<p>Request priorities are a separate switch, <code>--enable-priority-scheduling</code>, which sorts by priority before applying the policy. The default is first come, first served. Cache-aware order is something you turn on, and the code shows why it is not free. Longest-prefix-first has no notion of waiting time, so a request with an unusual prompt can be overtaken for as long as better-matching requests keep arriving. Matching and sorting the whole queue on every scheduling round also costs CPU time in the loop that feeds the GPU, so when more than 128 requests are waiting, <code>lpm</code> (and <code>hrrn</code>) falls back to arrival order (<code>_determine_active_policy</code>).</p>\n<p>The <code>lpm</code> policy does one more thing, as do <code>hrrn</code> and <code>shortest-prefill-first</code> but not <code>dfs-weight</code> or any cache-agnostic order. Under the heading of in-batch prefix caching, <code>_compute_prefix_matches</code> builds a second, temporary radix tree out of the waiting queue itself. A request that matches little of the real cache (32 tokens or fewer) is looked up in this tree. If an earlier waiting request shares at least 32 tokens with it, the later one is moved to the back of the queue for this round. The effect is that when many requests with the same new prefix arrive together, one of them is run first, its prompt is published, and the others then find it. Without this, the requests admitted in the same batch each compute the shared prefix, since SGLang publishes a prompt only after its prefill.</p>","headings":[{"id":"cache-aware-scheduling","title":"Cache-aware scheduling","depth":3}]},{"id":"simulation","title":"Simulation","html":"<h3 id=\"simulation\">Simulation</h3>\n<p>The widget is a radix-tree cache over a pool of a few dozen token slots. Each button sends one request, which runs to completion before the next. Chats A, B and C start with the same 12-token system prompt, and each turn adds a 3-token message and a 5-token answer to that chat's history. A few-shot question is 16 tokens of examples and a 3-token question.</p>\n<div class=\"widget-slot\" data-widget=\"radix-tree\"></div>\n<p>Try these, in order.</p>\n<ol>\n<li>Send a turn of chat A and then a turn of chat B. The tree had one node of 20 tokens, and the second request matched only its first 12, so the node was split into the system prompt and chat A's turn.</li>\n<li>Send two more turns of chat A. Each finds its whole history, including the previous answer, and adds one node. Compare \"cached\" with the prompt length: only the new message is computed.</li>\n<li>Press \"Same prompt again\". The whole prompt is in the tree, and the match is still one token short of it.</li>\n<li>Reset, choose a pool of 32, and send one turn each of chats A, B and C. The third request needs 8 slots and only 4 are free, and the leaf that was used longest ago, chat A's turn, is evicted. The system prompt survives because it is not a leaf, and because the request being served has locked it.</li>\n<li>With a pool of 48, alternate few-shot questions with chat turns until something is evicted, and predict which node it will be from the \"used\" numbers before you press.</li>\n<li>Watch the hit rate for 4-token hashed blocks beside the tree's. Find a request where the two differ, and say which block boundary caused it.</li>\n</ol>","headings":[{"id":"simulation","title":"Simulation","depth":3}]},{"id":"both-engines-on-a-chat-workload","title":"Both engines on a chat workload","html":"<h3 id=\"both-engines-on-a-chat-workload\">Both engines on a chat workload</h3>\n<p>The lab script, <a href=\"/labs/infra/gpu/prefix_cache_bench.py\">prefix_cache_bench.py</a>, sends the same two workloads to vLLM 0.32.0 and to SGLang 0.5.23, each serving Llama 3.1 8B on an H100 with default settings. The two engines pin different versions of PyTorch, so each has its own image and its own run.</p>\n<p>The chat workload is 48 conversations of 6 turns through <code>/v1/chat/completions</code>. Every conversation starts with the same system prompt of about 2,000 tokens, every turn resends the whole history, and answers are limited to 150 tokens. Each simulated user waits between 1 and 4 seconds after an answer before sending the next message, so at any moment most conversations are idle and their KV is in the cache and not in use. Each engine reports the cached part of every prompt in the response's <code>usage.prompt_tokens_details.cached_tokens</code>, which vLLM fills in when started with <code>--enable-prompt-tokens-details</code> and SGLang with <code>--enable-cache-report</code>.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Configuration</th><th>Engine</th><th>KV pool, tokens</th><th>Prompt tokens found in the cache</th><th>First token, median</th><th>First token, p99</th></tr></thead><tbody><tr><td>Prefix cache on (default)</td><td>SGLang</td><td>410,558</td><td>98.3%</td><td>25.1 ms</td><td>34.8 ms</td></tr><tr><td></td><td>vLLM</td><td>451,056</td><td>97.9%</td><td>34.9 ms</td><td>77.0 ms</td></tr><tr><td>Prefix cache off</td><td>SGLang</td><td>410,558</td><td>0</td><td>243.4 ms</td><td>505.4 ms</td></tr><tr><td></td><td>vLLM</td><td>452,928</td><td>0</td><td>198.7 ms</td><td>569.5 ms</td></tr><tr><td>Pool cut to 32,768 tokens</td><td>SGLang</td><td>32,768</td><td>95.8%</td><td>25.6 ms</td><td>47.5 ms</td></tr><tr><td></td><td>vLLM</td><td>32,768</td><td>95.6%</td><td>31.8 ms</td><td>54.3 ms</td></tr><tr><td>Pool cut to 16,384 tokens</td><td>SGLang</td><td>16,384</td><td>85.9%</td><td>30.0 ms</td><td>328.8 ms</td></tr><tr><td></td><td>vLLM</td><td>16,384</td><td>85.1%</td><td>38.1 ms</td><td>219.1 ms</td></tr></tbody></table></div>\n<p>The 288 requests carried about 699,000 prompt tokens in every run. The flags that produce the rows are <code>--disable-radix-cache</code> and <code>--max-total-tokens</code> for SGLang, and <code>--no-enable-prefix-caching</code> and <code>--kv-cache-memory-bytes</code> for vLLM, where 32,768 tokens are 4 GiB at 131,072 bytes a token.</p>\n<p><strong>The cache removes nearly all of the prefill.</strong> With the cache on, each engine computes about 2% of the prompt tokens it receives. A median first-turn prompt is 2,063 tokens, of which SGLang finds 2,026 and computes 37, the user's question. The median time to first token falls from 243 ms to 25 ms on SGLang and from 199 ms to 35 ms on vLLM. The times without a cache are much longer than one 2,000-token prefill takes on an idle server, because the prefills of concurrent requests queue behind one another and share steps with decoding. The whole workload, which is mostly users thinking, also finishes sooner: 25.6 seconds against 32.1 on SGLang and 25.5 against 31.3 on vLLM.</p>\n<p><strong>The two indexes find the same prefixes.</strong> The hit rates differ by 0.4 points with a full-size pool, 0.2 with 32,768 tokens and 0.8 with 16,384. The first of these is the block rounding and nothing else. On a first turn vLLM finds 2,016 tokens where SGLang finds 2,026, and 2,016 is 126 blocks of 16. Over the run, SGLang found 2,497 more tokens than vLLM in 288 requests, which is 8.7 tokens per request, about what rounding down to a multiple of 16 costs on average. This is the simulator's result on real engines.</p>\n<p><strong>Under memory pressure both degrade the same way.</strong> By the last turn the conversations hold about 43,000 distinct tokens: the system prompt once, and roughly 860 tokens of history for each of the 48 conversations. A pool of 32,768 tokens cannot keep all of that, and a pool of 16,384 keeps less than half. The per-turn numbers show what is given up.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Turn</th><th>Median prompt, tokens</th><th>Cached with a full pool (SGLang, vLLM)</th><th>Cached with 16,384 tokens (SGLang, vLLM)</th></tr></thead><tbody><tr><td>1</td><td>2,063</td><td>2,026, 2,016</td><td>2,026, 2,016</td></tr><tr><td>2</td><td>about 2,210</td><td>2,169, 2,160</td><td>2,166, 2,160</td></tr><tr><td>3</td><td>about 2,355</td><td>2,315, 2,304</td><td>2,026, 2,016</td></tr><tr><td>4</td><td>about 2,500</td><td>2,450, 2,448</td><td>2,026, 2,016</td></tr><tr><td>5</td><td>about 2,645</td><td>2,603, 2,592</td><td>2,026, 2,016</td></tr><tr><td>6</td><td>about 2,785</td><td>2,744, 2,720</td><td>2,026, 2,016</td></tr></tbody></table></div>\n<p>With the small pool, from the third turn on, the median request finds exactly the system prompt and nothing of its own conversation. While a user was thinking, other conversations needed slots, and the idle conversation's history was the least recently used leaf in SGLang and the front of the free queue in vLLM. The system prompt survives in both, because some running request is nearly always holding it. That is the behaviour to expect from a loaded server. Shared prefixes stay cached under almost any pressure, and per-conversation history is cached only while the pool has room to spare. It is also why the hit rate stays at 85% and does not collapse: the system prompt is most of every prompt. In neither engine did the small pool cause requests to be thrown out. SGLang retracted none and vLLM preempted one.</p>\n<p><strong>Differences in latency between the engines are not explained by the cache.</strong> With the cache on, SGLang's median first token arrived in 25 ms and vLLM's in 35 ms, and without it vLLM was ahead, 199 ms against 243 ms. Both engines computed the same number of tokens in each case, so these gaps come from elsewhere: how each schedules a prefill among running decodes, the cost of each engine's request path on the CPU, and the fact that the two runs were on different hosts. Chapter 5.2 measured 60 ms and 86 ms for the same request on two H100 hosts. A fair comparison of the engines' speed needs repeated runs on one machine at a controlled arrival rate, which is the subject of the load-testing chapter in Part 8. What this experiment establishes is the part that does not depend on the host: which tokens each engine found in its cache.</p>\n<p>The second workload is the burst of chapter 5.2 with one change: all 64 requests, which share a 3,000-token prefix, are sent at the same moment to a server that has never seen the prefix. Any hit must come from another request in the same burst.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th></th><th>Burst finished in</th><th>Requests with no hit</th><th>Prompt tokens found in the cache</th></tr></thead><tbody><tr><td>vLLM, cache on</td><td>0.55 s</td><td>1</td><td>188,496 of 195,200</td></tr><tr><td>SGLang, cache on, <code>fcfs</code></td><td>0.54 s</td><td>1</td><td>189,000 of 195,200</td></tr><tr><td>SGLang, cache on, <code>lpm</code></td><td>0.40 s</td><td>1</td><td>189,000 of 195,200</td></tr><tr><td>vLLM, cache off</td><td>5.17 s</td><td>64</td><td>0</td></tr><tr><td>SGLang, cache off</td><td>5.09 s</td><td>64</td><td>0</td></tr></tbody></table></div>\n<p>A cold cache costs one request. In every run with a cache, exactly one request computed the prefix and the other 63 reused it, 2,992 tokens each in vLLM (187 blocks) and 3,000 each in SGLang. SGLang's log shows how: its first prefill batch held one request and computed 3,050 tokens, and its second held the other 63, computed 3,150 tokens between them and took 189,000 from the cache.</p>\n<p>This run does not show the in-batch problem that <code>lpm</code> exists to solve, and the reason is worth understanding. \"At the same moment\" for an HTTP client still means one request reaches the scheduler first. On an idle server it is admitted alone, and by the time its prefill has run, the rest are waiting and find its prompt in the tree. The default order therefore did as well as <code>lpm</code>. The two runs computed exactly the same tokens, so the difference between 0.54 and 0.40 seconds is not the cache's doing; each figure is a single run on a freshly started server, and the gap is too small to read anything into. In a trial on an A10 with Llama 3.2 1B, two requests were admitted before the prefix was published and both computed it, under <code>fcfs</code>, where <code>lpm</code> again had one. The case where the order matters is a busy server, where requests with the same new prefix accumulate in the waiting queue during a long step and are then admitted together.</p>","headings":[{"id":"both-engines-on-a-chat-workload","title":"Both engines on a chat workload","depth":3}]},{"id":"lab-two-caches-and-two-engines","title":"Lab: two caches and two engines","html":"<h3 id=\"lab-two-caches-and-two-engines\">Lab: two caches and two engines</h3>\n<p><strong>Tier: [Mac] for Part A, [1 GPU] for Part B.</strong></p>\n<p><strong>Part A, the simulator.</strong> <a href=\"/labs/infra/prefix_cache_sim.py\">prefix_cache_sim.py</a> needs only Python's standard library.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">python</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> prefix_cache_sim.py</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">           # about 17 seconds on an M2</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">python</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> prefix_cache_sim.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --test</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # only the self-checks</span></span></code></pre>\n<p>Read <code>RadixCache</code> and <code>BlockCache</code>, then do three things. First, before running, predict for each workload which design has the higher hit rate at the smallest pool and why, then check. Second, change <code>BlockCache.finish</code> to free a request's blocks from head to tail instead of tail to head, and measure what vLLM's reversed order is worth. Third, add an <code>order=\"dfs\"</code> option to <code>replay</code> that visits the waiting queue in depth-first order of the tree its prompts form, and compare it with <code>lpm</code> on the scheduling experiment.</p>\n<p><strong>Part B, the engines.</strong> The script is <a href=\"/labs/infra/gpu/prefix_cache_bench.py\">prefix_cache_bench.py</a>. It tests one engine per run, each in its own image.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">cd</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> gpu</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">modal</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> run</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> modal_run.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --script</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> prefix_cache_bench.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --gpu</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> 'H100!'</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --vllm</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">   --args</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"--engine vllm\"</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     # 10 to 20 min, $0.65 to $1.30</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">modal</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> run</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> modal_run.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --script</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> prefix_cache_bench.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --gpu</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> 'H100!'</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --sglang</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --args</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"--engine sglang\"</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   # about 9.5 min, $0.61</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">modal</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> run</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> modal_run.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --script</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> prefix_cache_bench.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --gpu</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> A10</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --vllm</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">   --args</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"--engine vllm --quick\"</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     # Llama 3.2 1B: 5.5 min, $0.10</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">modal</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> run</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> modal_run.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --script</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> prefix_cache_bench.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --gpu</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> A10</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --sglang</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --args</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"--engine sglang --quick\"</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   # 6.5 min, $0.12</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">modal</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> run</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> modal_run.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --script</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> prefix_cache_bench.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --gpu</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> A10</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --sglang</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --args</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"--engine sglang --quick --unique-system --only 'cache on'\"</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   # 2.5 min, $0.05</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">python</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> gcp_run.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --script</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> prefix_cache_bench.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --gpu</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> A100-80GB</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --sglang</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --args</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"--engine sglang\"</span></span></code></pre>\n<p>The first time <code>--sglang</code> is used, the launcher builds the SGLang image, which takes several minutes and is not billed as GPU time. The vLLM run's time depends mostly on how fast its host downloads the model: the reference run waited 14 minutes for the first server to start.</p>\n<p>From your own runs, fill in the tables of the previous section. Then explain three things from the source of each engine: why the two hit rates differ by a fraction of a percent, which tokens each engine gave up when the pool was cut to 16,384 tokens, and how one request's prefill served 63 others in the burst. Last, change one thing about the workload that you expect to lower the hit rate, such as putting the customer's number at the start of the system prompt (the script's <code>--unique-system</code> option does this), predict the new hit rate, and measure it with <code>--quick</code>.</p>\n<details>\n<summary>Reference solution</summary>\n<p><em>Part A.</em> The output of the unmodified script is in the sections \"Measuring the two designs without a GPU\" and \"Cache-aware scheduling\". At the smallest pool the block cache is level or ahead, because it evicts 16 tokens at a time and the tree evicts whole leaves.</p>\n<p>Freeing blocks head first means replacing <code>reversed(blocks)</code> with <code>blocks</code> in <code>BlockCache.finish</code>. The first block of a prompt then reaches the front of the free queue before the rest, and once it is taken, every later block of that prompt is unreachable, since a lookup stops at the first missing hash, yet those blocks still occupy the pool until they are taken too.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Workload</th><th>Pool, tokens</th><th>Tail first (vLLM)</th><th>Head first</th></tr></thead><tbody><tr><td>Chat</td><td>8,192</td><td>55.6%</td><td>55.4%</td></tr><tr><td>Chat</td><td>16,384</td><td>79.4%</td><td>77.9%</td></tr><tr><td>Chat</td><td>32,768</td><td>94.6%</td><td>94.6%</td></tr><tr><td>Tenants</td><td>8,192</td><td>52.8%</td><td>48.6%</td></tr><tr><td>Tenants</td><td>16,384</td><td>78.0%</td><td>76.3%</td></tr><tr><td>Tenants</td><td>32,768</td><td>92.4%</td><td>92.3%</td></tr></tbody></table></div>\n<p>The order is worth up to 4 points when the pool is tight and nothing when it is not. It matters most for the tenants, whose prompts are a long shared head and a short private tail: tail first gives up the private part and keeps the system prompt.</p>\n<p>For the depth-first order, build a <code>RadixCache</code> with a very large pool, insert every waiting prompt, then walk it depth first and emit each request when the walk reaches the node where its prompt ends. On the scheduling experiment it gives the same 93.2% as <code>lpm</code> at every pool size, which is the paper's theorem: with a cache at least as large as one request, longest-prefix-first produces a depth-first order.</p>\n<p><em>Part B.</em> The reference results are <a href=\"/labs/infra/gpu/prefix_cache_vllm.json\">prefix_cache_vllm.json</a> and <a href=\"/labs/infra/gpu/prefix_cache_sglang.json\">prefix_cache_sglang.json</a>, and the section above explains them. The hit rates differ because vLLM's hit is a whole number of 16-token blocks. With the small pool both engines gave up the histories of idle conversations and kept the system prompt. In the burst, the first request to arrive was scheduled alone, and its prompt was shareable by the time the others were admitted: in vLLM because blocks enter the hash table when they are allocated, and in SGLang because the prompt is inserted into the tree when the prefill's result is processed.</p>\n<p>Putting anything that differs between conversations at the start of the system prompt makes every conversation's first turn a miss on the whole system prompt, while later turns still hit their own history. The script has this change as <code>--unique-system</code>, and the reference run used it with <code>--quick</code> on an A10 (Llama 3.2 1B, 16 conversations of 4 turns, SGLang, <a href=\"/labs/infra/gpu/prefix_cache_sglang_quick_unique_system.json\">prefix_cache_sglang_quick_unique_system.json</a>).</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th></th><th>Hit rate, all turns</th><th>Turn 1, median cached of prompt</th><th>Turn 1, median first token</th><th>Turn 2, median cached of prompt</th></tr></thead><tbody><tr><td>Shared system prompt</td><td>98.2%</td><td>2,026 of 2,063</td><td>26.0 ms</td><td>2,162 of 2,206</td></tr><tr><td>Customer number first</td><td>76.2%</td><td>27 of 2,067</td><td>103.9 ms</td><td>2,175 of 2,216</td></tr></tbody></table></div>\n<p>The 27 tokens that still match are the chat template's header, which comes before the system prompt's text. With 4 turns, one turn in four pays for the whole prompt, so the hit rate falls by about a quarter. With a small pool it would fall much further, because there is no longer a shared prefix that survives eviction.</p>\n</details>","headings":[{"id":"lab-two-caches-and-two-engines","title":"Lab: two caches and two engines","depth":3}]},{"id":"interview-angle","title":"Interview angle","html":"<h3 id=\"interview-angle\">Interview angle</h3>\n<p>\"How does prefix caching work, and how would you implement it?\" is asked for inference roles both as a design question and as a coding question, since a radix tree with reference counts and eviction is a reasonable hour of work. A complete answer starts from why it is exact (a token's KV depends only on the tokens before it), states the constraint that follows (only prefixes, matched from position 0), and then describes the index, the protection of prefixes in use, and eviction. Be ready to describe both indexes and say what each costs: a radix tree gives token-level hits and a natural leaf-first eviction order, and hashed blocks give a flat table with constant-time lookups per block and hits rounded to the block size.</p>\n<p>Expect to be asked what decides the hit rate in production. The useful answer is that the index is the least of it. The measurements here put the two designs within a point of each other, while cutting the pool to 16,384 tokens cost each of them 12 to 13 points. What matters is how much of the KV pool running requests leave free, whether prompts are laid out with the stable part first, and, across several replicas, whether a conversation's next turn reaches the replica that holds its history, which is a routing problem that Part 6 of this handbook takes up.</p>\n<p>A common follow-up is scheduling: why would an engine not run requests in arrival order? Give the few-shot example, state the paper's result that longest-prefix-first is a depth-first walk and is optimal for a batch when the cache holds a request, and then give the costs that keep SGLang's default at first come, first served: starvation of requests with unusual prompts, and the CPU time of matching the whole queue.</p>\n<p>A good closing question to expect is about safety. A shared prefix cache leaks timing: a request that hits is faster, so one tenant can learn whether another tenant recently sent a given prefix. Both engines let a request name a separate cache namespace for this reason, <code>cache_salt</code> in vLLM and in SGLang.</p>","headings":[{"id":"interview-angle","title":"Interview angle","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Why can a server reuse the KV of a shared prefix but not of a shared document that appears in the middle of two different prompts?</summary>\n<p>Under causal attention the keys and values of a token depend on that token and every token before it. Two prompts with the same first n tokens therefore have identical KV for those tokens. A document in the middle of a prompt is preceded by different tokens in the two prompts, so every one of its tokens attends to a different past and has different KV, even though the document's text is the same.</p>\n</details>\n<details>\n<summary>What happens in SGLang's radix tree when a new prompt agrees with the first 12 tokens of a node that holds 20?</summary>\n<p>The node is split. A new node takes the first 12 tokens and their 12 slot indices and becomes the parent of the old node, which keeps the other 8. No keys or values move, since nodes hold slot indices. The new request's match ends at the new node, which it locks, and its remaining tokens are later inserted as a second child of that node.</p>\n</details>\n<details>\n<summary>How does SGLang decide what to evict, and why does it start from the leaves?</summary>\n<p>Eviction happens only when an allocation finds too few free slots, and it stops once the shortfall is covered. The candidates are leaves whose <code>lock_ref</code> is 0, taken in order of the eviction policy, by default least recently accessed first. When a leaf is removed and its parent becomes an unlocked leaf, the parent becomes a candidate. A leaf is the end of one stored sequence, so removing it loses only that suffix, while removing an inner node would make everything beneath it unreachable.</p>\n</details>\n<details>\n<summary>On the chat workload, vLLM found 97.9% of prompt tokens in its cache and SGLang 98.3%. Where does the difference come from?</summary>\n<p>vLLM's cache works in blocks of 16 tokens, so a hit is rounded down to a multiple of 16, while SGLang's default page size is 1 and a hit can end at any token. On a first turn vLLM found 2,016 tokens, which is 126 blocks, where SGLang found 2,026. Over the run the difference was 8.7 tokens per request.</p>\n</details>\n<details>\n<summary>With the KV pool cut to 16,384 tokens, the median request from turn 3 on found only the system prompt in both engines. Why that, and why not less?</summary>\n<p>The conversations needed about 43,000 tokens of KV in all, so most of it had to be evicted. Each conversation's own history is used by one request every few seconds and is idle in between, which makes it the least recently used data when other requests need slots. The system prompt is the prefix of every request, so it is in use by some running request almost all the time, and a prefix in use is locked in SGLang and reference-counted in vLLM.</p>\n</details>\n<details>\n<summary>What does `--schedule-policy lpm` change in SGLang, and why is it not the default?</summary>\n<p>It sorts the waiting queue so that the request with the longest cached prefix is admitted first, and it holds back a request that shares a new prefix with an earlier waiting request, so that the prefix is computed once and then reused. This raises the hit rate when the cache cannot hold every waiting request's prefix. It is not the default because it can starve requests whose prompts match nothing, and because matching every waiting request against the tree on each scheduling round costs CPU time. When more than 128 requests are waiting, it falls back to arrival order.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li>Zheng et al., <a href=\"https://arxiv.org/abs/2312.07104\" target=\"_blank\" rel=\"noopener noreferrer\">SGLang: Efficient Execution of Structured Language Model Programs</a> (2023). Section 3 describes RadixAttention, leaf-first LRU eviction and cache-aware scheduling, and states the theorem on depth-first order.</li>\n<li><a href=\"https://github.com/sgl-project/sglang/tree/v0.5.23\" target=\"_blank\" rel=\"noopener noreferrer\">SGLang source at v0.5.23</a>: <a href=\"https://github.com/sgl-project/sglang/blob/v0.5.23/python/sglang/srt/mem_cache/radix_cache.py\" target=\"_blank\" rel=\"noopener noreferrer\">radix_cache.py</a> for the tree in Python, <a href=\"https://github.com/sgl-project/sglang/blob/v0.5.23/python/sglang/srt/managers/schedule_policy.py\" target=\"_blank\" rel=\"noopener noreferrer\">schedule_policy.py</a> for the scheduling orders and in-batch prefix caching, and <a href=\"https://github.com/sgl-project/sglang/blob/v0.5.23/python/sglang/srt/mem_cache/evict_policy.py\" target=\"_blank\" rel=\"noopener noreferrer\">evict_policy.py</a> for the eviction policies.</li>\n<li><a href=\"https://www.lmsys.org/blog/2024-01-17-sglang/\" target=\"_blank\" rel=\"noopener noreferrer\">Fast and Expressive LLM Inference with RadixAttention and SGLang</a>, the project's first announcement, with diagrams of the tree under several workloads.</li>\n<li>SGLang documentation: <a href=\"https://docs.sglang.io/docs/advanced_features/server_arguments\" target=\"_blank\" rel=\"noopener noreferrer\">server arguments</a> and <a href=\"https://docs.sglang.io/docs/advanced_features/hyperparameter_tuning\" target=\"_blank\" rel=\"noopener noreferrer\">hyperparameter tuning</a>.</li>\n<li>vLLM documentation: <a href=\"https://docs.vllm.ai/en/latest/design/prefix_caching/\" target=\"_blank\" rel=\"noopener noreferrer\">automatic prefix caching</a>, the design of the hashed blocks this chapter compares against.</li>\n<li>Zhang, Li, Moallemi and Peng, <a href=\"https://arxiv.org/abs/2510.15152\" target=\"_blank\" rel=\"noopener noreferrer\">Tail-Optimized Caching for LLM Inference</a> (2025), the source of SGLang's <code>tlru</code> eviction policy, which evicts the part of a conversation's history that cannot affect tail latency first.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}