{"id":"inference-fleets","title":"Running inference at scale","sections":[{"id":"overview","title":"Overview","html":"<p>A model server on one machine answers quickly while its requests fit in its GPUs. A product needs many servers, and a fleet of GPU servers differs from the web fleet in the <a href=\"/learn/load-balancing\">load balancing</a> chapter. A recipe page on Crumb, the recipe site from earlier chapters, takes about 20 ms and frees its server. A reply from a large model holds GPU memory for seconds, and each server's cache decides how much work its next request needs. So open requests stop measuring load, and the choice of server changes what a request costs.</p>\n<p>This chapter answers five questions:</p>\n<ul>\n<li>What makes a fleet of GPU servers different from a fleet of web servers?</li>\n<li>How should a router pick a server when each server's cache matters?</li>\n<li>How do you autoscale when a new server takes minutes to become ready?</li>\n<li>When should prefill and decode run on separate machines, and how do urgent and patient work share GPUs?</li>\n<li>What happens to requests when a GPU fails?</li>\n</ul>\n<p>The examples use the running example: Llama 3.1 70B with BF16 weights (2 bytes per number) on four H100 GPUs, split by tensor parallelism, which divides each layer's matrix multiplications across the GPUs. The weights take about 140 GB. After 10% of the 320 GB is kept free, about 148 GB remains for the KV cache, which stores the keys and values of past tokens. That holds about 450,000 tokens. We call one copy of a model on its own GPUs a <strong>replica</strong>.</p>","headings":[]},{"id":"from-one-server-to-a-fleet","title":"From one server to a fleet","html":"<h3 id=\"from-one-server-to-a-fleet\">From one server to a fleet</h3>\n<p>A recipe page is short. A reply on the running example is long. Decoding makes one token per step, and with 64 users each user gets about 74 tokens a second. A 500-token reply takes 500 ÷ 74 ≈ 6.8 seconds, and it holds a batch slot and cache for all of that time. A 2,000-token conversation holds 2,000 × 327,680 bytes ≈ 655 MB.</p>\n<p>So a replica is limited by memory and batch slots, and least connections ignores that. Two replicas with 40 open requests each can hold very different amounts of cache. Forty 500-token chats hold 40 × 500 × 327,680 bytes ≈ 6.6 GB. Forty 4,096-token documents hold about 53.7 GB, eight times as much.</p>\n<p>The Kubernetes Gateway API Inference Extension calls a set of replicas an InferencePool and sends each request to an endpoint picker, which chooses one replica. That picker is the router in this chapter.</p>\n<figure class=\"diagram\" data-text=\"Clients -> Gateway auth and limits (HTTPS); Gateway auth and limits -> Router queue and cache; Router queue and cache -> Replica A 4 GPUs; Router queue and cache -> Replica B 4 GPUs; Router queue and cache -> Replica C 4 GPUs\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 640 320\" width=\"640\" role=\"img\" aria-label=\"One request passes a gateway, then a router that reads each replica&#x27;s state.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"438.7\" y1=\"133\" x2=\"518.7\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"438.7\" y1=\"191\" x2=\"518.7\" y2=\"251\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Clients</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"158.5\">Gateway</tspan><tspan x=\"240\" y=\"174.5\">auth and limits</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"158.5\">Router</tspan><tspan x=\"400\" y=\"174.5\">queue and cache</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Replica A</tspan><tspan x=\"560\" y=\"54.5\">4 GPUs</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">Replica B</tspan><tspan x=\"560\" y=\"174.5\">4 GPUs</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"278.5\">Replica C</tspan><tspan x=\"560\" y=\"294.5\">4 GPUs</tspan></text><rect class=\"dg-edge-bg\" x=\"138.5\" y=\"114\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"127\" text-anchor=\"middle\">HTTPS</text></svg></div><figcaption>One request passes a gateway, then a router that reads each replica's state.</figcaption></figure>","headings":[{"id":"from-one-server-to-a-fleet","title":"From one server to a fleet","depth":3}]},{"id":"routing-requests","title":"Routing requests","html":"<h3 id=\"routing-requests\">Routing requests</h3>\n<p>Round robin assumes every request costs the same. Least connections assumes an open connection means the same work everywhere. Better signals come from the engine. vLLM publishes <code>vllm:num_requests_waiting</code> (requests queued, not yet running) and <code>vllm:kv_cache_usage_perc</code> (the fraction of cache in use) as Prometheus metrics. Queue length says how long a new request waits, and cache usage says whether a long prompt fits.</p>\n<p>The llm-d router's endpoint picker weighs KV-cache locality, load and priority, according to its README. The Gateway API Inference Extension's README, read in October 2026, says its endpoint picker has moved to llm-d. Both projects are changing quickly, so check their status before you depend on either.</p>\n<p>Routing needs more than load because of prefix caching. An engine that holds a prompt's shared beginning, such as a system prompt, skips prefilling it. A 1,500-token system prompt costs 1,500 × 140 GFLOP = 210 TFLOP to prefill (a TFLOP is a trillion floating-point operations): about 53 ms at full compute on four H100s, or 106 ms at 50%. But that cache lives on one replica, and a request sent elsewhere pays again.</p>\n<p>The gain can be large. In December 2024, the SGLang team reported that its cache-aware router, released in v0.4, raised throughput from 82,665 to 158,596 tokens a second and cache hits from 20% to 75%. The run used 8 A100 80 GB GPUs, Llama 3.1 8B and a synthetic shared-prefix workload. The table compares v0.3 with v0.4, so these are release-to-release gains.</p>\n<p>A router can't see each replica's cache without a round trip per request. SGLang's router keeps an approximate radix tree (a tree of prompt prefixes) for each worker, built from the requests it routed. Its current docs make <code>cache_aware</code> the default, which balances cache locality against load. They also warn that the tree isn't shared across router replicas, which can cost 10 to 20 percent of cache hits. Mooncake's Conductor scheduler goes further. It checks each prefill instance's matching cache and picks the lowest estimated time to first token.</p>\n<p>A simpler design hashes the prefix or session ID onto a <a href=\"/learn/consistent-hashing\">consistent hashing</a> ring. Each prefix gets a home replica with no router state, and adding a replica moves only about 1/N of prefixes. The weakness is a hot prefix, such as a system prompt shared by all traffic, which overloads its home. The <a href=\"/learn/consistent-hashing#hot-keys\">hot keys</a> section's fix sends the overflow to the next replica clockwise. The vLLM production stack's router offers round-robin and session-ID routing, with prefix-aware routing still in progress.</p>\n<p>The sketch below routes to the longest cached prefix. It falls back to the least loaded replica when nothing matches, or when the match is on a much busier replica. Each block's hash includes the previous one, so a match on one block implies a match on every earlier block. Blocks are four words here, where engines use tokens.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BLOCK</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 4</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # words per cache block (real engines use tokens, often 16 per block)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> block_hashes</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(words):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Chain each block's hash onto the last, so a match implies the whole prefix.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    hashes, prev </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(words) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> BLOCK</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BLOCK</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        prev </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib.sha256(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">prev</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">' '</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(words[i:i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> BLOCK</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.encode()).hexdigest()[:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">12</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        hashes.append(prev)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashes</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> leading_matches</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(replica_cache, hashes):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"How many leading blocks of the prompt a replica already holds.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> next</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">((i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i, h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> enumerate</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(hashes) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">not</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> replica_cache), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(hashes))</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> PrefixRouter</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, names, max_imbalance</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {n: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> names}      </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># requests in flight: the load signal</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.cached </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {n: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">set</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> names}  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># block hashes each replica probably holds</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_imbalance </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_imbalance</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> route</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, prompt):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        hashes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> block_hashes(prompt.split())</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        score </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {n: leading_matches(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.cached[n], hashes) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active}</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        least </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">key</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active.get)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        best </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">key</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=lambda</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n: (score[n], </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active[n]))</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">        # Take the cache hit unless that replica is much busier than the least loaded one.</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        hit_ok </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> score[best] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> and</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active[best] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active[least] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_imbalance</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        chosen </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> best </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hit_ok </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> least</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.cached[chosen].update(hashes)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># assume the whole prompt is cached afterwards</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.active[chosen] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> chosen, score[chosen]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    router </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> PrefixRouter([</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"b\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    system </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"You are the support assistant for Acme. Answer briefly and cite the policy page.\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    questions </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"How do I reset my password?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Where is my invoice?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"How do I cancel?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                 \"Can I change my email?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Why was I charged twice?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Do you offer refunds?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> q </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> questions:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(router.route(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">system</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> {</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">q</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(router.route(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Summarize this 40 page contract for me.\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span></code></pre>\n<p>The demo sends six questions sharing a system prompt, then one unrelated request:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>('a', 0)</span></span>\n<span class=\"line\"><span>('a', 3)</span></span>\n<span class=\"line\"><span>('a', 4)</span></span>\n<span class=\"line\"><span>('a', 3)</span></span>\n<span class=\"line\"><span>('a', 3)</span></span>\n<span class=\"line\"><span>('b', 0)</span></span>\n<span class=\"line\"><span>('b', 0)</span></span></code></pre>\n<p>Replica <code>a</code> takes the first request, since nothing is cached. The third question begins like the first, so it matches four blocks. The sixth goes to <code>b</code>, because <code>a</code> has five requests in flight and <code>b</code> has none, a gap beyond the limit of four. The sketch omits two things a real router needs: lowering <code>active</code> when a stream ends, and bounding <code>cached</code>. SGLang bounds its tree with <code>--max-tree-size</code> and <code>--eviction-interval-secs</code>.</p>","headings":[{"id":"routing-requests","title":"Routing requests","depth":3}]},{"id":"autoscaling","title":"Autoscaling","html":"<h3 id=\"autoscaling\">Autoscaling</h3>\n<p>The signal matters more than the threshold. CPU use says little about a GPU replica, which does the work while the CPU waits. Queue length and time to first token say more about whether users are waiting.</p>\n<p>The Kubernetes Horizontal Pod Autoscaler (HPA) sets desired replicas to ceiling(current replicas × current metric ÷ target). Six replicas that each average 6 waiting requests, against a target of 4, give ceiling(6 × 6 ÷ 4) = 9. The HPA evaluates this every 15 seconds by default. For scale-down, it uses the highest desired count from the last 300 seconds, so a short dip doesn't remove replicas the next minute needs. Scale-up has no window.</p>\n<p>New replicas arrive late. The next section estimates about 209 seconds for a fresh node on a 10 Gbps link, so a spike at minute zero waits about 15 + 209 = 224 seconds for relief. Four responses follow, and most fleets combine several:</p>\n<ul>\n<li><strong>Keep headroom.</strong> Run more replicas than average load needs. Each replica is four GPUs, so this is costly.</li>\n<li><strong>Keep a warm pool.</strong> Hold nodes that already have the image and weights on local disk. A scale-up then pays only for the GPU load and the engine start, about 88 seconds below, while the idle GPUs cost money.</li>\n<li><strong>Shed or queue low-priority work</strong> while replicas start (see <a href=\"/learn/inference-fleets#priorities-and-fairness\">priorities and fairness</a>).</li>\n<li><strong>Scale on queue length and time to first token,</strong> which rise as soon as requests start to wait.</li>\n</ul>","headings":[{"id":"autoscaling","title":"Autoscaling","depth":3}]},{"id":"loading-models-quickly","title":"Loading models quickly","html":"<h3 id=\"loading-models-quickly\">Loading models quickly</h3>\n<p>A new replica pulls a container image with the serving software and CUDA libraries (NVIDIA's GPU software), downloads the weights, loads them into GPU memory, and starts the engine. Each stage takes seconds to minutes.</p>\n<p>The weights come from the Hugging Face metadata for Llama-3.1-70B-Instruct, a gated repository, so you need an approved account to download it. It lists 70,553,706,496 BF16 parameters across 30 files, which is 141.1 GB at 2 bytes each. The other inputs are assumptions to replace with your own measurements: a 10 GB image, a 10 Gbps download link (1.25 GB/s), a local disk reading at 5 GB/s, and a 60-second engine start. Adding the stages with no overlap gives an upper bound.</p>\n<figure class=\"diagram\" data-text=\"Metric crosses target -> Pull image 8 s; Pull image 8 s -> Download weights 113 s; Download weights 113 s -> Load to GPUs 28 s; Load to GPUs 28 s -> Engine starts 60 s, then ready\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 80\" width=\"800\" role=\"img\" aria-label=\"A new replica serves nothing until every stage is done (assumed times).\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">Metric crosses</tspan><tspan x=\"80\" y=\"54.5\">target</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Pull image</tspan><tspan x=\"240\" y=\"54.5\">8 s</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Download weights</tspan><tspan x=\"400\" y=\"54.5\">113 s</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Load to GPUs</tspan><tspan x=\"560\" y=\"54.5\">28 s</tspan></text><rect class=\"dg-node dg-box\" x=\"652\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"38.5\">Engine starts</tspan><tspan x=\"720\" y=\"54.5\">60 s, then ready</tspan></text></svg></div><figcaption>A new replica serves nothing until every stage is done (assumed times).</figcaption></figure>\n<p>The working is:</p>\n<ul>\n<li>Image: 10 GB ÷ 1.25 GB/s = 8 s</li>\n<li>Weights download: 141.1 GB ÷ 1.25 GB/s = 112.9 s</li>\n<li>Load from disk into GPU memory: 141.1 GB ÷ 5 GB/s = 28.2 s</li>\n<li>Engine start: 60 s</li>\n<li>Total: 8 + 112.9 + 28.2 + 60 = 209.1 s, about 3.5 minutes</li>\n</ul>\n<p>The script below computes this for several links, a warm node, and the KV-cache transfer used in the next section.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Llama 3.1 70B in BF16 on 4 H100 GPUs: cold start and KV transfer, idealized.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">GB</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">WEIGHTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 70_553_706_496</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">         # 70.55 billion parameters x 2 bytes (Hugging Face metadata)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">IMAGE</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 10</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> GB</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                      # assumption: serving image size</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">DISK_READ</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 5</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> GB</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                   # assumption: local NVMe read, bytes per second</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">ENGINE_START</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 60</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                    # assumption: engine init, memory setup, warm-up (seconds)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">KV_PER_TOKEN</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 80</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 8</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 128</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # 327,680 bytes: 2 x layers x KV heads x head dim x bytes</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> cold_start</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(link_gbps, warm</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Seconds until a new replica can serve. The stages are added, not overlapped.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> warm:  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># image and weights are already on the node's disk</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> WEIGHTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> DISK_READ</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> ENGINE_START</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    download </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">IMAGE</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> WEIGHTS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (link_gbps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Gbps to bytes per second</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> download </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> WEIGHTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> DISK_READ</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> ENGINE_START</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> kv_transfer_ms</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(prompt_tokens, link_gbps):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> prompt_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> KV_PER_TOKEN</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (link_gbps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1000</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> gbps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">25</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">100</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cold start at </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">gbps</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:>3</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> Gbps: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">cold_start(gbps)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:4.0f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> s\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"warm node:            </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">cold_start(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">warm</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:4.0f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> s\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> gbps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">25</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">800</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"KV cache of 2,000 tokens over </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">gbps</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> Gbps: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">kv_transfer_ms(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, gbps)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:5.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> ms\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"prefill of 2,000 tokens at 50% of 4 x 989 TFLOPS: \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">      f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{2000</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 140e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 989e12</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1000</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.0f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> ms\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>cold start at  10 Gbps:  209 s</span></span>\n<span class=\"line\"><span>cold start at  25 Gbps:  137 s</span></span>\n<span class=\"line\"><span>cold start at 100 Gbps:  100 s</span></span>\n<span class=\"line\"><span>warm node:              88 s</span></span>\n<span class=\"line\"><span>KV cache of 2,000 tokens over 25 Gbps: 209.7 ms</span></span>\n<span class=\"line\"><span>KV cache of 2,000 tokens over 800 Gbps:   6.6 ms</span></span>\n<span class=\"line\"><span>prefill of 2,000 tokens at 50% of 4 x 989 TFLOPS: 142 ms</span></span></code></pre>\n<p>A 100 Gbps link cuts the download of the image and weights to about 12 seconds, which makes the engine start the largest stage. Four techniques attack the stages:</p>\n<ul>\n<li><strong>Keep weights near the GPUs,</strong> on node-local disks or a regional cache. ServerlessLLM loads checkpoints through multi-tier storage and places each model where it can start fastest. Its abstract reports 10 to 200 times lower latency than state-of-the-art serverless systems, without naming the baselines.</li>\n<li><strong>Load shards in parallel.</strong> Each GPU reads only its share of the weights.</li>\n<li><strong>Give slow starts time.</strong> Kubernetes startup probes exist for \"Pods that have containers that take a long time to come into service.\" Until one succeeds, the liveness and readiness probes wait.</li>\n<li><strong>Pre-pull images</strong> on every node, so a scale-up skips that stage.</li>\n</ul>","headings":[{"id":"loading-models-quickly","title":"Loading models quickly","depth":3}]},{"id":"separating-prefill-and-decode","title":"Separating prefill and decode","html":"<h3 id=\"separating-prefill-and-decode\">Separating prefill and decode</h3>\n<p>Prefill reads the whole prompt in one pass and is compute-heavy: 2,000 tokens need 280 TFLOP. Decode makes one token per step, reading the weights and the cache each time, so it is memory-heavy. On shared GPUs the two interfere. A long prefill delays other users' decode steps. DistServe, an OSDI 2024 paper, calls this \"strong prefill-decoding interferences.\" Splitwise draws the same line: token generation leaves compute idle, even with batching.</p>\n<p>The fix runs prefill and decode on different pools. After prefill, the KV cache moves over the network to a decode GPU.</p>\n<figure class=\"diagram\" data-text=\"Clients <-> Router (prompt in, reply out); Router -> Prefill pool compute-heavy (new prompts); Prefill pool compute-heavy -> Decode pool memory-heavy (KV cache moves); Decode pool memory-heavy -> Router (tokens out)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 480 320\" width=\"480\" role=\"img\" aria-label=\"Prefill and decode run on separate pools. The KV cache moves between them.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"153\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"276\" y1=\"135\" x2=\"358.7\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"400\" y1=\"71\" x2=\"400\" y2=\"251\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"361.3\" y1=\"253\" x2=\"278.7\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Clients</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"166.5\">Router</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Prefill pool</tspan><tspan x=\"400\" y=\"54.5\">compute-heavy</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"278.5\">Decode pool</tspan><tspan x=\"400\" y=\"294.5\">memory-heavy</tspan></text><rect class=\"dg-edge-bg\" x=\"93\" y=\"116\" width=\"134\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"160\" y=\"129\" text-anchor=\"middle\">prompt in, reply out</text><rect class=\"dg-edge-bg\" x=\"278.2\" y=\"95\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"317.3\" y=\"108\" text-anchor=\"middle\">new prompts</text><rect class=\"dg-edge-bg\" x=\"351.6\" y=\"152\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"400\" y=\"165\" text-anchor=\"middle\">KV cache moves</text><rect class=\"dg-edge-bg\" x=\"284\" y=\"213\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"226\" text-anchor=\"middle\">tokens out</text></svg></div><figcaption>Prefill and decode run on separate pools. The KV cache moves between them.</figcaption></figure>\n<p>The cost is the copy. A 2,000-token prompt makes about 655 MB of cache. Over an 800 Gbps link (100 GB/s), the copy takes about 6.6 ms. Over a 25 Gbps link (3.1 GB/s), it takes about 210 ms, longer than the 142 ms of prefill at 50% utilization. A fast link, or a copy that overlaps computation, makes separation pay.</p>\n<p>The papers measured different things, so check their setups:</p>\n<ul>\n<li><strong>DistServe</strong> (OSDI 2024) ran on 4 nodes with 32 A100 80 GB GPUs and 25 Gbps between nodes, against vLLM and DeepSpeed-MII. Its \"7.4 times more requests\" is against DeepSpeed-MII on chatbot work. Its \"12.6 times tighter SLO\" is against vLLM on summarization. These are separate comparisons.</li>\n<li><strong>Splitwise</strong> (arXiv 2023, ISCA 2024) replayed Azure production traces on DGX-A100 and DGX-H100 servers. Its abstract claims up to 1.4 times the throughput at 20% lower cost, or 2.35 times at equal cost and power. Against a DGX-A100 baseline at equal power and cost, the paper reports 2.15 times on the conversation trace.</li>\n<li><strong>Mooncake</strong> (arXiv 2407.00079), the serving platform for Kimi, pools KV cache across CPU memory and SSDs. Its abstract reports up to 525% higher throughput in certain simulated scenarios, and 75% more requests handled for Kimi on real workloads.</li>\n</ul>\n<p>Separation has costs. You size two pools, and the right split moves with the mix of prompt and reply lengths. Short prompts gain little, and a lost decode machine takes its cache with it. A simpler option is <a href=\"/learn/inference-engines#chunked-prefill\">chunked prefill</a>, which interleaves long prompts with decode steps on one pool.</p>","headings":[{"id":"separating-prefill-and-decode","title":"Separating prefill and decode","depth":3}]},{"id":"serving-many-models","title":"Serving many models","html":"<h3 id=\"serving-many-models\">Serving many models</h3>\n<p>Serving many models means several base models in separate pools, with the gateway routing by model name, and many fine-tunes of one base model through LoRA adapters. RouteLLM goes further and chooses between a stronger and a weaker model during inference. Its abstract reports cost reductions of over 2 times in certain cases, without compromising quality.</p>\n<p>A LoRA adapter is a small pair of matrices added to frozen base weights. Take Llama 3.1 8B: 32 layers, model dimension 4,096, 32 attention heads and 8 KV heads (Table 3 of the Llama 3 paper), so each head has dimension 128. Put a rank-16 adapter on the query and value projections. A rank-r adapter on a d_in × d_out matrix adds r × (d_in + d_out) parameters. Queries add 16 × 8,192 = 131,072. Values, with 8 KV heads of 128 making 1,024 outputs, add 16 × 5,120 = 81,920. That is 212,992 per layer, or 6,815,744 across 32 layers, about 13.6 MB in BF16. The base model is about 16 GB, so an adapter is roughly a thousandth of it.</p>\n<p>S-LoRA keeps all adapters in main memory, moves the ones that running requests need onto the GPU, and batches different adapters with custom kernels. Its abstract reports thousands of adapters on one or several GPUs, and up to 4 times the throughput of HuggingFace PEFT and of vLLM's early LoRA support. Punica batches different LoRA models in one CUDA kernel over a shared base model. Its abstract reports 12 times the throughput of state-of-the-art systems on a fixed GPU cluster, at 2 ms of extra latency per token.</p>\n<p>Adapters make placement a routing problem: send adapter X to a replica that has it loaded. The Gateway API Inference Extension's README lists \"LoRA Adapters availability\" among its signals.</p>","headings":[{"id":"serving-many-models","title":"Serving many models","depth":3}]},{"id":"priorities-and-fairness","title":"Priorities and fairness","html":"<h3 id=\"priorities-and-fairness\">Priorities and fairness</h3>\n<p>A person in a chat waits for the first token. A nightly summary waits for nothing. Serve the first first, and use spare capacity for the second. The llm-d endpoint picker weighs priority, and the Gateway API Inference Extension's roadmap lists \"Fairness and priority between workloads within the same criticality band\" as future work.</p>\n<p>Admission control turns away work that can't meet its target. Mooncake's Conductor rejects a request on arrival, with HTTP 429 (Too Many Requests), when predicted latency would break its SLO (the latency target), before any prefill work is spent. In a replay of 23,000 requests on 8 prefill and 8 decode instances, the baseline rejected 4,183, early rejection 3,771, and prediction-based early rejection 3,589. Early rejection wastes less work, but the fleet still rejects, so plan capacity for that.</p>\n<p>Limit tokens, not requests, because a tenant sending 100,000-token prompts uses far more prefill and cache than one sending 500-token chats. The <a href=\"/learn/rate-limiting#token-bucket\">token bucket</a> works with tokens as its unit. When the cache fills, the engine preempts a running request and resumes it later, at the cost of recomputation (see <a href=\"/learn/inference-engines#managing-kv-cache-memory\">managing KV cache memory</a>).</p>","headings":[{"id":"priorities-and-fairness","title":"Priorities and fairness","depth":3}]},{"id":"when-gpus-fail","title":"When GPUs fail","html":"<h3 id=\"when-gpus-fail\">When GPUs fail</h3>\n<p>Hardware fails often at fleet scale. Meta's Llama 3 paper reports a 54-day pre-training run on 16,000 GPUs with 466 job interruptions, 419 of them unexpected, and about 78% of the unexpected ones attributed to confirmed hardware issues. That is one interruption every 2.8 hours (1,296 hours ÷ 466). The run was training, and the failure modes carry over to serving. Plan for a replica to vanish in the middle of a request.</p>\n<p>Health checks must exercise the engine, because a process can answer a ping while its GPU has faulted. A small generation test catches more. A failed Kubernetes readiness probe removes the pod from matching Services, and a liveness probe that fails past its tolerance restarts the container. Passive checks, which watch real errors, catch what probes miss (see <a href=\"/learn/load-balancing#health-checks\">health checks</a>).</p>\n<p>If a replica dies after the user has seen part of a reply, the answer depends on what was sent:</p>\n<ol>\n<li>If no token reached the client, retry on another replica. The prefill repeats, but the client has seen nothing, so the retry is safe.</li>\n<li>If tokens reached the client, send the prompt plus those tokens to another replica and ask it to continue. Sampling is random, so the continuation may differ, which suits chat but not exact reproduction.</li>\n<li>If the gateway didn't keep the partial reply, or the product needs an exact answer, return an error and let the client retry.</li>\n</ol>\n<p>Retries are safe for a chat reply. They are unsafe once the model has called a tool that changes state, so give those calls an <a href=\"/learn/reliability-patterns#idempotency-keys\">idempotency key</a>. Tensor parallelism also widens the failure, since one GPU's loss stops all four. Treat a replica's GPUs as one unit for health checks and restarts.</p>","headings":[{"id":"when-gpus-fail","title":"When GPUs fail","depth":3}]},{"id":"offline-batch-inference","title":"Offline batch inference","html":"<h3 id=\"offline-batch-inference\">Offline batch inference</h3>\n<p>Some work has no user waiting, such as evaluations or nightly summaries. Throughput matters, and latency doesn't. Batch APIs trade latency for price. The OpenAI Batch API guide, read in October 2026, says each batch completes within 24 hours (\"and often more quickly\"), allows up to 50,000 requests per batch, and accepts input files up to 200 MB. The long window lets the provider fill spare capacity, and each provider sets its own price.</p>\n<p>Batch work can run a larger decode batch, because each step reads the weights once however many sequences share it. For weights alone, the H100's compute and memory traffic balance at about 295 FLOPs per byte, which for BF16 weights means about 295 sequences. The KV cache changes that. At 2,000 tokens, reading one sequence's cache takes about 49 µs per step, more than its 35 µs of arithmetic, so decode stays memory-bound at any batch size. The cache also caps the batch at 450,000 ÷ 2,000 = 225 sequences. An idealized step at that size reads 141.1 GB of weights and 225 × 2,000 × 327,680 bytes ≈ 147.5 GB of cache, 288.6 GB in all. At 4 × 3.35 TB/s, that takes about 21.5 ms, so 225 ÷ 0.0215 ≈ 10,400 tokens a second. The 64-user interactive case gave about 4,700.</p>\n<p>At an assumed $3.00 per H100-hour (real prices vary by provider and commitment), a four-GPU replica costs $12 an hour. That is about $0.32 per million tokens at 10,400 tokens a second with full utilization, or $0.71 at 4,700. Real fleets run below full utilization, so real costs are higher. The <a href=\"/learn/ai-estimation#cost-per-million-tokens\">AI estimation chapter</a> does this arithmetic in full.</p>\n<p>A separate batch pool never competes with interactive traffic, but it sits idle without batch work. A shared pool uses the slots that interactive traffic leaves, but its scheduler must admit interactive requests first and cap batch memory.</p>","headings":[{"id":"offline-batch-inference","title":"Offline batch inference","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>A new replica of the 70B model starts on a node with a 25 Gbps download link, a 10 GB image, a 5 GB/s local disk and a 60-second engine start. Estimate the time until it serves, and name the largest stage.</summary>\n<p>The image and weights download in (10 + 141.1) GB ÷ 3.125 GB/s ≈ 48.4 s. Loading from disk takes 28.2 s, and the engine start is 60 s. The total is about 137 s, or 2.3 minutes. The engine start is the largest stage.</p>\n</details>\n<details>\n<summary>Two replicas both show 40 open requests. Replica A's KV cache is 90% full and B's is 30% full. Where should the next long request go, and why would least connections get this wrong?</summary>\n<p>B, which has room for a long prompt. Least connections sees a tie, but 40 requests can hold very different amounts of cache. The KV usage and queue length metrics show what the connection count hides.</p>\n</details>\n<details>\n<summary>A prefix-aware router sends every request with the same 1,500-token system prompt to replica A, whose queue grows while B is idle. Name two changes and what each costs.</summary>\n<p>Lower the imbalance limit (the sketch's <code>max_imbalance</code>) so requests spill to B sooner, though each spill repeats the 1,500-token prefill, about 53 ms of compute. Or use a consistent hashing ring with bounded loads, which moves only the overflow, and that overflow loses its cache.</p>\n</details>\n<details>\n<summary>A four-GPU replica loses one GPU while a reply is 200 tokens into a 600-token answer. What happens to the replica, and what can the client see?</summary>\n<p>All four GPUs are needed, so the replica fails, and its readiness probe removes it from routing. The gateway can resend the prompt plus the 200 tokens to another replica, which may write a different continuation, or it can return an error. The client sees a pause, then either a continuation or an error.</p>\n</details>\n<details>\n<summary>Interactive time to first token at p99 rises from 0.5 to 3 seconds, but time per output token is unchanged. Long documents are a growing share of traffic. What would you check, and what would you consider changing?</summary>\n<p>Check the prefill queue wait and the prompt length distribution. Decode is healthy, so long prompts are queuing ahead of new arrivals. Consider prefix caching, chunked prefill, or a separate prefill pool, after checking that the KV transfer fits the link.</p>\n</details>\n<details>\n<summary>Someone proposes running a nightly job of 50,000 summaries on the interactive replicas to avoid buying GPUs. What limits that job on one replica, and what is the main risk?</summary>\n<p>The KV cache caps the batch at about 225 sequences of 2,000 tokens. The main risk is interactive latency, since a large batch and long prefills take the memory and compute chat users need. Run the job at lower priority with a cap on its cache share, or use a batch API.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<p>Papers:</p>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2401.09670\" target=\"_blank\" rel=\"noopener noreferrer\">DistServe</a>: disaggregated prefill and decode, with the setups behind its numbers.</li>\n<li><a href=\"https://arxiv.org/abs/2311.18677\" target=\"_blank\" rel=\"noopener noreferrer\">Splitwise</a>: phase splitting on production traces.</li>\n<li><a href=\"https://arxiv.org/abs/2407.00079\" target=\"_blank\" rel=\"noopener noreferrer\">Mooncake</a>: KV-cache-centric disaggregation and its scheduler.</li>\n<li><a href=\"https://arxiv.org/abs/2311.03285\" target=\"_blank\" rel=\"noopener noreferrer\">S-LoRA</a>: thousands of LoRA adapters with unified paging.</li>\n<li><a href=\"https://arxiv.org/abs/2310.18547\" target=\"_blank\" rel=\"noopener noreferrer\">Punica</a>: batching requests for different LoRA models.</li>\n<li><a href=\"https://arxiv.org/abs/2401.14351\" target=\"_blank\" rel=\"noopener noreferrer\">ServerlessLLM</a>: multi-tier checkpoint loading for cold starts.</li>\n<li><a href=\"https://arxiv.org/abs/2406.18665\" target=\"_blank\" rel=\"noopener noreferrer\">RouteLLM</a>: routing between a stronger and a cheaper model.</li>\n<li><a href=\"https://arxiv.org/abs/2407.21783\" target=\"_blank\" rel=\"noopener noreferrer\">The Llama 3 Herd of Models</a>: Table 3 and the interruption statistics in section 3.3.4.</li>\n</ul>\n<p>Documentation, code and data:</p>\n<ul>\n<li><a href=\"https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct\" target=\"_blank\" rel=\"noopener noreferrer\">Llama-3.1-70B-Instruct</a>: the gated model page, with the parameter metadata.</li>\n<li><a href=\"https://www.nvidia.com/en-us/data-center/h100/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA H100</a>: memory and bandwidth of the SXM module.</li>\n<li><a href=\"https://docs.sglang.io/advanced_features/sgl_model_gateway.html\" target=\"_blank\" rel=\"noopener noreferrer\">SGLang Model Gateway</a>: cache-aware routing and its thresholds.</li>\n<li><a href=\"https://lmsys.org/blog/2024-12-04-sglang-v0-4\" target=\"_blank\" rel=\"noopener noreferrer\">SGLang v0.4 post</a>: the cache-aware benchmark and its workload.</li>\n<li><a href=\"https://github.com/llm-d/llm-d-router\" target=\"_blank\" rel=\"noopener noreferrer\">llm-d router</a>: an endpoint picker that weighs cache, load and priority.</li>\n<li><a href=\"https://gateway-api-inference-extension.sigs.k8s.io/concepts/api-overview/\" target=\"_blank\" rel=\"noopener noreferrer\">Gateway API Inference Extension</a>: the InferencePool resource. Its <a href=\"https://github.com/kubernetes-sigs/gateway-api-inference-extension\" target=\"_blank\" rel=\"noopener noreferrer\">repository</a> has the status and roadmap.</li>\n<li><a href=\"https://github.com/vllm-project/production-stack\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM production stack</a>: the router's routing options.</li>\n<li><a href=\"https://docs.vllm.ai/en/stable/usage/metrics.html\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM metrics</a>: the metric names used here.</li>\n<li><a href=\"https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/\" target=\"_blank\" rel=\"noopener noreferrer\">Kubernetes HPA</a>: the formula, the sync period and the stabilization window.</li>\n<li><a href=\"https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/\" target=\"_blank\" rel=\"noopener noreferrer\">Kubernetes probes</a>: startup, readiness and liveness semantics.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/batch\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI Batch API guide</a>: the completion window and request limits.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}