{"id":"llm-inference","title":"How LLM inference works","sections":[{"id":"overview","title":"Overview","html":"<p>Inference is the work a model server does for each request. It takes the prompt, runs a trained model, and returns the reply. Training happens once, in a separate and much larger job. Inference runs for every message a user sends, so its speed and cost decide what a product can offer.</p>\n<p>A large language model (LLM) predicts the next token of a sequence. A token is a short piece of text, often a word or part of one. Hugging Face's guide to chat templates puts the idea this way: \"All causal LMs, whether chat-trained or not, continue a sequence of tokens.\" An answer therefore comes out one token at a time, and each token depends on every token before it.</p>\n<p>This chapter defines the vocabulary the rest of the AI part uses: prefill and decode, the KV cache (the keys and values an attention step keeps for every token so far), time to first token (TTFT, the wait before the first answer token) and time per output token (TPOT, the average gap between later tokens). Each one gets a concrete example before the general rule. The running example is Meta's Llama 3.1 family, with 8B, 70B and 405B parameters (the learned numbers inside the model). The Llama 3 paper publishes the shape of each size, and the memory arithmetic depends on those shapes.</p>\n<p>For GPU hardware, serving engines and fleet sizing, see <a href=\"/learn/gpus-for-ai\">GPUs and accelerators</a>, <a href=\"/learn/inference-engines\">inference engines</a> and <a href=\"/learn/ai-estimation\">estimating AI workloads</a>.</p>","headings":[]},{"id":"tokens","title":"Tokens","html":"<h3 id=\"tokens\">Tokens</h3>\n<p>A tokenizer splits text into pieces called tokens and maps each piece to an ID in a fixed vocabulary. The model reads and writes IDs, never letters, and a detokenizer turns IDs back into text.</p>\n<p>The OpenAI Cookbook's token-counting notebook shows how tiktoken, OpenAI's open-source tokenizer library, splits the sentence \"tiktoken is great!\" under its o200k_base encoding:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>Token  ID    Piece</span></span>\n<span class=\"line\"><span>1      83    \"t\"</span></span>\n<span class=\"line\"><span>2      8251  \"ikt\"</span></span>\n<span class=\"line\"><span>3      2488  \"oken\"</span></span>\n<span class=\"line\"><span>4      382   \" is\"</span></span>\n<span class=\"line\"><span>5      2212  \" great\"</span></span>\n<span class=\"line\"><span>6      0     \"!\"</span></span></code></pre>\n<p>Six tokens cover three words and a punctuation mark. Each space attaches to the token after it (\" is\", \" great\"). Even \"tiktoken\" splits into \"t\", \"ikt\" and \"oken\", so token counts and word counts differ. Under cl100k_base, the single word \"antidisestablishmentarianism\" becomes six tokens.</p>\n<p>The merge rules come from byte pair encoding (BPE). tiktoken's README calls BPE a way of converting text into tokens, and its educational module shows the procedure: count adjacent pairs, merge the most frequent pair into a new token, and repeat. Sennrich and colleagues used the same idea to split rare words for machine translation. The script below runs the procedure on a tiny corpus, starting from characters rather than bytes.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> collections </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> Counter</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> apply_merge</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(pieces, pair):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    out, i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [], </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    while</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(pieces):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> &#x3C;</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(pieces) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (pieces[i], pieces[i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pair:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            out.append(pieces[i] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pieces[i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            out.append(pieces[i])</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> out</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> learn_merges</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(text, num_merges):  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># start from characters, merge the most common pair</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    words, merges </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> Counter(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">tuple</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(w) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> w </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> text.split()), []  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># word -> count</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> _ </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(num_merges):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        pairs </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> Counter()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> word, freq </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> words.items():</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pair </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> zip</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(word, word[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:]):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                pairs[pair] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> freq</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pairs:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            break</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        best </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(pairs, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">key</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">pairs.get)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># ties go to the pair seen first</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        merges.append(best)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        new_words </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> Counter()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> word, freq </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> words.items():</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            new_words[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">tuple</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(apply_merge(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">list</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(word), best))] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> freq</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        words </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> new_words</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> merges</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">merges </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> learn_merges(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"low low low lower lower newest newest widest\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"merges:\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(p) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> merges])</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># merges: ['lo', 'low', 'es', 'est']</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> word </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"lowest\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"widest\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]:  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># encode: apply the merges in learned order</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    pieces </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> list</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(word)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pair </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> merges:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        pieces </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> apply_merge(pieces, pair)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(word, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"->\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, pieces)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># lowest -> ['low', 'est']</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># widest -> ['w', 'i', 'd', 'est']</span></span></code></pre>\n<p>The first merge is \"lo\", because the pairs l-o and o-w tie at five and the first one counted wins. Encoding applies the merges in the order they were learned, so \"lowest\" becomes \"low\" and \"est\". Frequent pieces become single tokens, and rare text falls back to smaller pieces, so any input can be encoded.</p>\n<p>Llama 3's vocabulary has 128K tokens: 100K from tiktoken plus 28K added for better non-English coverage, according to its paper. On every pass the model produces one score per vocabulary entry. A softmax, which turns scores into probabilities that sum to one, converts them into next-token probabilities.</p>\n<p>Tokens matter here for four reasons. Context windows count tokens. Providers price input and output by the token, and Anthropic's documentation lists those prices for each model. Decode makes one pass per output token, so output length sets the work. And each token adds a fixed slice to the KV cache.</p>","headings":[{"id":"tokens","title":"Tokens","depth":3}]},{"id":"from-prompt-to-answer","title":"From prompt to answer","html":"<h3 id=\"from-prompt-to-answer\">From prompt to answer</h3>\n<p>A chat request is a list of role-tagged messages. The server renders that list into one token sequence. A chat template adds control tokens that mark where each message starts and ends. Hugging Face's guide says the messages become \"a token sequence, often with control tokens,\" which let the model see the chat structure.</p>\n<p>The model then runs in three stages.</p>\n<ol>\n<li><strong>Prefill.</strong> The server runs the model once over every prompt token. This builds the key-value (KV) cache for the prompt and produces scores for the first answer token.</li>\n<li><strong>Decode.</strong> The server picks one token from the scores, appends it, runs the model on that one new token using the cache, and repeats until a stop condition is met.</li>\n<li><strong>Detokenize.</strong> The server turns the output IDs back into text and returns or streams them.</li>\n</ol>\n<figure class=\"diagram\" data-text=\"Tokenizer -> Prefill all prompt tokens (token IDs); Prefill all prompt tokens -> Decode one token per pass (scores for token 1); Decode one token per pass -> Detokenize and stream (each token); Prefill all prompt tokens -> KV cache (write); KV cache <-> Decode one token per pass (read, add)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -10 640 333\" width=\"640\" role=\"img\" aria-label=\"One request on the server. Prefill runs once, then decode repeats.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"249.7\" y1=\"71\" x2=\"308.7\" y2=\"248\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"331.3\" y1=\"248\" x2=\"389.7\" y2=\"73\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">Tokenizer</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Prefill</tspan><tspan x=\"240\" y=\"54.5\">all prompt tokens</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Decode</tspan><tspan x=\"400\" y=\"54.5\">one token per pass</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Detokenize</tspan><tspan x=\"560\" y=\"54.5\">and stream</tspan></text><path class=\"dg-node dg-store\" d=\"M252,260 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M252,260 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"289.5\">KV cache</tspan></text><rect class=\"dg-edge-bg\" x=\"126.1\" y=\"-6\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"7\" text-anchor=\"middle\">token IDs</text><rect class=\"dg-edge-bg\" x=\"258.2\" y=\"-6\" width=\"121.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"7\" text-anchor=\"middle\">scores for token 1</text><rect class=\"dg-edge-bg\" x=\"443\" y=\"-6\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"7\" text-anchor=\"middle\">each token</text><rect class=\"dg-edge-bg\" x=\"258.7\" y=\"150.5\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"279.2\" y=\"163.5\" text-anchor=\"middle\">write</text><rect class=\"dg-edge-bg\" x=\"327.6\" y=\"151.5\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"360.5\" y=\"164.5\" text-anchor=\"middle\">read, add</text></svg></div><figcaption>One request on the server. Prefill runs once, then decode repeats.</figcaption></figure>\n<h4 id=\"the-transformer-at-a-glance\">The transformer at a glance</h4>\n<p>Llama 3 uses what its paper calls \"a standard, dense Transformer architecture,\" and Meta describes it as a decoder-only model. Decoder-only means each position attends only to earlier tokens, which the Transformer paper enforces with a mask on the decoder's self-attention.</p>\n<p>The 8B model has 32 layers, the 70B has 80 and the 405B has 126. Each layer has an attention step, where a token gathers information from the tokens before it, and a feed-forward network, which transforms each token's vector on its own. In \"The trophy did not fit in the suitcase because it was too big,\" the word \"it\" refers back to \"trophy,\" and attention is the step that lets the model make that link. Attention also drives the cost of long prompts: each token compares itself with every earlier token. The Transformer paper's table gives self-attention cost per layer as O(n²·d), where n is the sequence length, so doubling the prompt roughly quadruples that part of the work.</p>\n<h4 id=\"when-generation-stops\">When generation stops</h4>\n<p>Generation ends when the model emits its end-of-sequence token, a special token that marks the end of a reply. It also ends at the output limit (max_new_tokens in Hugging Face, which counts \"ignoring the number of tokens in the prompt\"), at a custom stop string, or when the context window fills. Anthropic's API reports which one happened. end_turn means Claude \"finished its response naturally,\" and max_tokens means it \"reached the max_tokens limit specified in your request.\"</p>","headings":[{"id":"from-prompt-to-answer","title":"From prompt to answer","depth":3},{"id":"the-transformer-at-a-glance","title":"The transformer at a glance","depth":4},{"id":"when-generation-stops","title":"When generation stops","depth":4}]},{"id":"prefill-and-decode","title":"Prefill and decode","html":"<h3 id=\"prefill-and-decode\">Prefill and decode</h3>\n<p>Take a 2,000-token prompt. The server can read all 2,000 tokens in one pass, but writing a 300-token reply takes 300 passes. Prefill is the first pass, over the whole prompt at once. The prompt is known in advance, so the GPU can process all of its tokens together as large matrix operations. Splitwise, a paper about running the two phases on separate machines, calls the prompt phase \"a compute-intensive prompt computation.\" Decode produces the answer one token per pass. Each pass processes only the newest token, but it must read all the weights and the cached keys and values from GPU memory. Splitwise calls this \"a memory-intensive token generation.\" Pope and colleagues put it in hardware terms: each step loads the parameters and KV cache from high-bandwidth memory (HBM) into the compute cores, and at small batch sizes \"the time to load weights dominates.\"</p>\n<p>The running example puts numbers on this. It uses a 70B model with BF16 weights (16-bit numbers, 2 bytes each) split across four NVIDIA H100 GPUs with tensor parallelism, which divides each layer's weights among the GPUs. Hugging Face's optimization guide gives the rule of thumb of about 2 GB per billion parameters in bfloat16, so the weights take 140 GB. A FLOP is one floating-point operation. The EleutherAI guide gives the cost of a forward pass as about 2PD FLOPs, where P is the parameter count and D the token count. So one token costs about 2 × 70 × 10^9 = 140 GFLOP.</p>\n<ul>\n<li><strong>Prefill of 2,000 tokens.</strong> The work is 2,000 × 140 GFLOP = 280 TFLOP. NVIDIA's H100 page lists 1,979 TFLOPS for BF16 with sparsity, and NVIDIA's Hopper architecture post explains that sparsity doubles the rate, so the dense figure is about 989 TFLOPS. Four GPUs give 4 × 989 = 3,956 TFLOPS, so prefill takes 280 ÷ 3,956 ≈ 71 ms at full speed, or about 142 ms at an assumed 50% of peak.</li>\n<li><strong>Decode for one user.</strong> Each step reads all 140 GB. Four H100s at 3.35 TB/s each give 13.4 TB/s, so a step takes at least 140 ÷ 13,400 ≈ 10.4 ms. That caps one user at about 96 tokens a second (1 ÷ 0.01045 ≈ 95.7). The step's arithmetic, 140 GFLOP, would take about 0.04 ms at full speed, so the step waits almost entirely on memory.</li>\n</ul>\n<p>Prefill reads each weight once and applies it to every prompt token, so it does a lot of arithmetic per byte read. Decode reads every weight to produce one token. The GPU chapter explains this with the roofline model, which compares a chip's arithmetic speed with its memory speed. A longer prompt takes longer to prefill, so long prompts raise TTFT.</p>","headings":[{"id":"prefill-and-decode","title":"Prefill and decode","depth":3}]},{"id":"the-kv-cache","title":"The KV cache","html":"<h3 id=\"the-kv-cache\">The KV cache</h3>\n<p>Consider one 4,096-token conversation on the 70B model. Its cache takes about 1.34 GB (4,096 × 327,680 bytes, computed below), next to 140 GB of weights.</p>\n<p>Each attention step needs the keys and values of every earlier token in every layer. Without a cache, each decode step would recompute them for the whole sequence, \"even though the earlier tokens haven't changed,\" as Hugging Face's KV cache guide puts it. With a cache, each token's keys and values are computed once, stored, and appended to. Hugging Face calls this a trade-off between speed and memory.</p>\n<p>The cache size per token follows from the model's shape:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>bytes per token = 2 × layers × KV heads × head dimension × bytes per value</span></span></code></pre>\n<p>The leading 2 covers keys and values. The head dimension is the model dimension divided by the number of attention heads, so the 70B model has 8,192 ÷ 64 = 128. Llama 3 uses grouped-query attention (GQA) with 8 key-value heads at every size, so several query heads share each key-value head. The GQA paper describes the key-value heads as \"more than one, less than number of query heads.\" The vLLM paper runs a similar calculation for OPT-13B, a 13-billion-parameter model, and reports 800 KB per token in FP16, which matches 2 × 5,120 × 40 × 2 = 819,200 bytes.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Model</th><th>Layers</th><th>KV heads</th><th>Head dimension</th><th>Bytes per token, BF16</th></tr></thead><tbody><tr><td>Llama 3.1 8B</td><td>32</td><td>8</td><td>128</td><td>2 × 32 × 8 × 128 × 2 = 131,072</td></tr><tr><td>Llama 3.1 70B</td><td>80</td><td>8</td><td>128</td><td>2 × 80 × 8 × 128 × 2 = 327,680</td></tr><tr><td>Llama 3.1 405B</td><td>126</td><td>8</td><td>128</td><td>2 × 126 × 8 × 128 × 2 = 516,096</td></tr></tbody></table></div>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Bytes of KV cache per token: keys and values (the 2), for every layer and KV head.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LLAMA_3_1</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"8B\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">layers</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">32</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">kv_heads</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">head_dim</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">128</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"70B\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">layers</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">80</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">kv_heads</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">head_dim</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">128</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"405B\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">layers</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">126</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">kv_heads</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">head_dim</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">128</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> kv_bytes_per_token</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(layers: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, kv_heads: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, head_dim: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                       bytes_per_value: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> layers </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> kv_heads </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> head_dim </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> bytes_per_value</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> name, shape </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> LLAMA_3_1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.items():</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">name</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">kv_bytes_per_token(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">**</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">shape)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:,</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> bytes per token\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 8B: 131,072 bytes per token</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 70B: 327,680 bytes per token</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 405B: 516,096 bytes per token</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">per_token </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> kv_bytes_per_token(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">**</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LLAMA_3_1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"70B\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"70B, one 4,096-token conversation: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{4096</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> per_token </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GB\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 70B, one 4,096-token conversation: 1.34 GB</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">cache_budget </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 148</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 10</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">**</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">9</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # bytes left after the weights, in the running example</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"70B cache budget holds </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">cache_budget </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">//</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> per_token</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:,</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 70B cache budget holds 451,660 tokens</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Without GQA (64 KV heads): </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">kv_bytes_per_token(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">80</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">64</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">128</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:,</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> bytes per token\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Without GQA (64 KV heads): 2,621,440 bytes per token</span></span></code></pre>\n<p>One 4,096-token conversation takes 4,096 × 327,680 = 1,342,177,280 bytes, about 1.34 GB. Without GQA, the 70B model would keep 64 key-value heads, and the same formula gives 2,621,440 bytes per token, eight times as much.</p>\n<p>In the running example, four 80 GB H100s hold 320 GB. Keeping an assumed 10% (32 GB) free for activations and runtime, and loading 140 GB of weights, leaves 148 GB for the cache. That holds 148 × 10^9 ÷ 327,680 ≈ 451,660 tokens, about 110 conversations of 4,096 tokens (451,660 ÷ 4,096 ≈ 110).</p>\n<p>The cache grows with every token, and each request has its own. The vLLM paper observes that it \"grows and shrinks dynamically\" and that memory \"can be significantly wasted by fragmentation and redundant duplication.\" The <a href=\"/learn/inference-engines#managing-kv-cache-memory\">managing KV cache memory</a> and <a href=\"/learn/inference-engines#prefix-caching\">prefix caching</a> sections of the inference engines chapter cover how engines cope.</p>","headings":[{"id":"the-kv-cache","title":"The KV cache","depth":3}]},{"id":"sampling","title":"Sampling","html":"<h3 id=\"sampling\">Sampling</h3>\n<p>Suppose the model has just read \"The cat sat on the\" and gives \"mat\" a score 2.0 higher than each of seven other entries. Sampling is the rule that picks the next token from those scores. Four rules matter.</p>\n<ul>\n<li><strong>Greedy decoding</strong> picks the most likely token each time. Hugging Face calls it the default, and warns that it \"breaks down when generating longer sequences because it begins to repeat itself.\"</li>\n<li><strong>Sampling</strong> draws the token at random, weighted by its probability.</li>\n<li><strong>Temperature</strong> divides each score by the temperature before the softmax. The transformers library does this with <code>scores / self.temperature</code>. Values below 1 sharpen the distribution, and 0 is \"equivalent to shifting all probability mass to the most likely token,\" which is greedy decoding. Temperature only matters when sampling is on.</li>\n<li><strong>Top-k and top-p</strong> cut the candidates first. Top-k keeps the K most likely tokens. Top-p keeps \"the smallest possible set of words whose cumulative probability exceeds the probability p.\" The kept probabilities are then rescaled to sum to one.</li>\n</ul>\n<p>The script below uses a fake model that scores the right next token 2.0 and every other token 0.0. It shows greedy decoding, sampling at temperature 1, and both stop conditions.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> math</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> random</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">VOCAB</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"The\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"sat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"on\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"the\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"mat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\".\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"&#x3C;eos>\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">NEXT</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"The\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"sat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"sat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"on\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"on\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"the\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"the\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"mat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"mat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\".\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\".\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"&#x3C;eos>\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> forward</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(tokens, kv_cache):  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># fake model: caches each token, scores the next one</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    kv_cache.extend(tokens)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># a real engine stores keys and values per layer</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    scores </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">VOCAB</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    scores[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">VOCAB</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.index(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">NEXT</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[tokens[</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]])] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2.0</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # the right next token scores highest</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> scores</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sample</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(scores, temperature, rng):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> temperature </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> VOCAB</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[scores.index(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(scores))]  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># greedy</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    weights </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [math.exp(s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> temperature) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> scores]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> rng.choices(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">VOCAB</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">weights</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">weights)[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> generate</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(prompt, max_new_tokens, temperature):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    rng, kv_cache, output </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> random.Random(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), [], []</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    scores </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> forward(prompt, kv_cache)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># prefill: the whole prompt in one pass</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    while</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(output) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_new_tokens:  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># decode: one token per pass</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        token </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sample(scores, temperature, rng)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        output.append(token)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> token </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"&#x3C;eos>\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># stop condition 1: end-of-sequence token</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> output, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"end-of-sequence\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(kv_cache)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        scores </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> forward([token], kv_cache)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> output, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"max_new_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(kv_cache)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># stop condition 2: the limit</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> limit, temp </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)]:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    output, reason, cache </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> generate([</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"The\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], limit, temp)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\" \"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(output), </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"|\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, reason, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"| cache:\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, cache)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># cat sat on the mat . &#x3C;eos> | end-of-sequence | cache: 7</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># cat sat on | max_new_tokens | cache: 4</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># mat . . on the mat . the mat . | max_new_tokens | cache: 11</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> t </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> math.exp(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> t) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (math.exp(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> t) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">VOCAB</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"temperature </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">t</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: right token </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">p</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># temperature 0.5: right token 0.89</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># temperature 1.0: right token 0.51</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># temperature 2.0: right token 0.28</span></span></code></pre>\n<p>Greedy decoding writes the whole sentence and stops at the end-of-sequence token, with seven cache entries, one for each token read. A limit of three stops on the limit with four entries, because the newest token has not run through the model yet. At temperature 1 the fake model wanders, and the right token's probability is 0.51. It is 0.89 at temperature 0.5 and 0.28 at 2.0.</p>\n<p>For extraction or classification, where one answer is right, use greedy decoding or a low temperature. For brainstorming, use a higher temperature with top-p.</p>","headings":[{"id":"sampling","title":"Sampling","depth":3}]},{"id":"streaming-the-answer","title":"Streaming the answer","html":"<h3 id=\"streaming-the-answer\">Streaming the answer</h3>\n<p>Here is one streamed reply from Anthropic's API, abridged with the ping events removed. Most events carry one piece of the answer.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>event: message_start</span></span>\n<span class=\"line\"><span>data: {\"type\": \"message_start\", \"message\": {\"id\": \"msg_...\", \"role\": \"assistant\", \"content\": [], ...}}</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>event: content_block_start</span></span>\n<span class=\"line\"><span>data: {\"type\": \"content_block_start\", \"index\": 0, \"content_block\": {\"type\": \"text\", \"text\": \"\"}}</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>event: content_block_delta</span></span>\n<span class=\"line\"><span>data: {\"type\": \"content_block_delta\", \"index\": 0, \"delta\": {\"type\": \"text_delta\", \"text\": \"Hello\"}}</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>event: content_block_delta</span></span>\n<span class=\"line\"><span>data: {\"type\": \"content_block_delta\", \"index\": 0, \"delta\": {\"type\": \"text_delta\", \"text\": \"!\"}}</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>event: content_block_stop</span></span>\n<span class=\"line\"><span>data: {\"type\": \"content_block_stop\", \"index\": 0}</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>event: message_delta</span></span>\n<span class=\"line\"><span>data: {\"type\": \"message_delta\", \"delta\": {\"stop_reason\": \"end_turn\", \"stop_sequence\": null}, ...}</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>event: message_stop</span></span>\n<span class=\"line\"><span>data: {\"type\": \"message_stop\"}</span></span></code></pre>\n<p>Streaming sends each token as soon as it is decoded, usually as server-sent events (SSE). Anthropic's API uses SSE when a request sets <code>\"stream\": true</code>. MDN describes the format: the content type is text/event-stream, a blank line ends each event, and lines starting with <code>data:</code> carry its payload. The connection runs one way, from server to client.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sse_payloads</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(lines):  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># yields the data field of each event; a blank line ends an event</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    data </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> []</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> line </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> lines:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        line </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> line.rstrip(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> line </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> data:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                yield</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(data)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            data </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> []</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        elif</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> line.startswith(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"data:\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># event: and comment lines are ignored</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            value </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> line[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            data.append(value[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> value.startswith(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\" \"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> value)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> fake_server</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">():  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># stand-in for an HTTP body: one event per token, then a stop event</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> piece </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Hello\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\" there\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"!\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        event </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"content_block_delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text_delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: piece}}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        yield</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"event: content_block_delta\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        yield</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"data: \"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json.dumps(event)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        yield</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    yield</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"event: message_stop\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    yield</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> 'data: {\"type\": \"message_stop\"}'</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    yield</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">text </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> payload </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sse_payloads(fake_server()):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    event </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json.loads(payload)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> event[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"content_block_delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        text </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> event[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">][</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"token: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">event[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'delta'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">][</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'text'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">!r</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    elif</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> event[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"message_stop\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"stream finished:\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">repr</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(text))</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># token: 'Hello'</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># token: ' there'</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># token: '!'</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># stream finished: 'Hello there!'</span></span></code></pre>\n<p>Streaming changes when the first token appears. The full answer still takes as long as before, because every decode step still runs. NVIDIA's documentation says time to first token \"measures how long you wait before seeing the model's output.\"</p>\n<p>Each hop must forward events as they arrive. A gateway or proxy that buffers the response hides the stream, and the user waits for the whole reply.</p>\n<figure class=\"diagram\" data-text=\"Client <-> Gateway or proxy (request, then events); Gateway or proxy <-> Inference server (prompt, then tokens)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -8 480 86\" width=\"480\" role=\"img\" aria-label=\"Each hop must forward events as they arrive.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"153\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"313\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">Client</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"46.5\">Gateway or proxy</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"46.5\">Inference server</tspan></text><rect class=\"dg-edge-bg\" x=\"93\" y=\"-4\" width=\"134\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"160\" y=\"9\" text-anchor=\"middle\">request, then events</text><rect class=\"dg-edge-bg\" x=\"256.1\" y=\"-4\" width=\"127.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"9\" text-anchor=\"middle\">prompt, then tokens</text></svg></div><figcaption>Each hop must forward events as they arrive.</figcaption></figure>\n<p>When the client disconnects, the server should stop decoding, or the GPU keeps producing tokens nobody reads. The <a href=\"/learn/realtime\">realtime</a> and <a href=\"/learn/protocols-and-apis\">protocols and APIs</a> chapters cover SSE and other transports.</p>","headings":[{"id":"streaming-the-answer","title":"Streaming the answer","depth":3}]},{"id":"latency-and-throughput-metrics","title":"Latency and throughput metrics","html":"<h3 id=\"latency-and-throughput-metrics\">Latency and throughput metrics</h3>\n<p>Picture one response. The request waits in a queue, prefill runs, the first token appears, and then the rest arrive one at a time.</p>\n<figure class=\"diagram\" data-text=\"Request sent -> Queue wait; Queue wait -> Prefill of the prompt; Prefill of the prompt -> First token returned; First token returned -> Decode, one TPOT per token (rest of the answer)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -10 800 90\" width=\"800\" role=\"img\" aria-label=\"One response. TTFT covers everything before the first token.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">Request sent</tspan></text><rect class=\"dg-node dg-queue\" x=\"172\" y=\"18\" width=\"136\" height=\"48\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"292\" y1=\"26\" x2=\"292\" y2=\"58\"></line><line class=\"dg-line\" x1=\"282\" y1=\"26\" x2=\"282\" y2=\"58\"></line><line class=\"dg-line\" x1=\"272\" y1=\"26\" x2=\"272\" y2=\"58\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"226\" y=\"46.5\">Queue wait</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Prefill of</tspan><tspan x=\"400\" y=\"54.5\">the prompt</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">First token</tspan><tspan x=\"560\" y=\"54.5\">returned</tspan></text><rect class=\"dg-node dg-box\" x=\"652\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"38.5\">Decode, one TPOT</tspan><tspan x=\"720\" y=\"54.5\">per token</tspan></text><rect class=\"dg-edge-bg\" x=\"578.2\" y=\"-6\" width=\"121.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"639\" y=\"7\" text-anchor=\"middle\">rest of the answer</text></svg></div><figcaption>One response. TTFT covers everything before the first token.</figcaption></figure>\n<p>Two numbers describe that timeline. NVIDIA's benchmarking documentation, as of October 2026, defines them and two related measures:</p>\n<ul>\n<li><strong>Time to first token (TTFT)</strong> is \"the time from query submission to the first received token, if the response is not empty.\" It generally includes queueing, prefill and network time.</li>\n<li><strong>Time per output token (TPOT)</strong>, also called inter-token latency, is \"the average time between consecutive tokens,\" not counting the first. It equals (end-to-end latency − TTFT) ÷ (output tokens − 1).</li>\n<li><strong>End-to-end latency</strong> is the time from query to full response. It equals TTFT plus the generation time.</li>\n<li><strong>Throughput</strong> is total output tokens per second across all simultaneous requests.</li>\n</ul>\n<p>Other tools define these differently, and NVIDIA notes that results should be compared \"only when definitions align.\" With these definitions, end-to-end latency ≈ TTFT + (output tokens − 1) × TPOT. Assume a TTFT of 200 ms and a TPOT of 13.6 ms, the running example's step time below. A 300-token answer takes 200 + 299 × 13.6 = 4,266.4 ms, about 4.3 seconds.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Situation</th><th>Time per step</th><th>Tokens a second, total</th><th>Per user</th></tr></thead><tbody><tr><td>One user</td><td>140 GB ÷ 13.4 TB/s ≈ 10.4 ms</td><td>about 96</td><td>about 96</td></tr><tr><td>64 users, 2,000 tokens each</td><td>(140 + 42) GB ÷ 13.4 TB/s ≈ 13.6 ms</td><td>about 4,700</td><td>about 74</td></tr></tbody></table></div>\n<p>The second row's cache reads come to 64 × 2,000 × 327,680 bytes ≈ 42 GB. Each step makes one token per user, so the server makes 64 ÷ 0.0136 ≈ 4,706 tokens a second, or about 74 for each user. The step's arithmetic, 64 × 140 GFLOP ≈ 9 TFLOP, takes about 2.3 ms at full speed, so the step still waits on memory.</p>\n<p>Batching raises total throughput about 49 times (4,706 ÷ 96 ≈ 49) and cuts each user's speed by about 23%. The weights are read once per step however many users share it, so each extra user adds mainly its own cache.</p>\n<p>Throughput sets cost. Assume $3.00 per H100-hour, so four GPUs cost $12 an hour, or $0.00333 a second. That is about $34.83 per million tokens at 95.7 tokens a second, and about $0.71 at 4,706. Real prices vary by provider and commitment, and these figures ignore idle time and engine overhead. The <a href=\"/learn/ai-estimation#cost-per-million-tokens\">cost per million tokens</a> section does this properly.</p>\n<p>Averages hide slow requests (<a href=\"/learn/performance-and-latency#averages-hide-the-slow-requests\">performance and latency</a>), so dashboards track percentiles of TTFT and TPOT, such as the 99th percentile, the value that 99% of requests beat. Under load, TTFT rises first, because requests wait in a queue before prefill starts.</p>","headings":[{"id":"latency-and-throughput-metrics","title":"Latency and throughput metrics","depth":3}]},{"id":"context-windows","title":"Context windows","html":"<h3 id=\"context-windows\">Context windows</h3>\n<p>Suppose a model's window is 8,192 tokens and a conversation already uses 8,000. The reply can be at most 192 tokens, because the prompt and the reply share one budget. That window is hypothetical. Anthropic's docs define the context window as \"all the text a language model can reference when generating a response, including the response itself.\"</p>\n<p>Llama 3.1 has a far larger window. Meta says the 8B and 70B models have \"a significantly longer context length of 128K,\" and the Llama 3 paper says its largest model has \"a context window of up to 128K tokens.\" Taking 128K as 131,072 tokens, a full window limits the 70B model in three ways.</p>\n<ol>\n<li><strong>Memory.</strong> One full conversation needs 131,072 × 327,680 bytes ≈ 42.9 GB of KV cache. The 148 GB budget holds three (148 ÷ 42.9 ≈ 3.4).</li>\n<li><strong>Compute.</strong> Prefilling a full window takes 131,072 × 140 GFLOP ≈ 18,350 TFLOP. At 50% of four H100s' peak that is 18,350 ÷ 1,978 ≈ 9.3 seconds, before the attention work, which grows faster than the length.</li>\n<li><strong>Quality.</strong> Anthropic's docs warn that \"as token count grows, accuracy and recall degrade, a phenomenon known as context rot.\" That is the vendor's own description, so test it on your workload.</li>\n</ol>\n<p>Chat multiplies the cost, because each message resends the whole history. Hugging Face's optimization guide says the cache lets a chat \"continuously grow the encoded chat history instead of having to re-encode the chat history again from scratch.\" Without that reuse, every turn repeats prefill over all earlier turns.</p>\n<p>At the limit, Anthropic's docs say that an input which alone exceeds the window fails with an error. On newer models, an input plus max_tokens that exceeds the window is accepted, and generation stops at the window with stop reason model_context_window_exceeded. Code should check the budget before sending.</p>","headings":[{"id":"context-windows","title":"Context windows","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>The team serves the 70B model in BF16 with 148 GB of cache. Typical conversations are 8,000 tokens. About how many fit at once?</summary>\n<p>One conversation takes 8,000 × 327,680 bytes ≈ 2.6 GB, so 148 ÷ 2.62 ≈ 56 fit. Doubling the length halves the count, because the cache grows linearly with the number of tokens.</p>\n</details>\n<details>\n<summary>A user waits about 6 seconds before the first word of an answer about a 50,000-token document, then tokens stream steadily. Which metric is high, and what do you check first?</summary>\n<p>TTFT is high, and TPOT is normal. Prefill dominates: 50,000 × 140 GFLOP = 7,000 TFLOP, about 3.5 seconds at 50% of the four-H100 peak. Check the queue depth first, then whether many requests share this document, since <a href=\"/learn/inference-engines#prefix-caching\">prefix caching</a> would skip most of that prefill.</p>\n</details>\n<details>\n<summary>You ship invoice extraction as JSON and bakery name suggestions on the same model. Which decoding settings do you use for each?</summary>\n<p>Use greedy decoding or a low temperature for extraction, since one answer is right and low temperature keeps the output stable. For names, sample at a higher temperature with top-p, since variety is the goal. Temperature only matters when sampling is on.</p>\n</details>\n<details>\n<summary>A 405B model in BF16 must run with tensor parallelism. Does it fit on 8 H100s, and what is the one-user decode ceiling on 16?</summary>\n<p>The weights are 405 × 10^9 × 2 bytes = 810 GB, more than 8 × 80 = 640 GB, so it does not fit. On 16 H100s, each step reads 810 GB at 16 × 3.35 = 53.6 TB/s, about 15.1 ms, so the ceiling is about 66 tokens a second (1 ÷ 0.0151 ≈ 66).</p>\n</details>\n<details>\n<summary>The server streams tokens, but the user sees nothing for 30 seconds and then the whole reply. Where do you look?</summary>\n<p>Compare when the first event leaves the inference server with when the client receives it. If the gap is at a gateway or proxy, that hop is buffering the response. Configure it to forward each event as it arrives.</p>\n</details>\n<details>\n<summary>A request carries 7,900 tokens of history, the window is 8,192 tokens, and max_tokens is 1,000. What happens?</summary>\n<p>The request needs 7,900 + 1,000 = 8,900 tokens, so only 8,192 − 7,900 = 292 tokens fit for the reply. Depending on the API, the request is rejected, or it is accepted and stops at the window with stop reason model_context_window_exceeded. Check the budget before sending.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://arxiv.org/html/2407.21783\" target=\"_blank\" rel=\"noopener noreferrer\">The Llama 3 Herd of Models</a>: the architecture table, the 128K vocabulary and grouped-query attention.</li>\n<li><a href=\"https://ai.meta.com/blog/meta-llama-3-1/\" target=\"_blank\" rel=\"noopener noreferrer\">Meta, Introducing Llama 3.1</a>: the 128K context for the 8B and 70B models, and the decoder-only design.</li>\n<li><a href=\"https://arxiv.org/html/1706.03762\" target=\"_blank\" rel=\"noopener noreferrer\">Vaswani et al., Attention Is All You Need</a>: the Transformer, its self-attention cost, and decoder masking.</li>\n<li><a href=\"https://arxiv.org/abs/1508.07909\" target=\"_blank\" rel=\"noopener noreferrer\">Sennrich et al., Rare Words with Subword Units</a>: byte pair encoding applied to subword units.</li>\n<li><a href=\"https://github.com/openai/tiktoken\" target=\"_blank\" rel=\"noopener noreferrer\">tiktoken</a>: OpenAI's tokenizer library, with its BPE explanation and educational merge code.</li>\n<li><a href=\"https://raw.githubusercontent.com/openai/openai-cookbook/main/examples/How_to_count_tokens_with_tiktoken.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI Cookbook, How to count tokens with tiktoken</a>: the stored outputs behind the token splits above.</li>\n<li><a href=\"https://arxiv.org/abs/2311.18677\" target=\"_blank\" rel=\"noopener noreferrer\">Patel et al., Splitwise</a>: the compute-intensive prompt phase and the memory-intensive token phase.</li>\n<li><a href=\"https://arxiv.org/html/2211.05102\" target=\"_blank\" rel=\"noopener noreferrer\">Pope et al., Efficiently Scaling Transformer Inference</a>: why each decode step loads weights and the KV cache from HBM.</li>\n<li><a href=\"https://arxiv.org/html/2309.06180\" target=\"_blank\" rel=\"noopener noreferrer\">Kwon et al., PagedAttention</a>: the 800 KB per-token cache example for OPT-13B, and how cache memory is wasted.</li>\n<li><a href=\"https://arxiv.org/abs/2305.13245\" target=\"_blank\" rel=\"noopener noreferrer\">Ainslie et al., GQA</a>: grouped-query attention, with fewer key-value heads than query heads.</li>\n<li><a href=\"https://blog.eleuther.ai/transformer-math/\" target=\"_blank\" rel=\"noopener noreferrer\">EleutherAI, transformer math</a>: about 2PD FLOPs for a forward pass.</li>\n<li><a href=\"https://huggingface.co/docs/transformers/en/llm_tutorial_optimization\" target=\"_blank\" rel=\"noopener noreferrer\">Hugging Face, Optimizing LLMs for Speed and Memory</a>: the 2 GB per billion parameters rule, and how cache memory grows.</li>\n<li><a href=\"https://huggingface.co/blog/kv-cache\" target=\"_blank\" rel=\"noopener noreferrer\">Hugging Face blog on the KV cache</a>: prefill, decode and the speed and memory trade-off.</li>\n<li><a href=\"https://huggingface.co/blog/how-to-generate\" target=\"_blank\" rel=\"noopener noreferrer\">Hugging Face, How to generate text</a>: greedy search, sampling, top-k and top-p, with examples.</li>\n<li><a href=\"https://huggingface.co/docs/transformers/en/generation_strategies\" target=\"_blank\" rel=\"noopener noreferrer\">Hugging Face, Generation strategies</a>: greedy decoding as the default, and its repetition problem.</li>\n<li><a href=\"https://huggingface.co/docs/transformers/en/internal/generation_utils\" target=\"_blank\" rel=\"noopener noreferrer\">Hugging Face, generation utilities</a>: what temperature does, and what 0 means.</li>\n<li><a href=\"https://raw.githubusercontent.com/huggingface/transformers/main/src/transformers/generation/logits_process.py\" target=\"_blank\" rel=\"noopener noreferrer\">transformers, logits_process.py</a>: the code that divides the scores by the temperature.</li>\n<li><a href=\"https://huggingface.co/docs/transformers/en/main_classes/text_generation\" target=\"_blank\" rel=\"noopener noreferrer\">Hugging Face, text generation parameters</a>: max_new_tokens, eos_token_id and stop_strings.</li>\n<li><a href=\"https://huggingface.co/docs/transformers/en/chat_templating\" target=\"_blank\" rel=\"noopener noreferrer\">Hugging Face, Chat templates</a>: how role-tagged messages become one token sequence.</li>\n<li><a href=\"https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events\" target=\"_blank\" rel=\"noopener noreferrer\">MDN, Using server-sent events</a>: the event stream format.</li>\n<li><a href=\"https://platform.claude.com/docs/en/api/messages-streaming\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic, Streaming Messages</a>: a real event stream.</li>\n<li><a href=\"https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic, Handling stop reasons</a>: what each stop reason means.</li>\n<li><a href=\"https://platform.claude.com/docs/en/build-with-claude/context-windows\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic, Context windows</a>: what the window holds, context rot, and overflow behavior.</li>\n<li><a href=\"https://docs.nvidia.com/nim/benchmarking/llm/metrics.html\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA, NIM LLM benchmarking metrics</a>: the TTFT, TPOT, end-to-end latency and throughput definitions.</li>\n<li><a href=\"https://www.nvidia.com/en-us/data-center/h100/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA, H100 product page</a>: 80 GB of memory, 3.35 TB/s, and 1,979 TFLOPS with sparsity.</li>\n<li><a href=\"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA, Hopper Architecture In-Depth</a>: why the sparsity figure is twice the dense rate.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}