{"id":"rag","title":"Retrieval-augmented generation","sections":[{"id":"overview","title":"Overview","html":"<p>A plain language model answers from what it learned in training. That knowledge is fixed when training ends, the model can't look anything up while it writes, and it doesn't say where a claim came from. Retrieval-augmented generation (RAG) adds a search step before the model writes. A retriever finds passages likely to help with the question, those passages go into the prompt, and the model answers from them and cites them by number.</p>\n<p>The term comes from a 2020 paper by Lewis and colleagues. Their central idea was that a model with a searchable external memory can pick up new facts by changing that memory, without retraining.</p>\n<p>The running example is Halden, an invented payroll software company. Its support agents work from a 380-page internal handbook and 1,200 public support articles. The assistant must answer an agent's question from those documents, cite the paragraphs it used, and never show a policy that the asking agent can't read. Numbers about Halden are assumptions, labeled where they appear.</p>\n<p>Retrieval usually runs on embeddings. An embedding model turns a passage into a vector, a list of numbers, so that passages with similar meaning point in similar directions. A vector index returns the stored vectors nearest a query's vector. A chunk is a short piece of a document, usually a few paragraphs, stored and retrieved as one unit. <a href=\"/learn/vector-search\">Vector search</a> explains how embeddings and approximate nearest neighbor indexes work, so here the index is a black box.</p>","headings":[]},{"id":"why-retrieve","title":"Why retrieve","html":"<h3 id=\"why-retrieve\">Why retrieve</h3>\n<p>The weights hold what the model learned in training, and they are hard to change. The RAG paper names the problem: models \"cannot easily expand or revise their memory, can't straightforwardly provide insight into their predictions, and may produce 'hallucinations.'\" Halden needs private knowledge the model never trained on, answers that follow policies as they change, and a source for every answer.</p>\n<p>Retrieval can improve accuracy, but the evidence is specific to the test setup. The authors reported, qualitatively, that their models \"hallucinate less\" than a BART baseline, a sequence-to-sequence model without retrieval. Their experiments used a December 2018 Wikipedia snapshot, so measure the effect on your own documents.</p>\n<h4 id=\"rag-fine-tuning-and-long-context\">RAG, fine-tuning and long context</h4>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th></th><th>RAG</th><th>Fine-tuning</th><th>Long context</th></tr></thead><tbody><tr><td>How knowledge gets in</td><td>Retrieved into the prompt per question</td><td>Trained into the weights</td><td>Whole documents pasted into the prompt</td></tr><tr><td>Changing a fact</td><td>Re-index the document</td><td>Train again</td><td>Edit the text and resend it every request</td></tr><tr><td>Citations</td><td>Easy: each chunk has an ID</td><td>Hard: no source is stored</td><td>Possible, but the model must find the passage</td></tr><tr><td>Best for</td><td>Changing facts, private documents, citations</td><td>Tone, format, behavior</td><td>Small, stable document sets</td></tr></tbody></table></div>\n<p>Fine-tuning is a weak way to add facts. Ovadia and colleagues found that \"while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge.\"</p>\n<p>Long context works when the whole corpus fits and traffic is low. Every prompt token is processed on every request, and the model keeps a key and a value for each one for the request's whole life (the KV cache, which <a href=\"/learn/llm-inference#the-kv-cache\">LLM inference</a> explains). Models also use the middle of a long prompt less reliably. Halden's documents hold about 910,000 words, or roughly 1.2 million tokens at an assumed 1.3 tokens per word. That is about nine times the 128,000-token window that the Llama 3 paper describes.</p>\n<figure class=\"diagram\" data-text=\"Agent question -> Retriever keyword + vector; Retriever keyword + vector <-> Index chunks and vectors (search); Retriever keyword + vector -> Prompt sources + question (top chunks); Prompt sources + question -> Language model; Language model -> Answer with citations [1] [2]\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -10 800 215\" width=\"800\" role=\"img\" aria-label=\"Retrieval adds one step before the model writes\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"240\" y1=\"73\" x2=\"240\" y2=\"126\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">Agent</tspan><tspan x=\"80\" y=\"54.5\">question</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Retriever</tspan><tspan x=\"240\" y=\"54.5\">keyword + vector</tspan></text><path class=\"dg-node dg-store\" d=\"M172,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M172,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"161.5\">Index</tspan><tspan x=\"240\" y=\"177.5\">chunks and vectors</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Prompt</tspan><tspan x=\"400\" y=\"54.5\">sources + question</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"46.5\">Language model</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"16\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"38.5\">Answer with</tspan><tspan x=\"720\" y=\"54.5\">citations [1] [2]</tspan></text><rect class=\"dg-edge-bg\" x=\"216.4\" y=\"90.5\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"103.5\" text-anchor=\"middle\">search</text><rect class=\"dg-edge-bg\" x=\"283\" y=\"-6\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"7\" text-anchor=\"middle\">top chunks</text></svg></div><figcaption>Retrieval adds one step before the model writes</figcaption></figure>","headings":[{"id":"why-retrieve","title":"Why retrieve","depth":3},{"id":"rag-fine-tuning-and-long-context","title":"RAG, fine-tuning and long context","depth":4}]},{"id":"the-indexing-pipeline","title":"The indexing pipeline","html":"<h3 id=\"the-indexing-pipeline\">The indexing pipeline</h3>\n<p>The indexing pipeline runs whenever a document is added or changed, ahead of any question, so the question path stays fast.</p>\n<p>Parsing converts each file to text and keeps its structure: headings, lists and tables. Parsers fail quietly. A two-column PDF can come out in the wrong order, and a page can vanish without an error. Compare the pages going in with the characters coming out, and read a sample by hand.</p>\n<p>The pipeline then chunks the text, attaches metadata, embeds each chunk and writes three things: vectors to a vector index, words to a keyword index that supports BM25 scoring (a standard keyword-search method), and chunk text and metadata to a chunk store keyed by chunk ID. The prompt is built from the chunk store, so the source is never reread at question time.</p>\n<figure class=\"diagram\" data-text=\"Sources handbook, articles -> Parse text and structure; Parse text and structure -> Chunk and tag metadata, version; Chunk and tag metadata, version -> Embedding model; Embedding model -> Vector index (vectors); Chunk and tag metadata, version -> Chunk store text, metadata (keep text); Chunk and tag metadata, version -> Keyword index BM25 (words)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -13 800 218\" width=\"800\" role=\"img\" aria-label=\"The indexing pipeline runs when documents change, ahead of questions\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"400\" y1=\"71\" x2=\"400\" y2=\"126\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"438.7\" y1=\"71\" x2=\"512\" y2=\"126\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">Sources</tspan><tspan x=\"80\" y=\"54.5\">handbook, articles</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Parse</tspan><tspan x=\"240\" y=\"54.5\">text and structure</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Chunk and tag</tspan><tspan x=\"400\" y=\"54.5\">metadata, version</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Embedding</tspan><tspan x=\"560\" y=\"54.5\">model</tspan></text><path class=\"dg-node dg-store\" d=\"M652,20 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,20 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"49.5\">Vector index</tspan></text><path class=\"dg-node dg-store\" d=\"M332,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"161.5\">Chunk store</tspan><tspan x=\"400\" y=\"177.5\">text, metadata</tspan></text><path class=\"dg-node dg-store\" d=\"M492,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"161.5\">Keyword index</tspan><tspan x=\"560\" y=\"177.5\">BM25</tspan></text><rect class=\"dg-edge-bg\" x=\"612.3\" y=\"-9\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"639\" y=\"4\" text-anchor=\"middle\">vectors</text><rect class=\"dg-edge-bg\" x=\"367.1\" y=\"89.5\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"400\" y=\"102.5\" text-anchor=\"middle\">keep text</text><rect class=\"dg-edge-bg\" x=\"454.8\" y=\"89.5\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"475.3\" y=\"102.5\" text-anchor=\"middle\">words</text></svg></div><figcaption>The indexing pipeline runs when documents change, ahead of questions</figcaption></figure>\n<p>Halden's sizes, as assumptions: 380 pages of about 500 words and 1,200 articles of about 600 words make 910,000 words. With 300-word chunks that overlap their neighbors by 50 words, each chunk adds 250 new words, giving 910,000 ÷ 250 = 3,640 chunks. At 1,024 dimensions and 4 bytes per number, the vectors take 3,640 × 1,024 × 4 = 14,909,440 bytes, about 14.9 MB. That fits in one server's memory many times over, so for Halden storage is easy and retrieval quality is the hard part.</p>","headings":[{"id":"the-indexing-pipeline","title":"The indexing pipeline","depth":3}]},{"id":"chunking","title":"Chunking","html":"<h3 id=\"chunking\">Chunking</h3>\n<p>A chunk is the unit the system embeds, retrieves and quotes. Its size sets how exactly a search matches a question, how much context each match brings, and how many tokens each retrieved item costs. A token is a piece of text the model reads as one unit, often a word or part of one.</p>\n<p>Small chunks match precisely but lose context: \"This exception applies to the plans above\" means nothing alone. Large chunks keep context, but their vector averages everything in them. The original RAG paper split Wikipedia into disjoint 100-word chunks. No size suits every corpus, so measure recall, as described below, and adjust.</p>\n<p>Split on structure first: headings, then paragraphs, and sentences only when a paragraph is too long. Overlap repeats text across boundaries, so a sentence that falls on a boundary appears whole somewhere, at the cost of extra storage.</p>\n<h4 id=\"chunking-with-overlap\">Chunking with overlap</h4>\n<p>The sketch packs paragraphs into chunks of at most <code>max_words</code>, and each chunk starts with the last <code>overlap</code> units of the one before. A paragraph longer than the limit is split at sentence ends. The sketch counts words to stay dependency-free; production systems count tokens with the embedding model's tokenizer.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> re</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> split_long</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(paragraph: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, max_words: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Split a long paragraph into sentence groups of at most max_words.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    groups, current </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sentence </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> re.split(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">r</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">(?&#x3C;=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">[.!?]</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">)</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\s</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, paragraph):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        candidate </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">current</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> {</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">sentence</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.strip()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> current </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(candidate.split()) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_words:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            groups.append(current)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            candidate </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sentence</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        current </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> candidate</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> groups </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [current]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> chunk_document</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(doc_id: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, text: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, max_words: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 120</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, overlap: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> list[tuple[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]]:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Pack whole paragraphs into chunks. Each new chunk starts with the last</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    `overlap` units of the previous one, so text near a boundary appears twice.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    units </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> []  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># paragraphs, each with the separator that comes before it</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> para </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (p.strip() </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> text.split(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> para:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            pieces </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [para] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(para.split()) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_words </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> split_long(para, max_words)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            units </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> else</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \" \"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> piece </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i, piece </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> enumerate</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(pieces)]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    chunks, current, size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [], [], </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> unit </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> units:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(unit.split())</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> current </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_words:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            chunks.append(current)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            current </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> current[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(current) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> overlap):]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(u.split()) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> u </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> current)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_words:  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the overlap doesn't fit, so start fresh</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                current, size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [], </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        current.append(unit)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        size </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> current:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        chunks.append(current)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">doc_id</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">#</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(c).strip()) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i, c </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> enumerate</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(chunks)]</span></span></code></pre>\n<p>Here is the refund section of Halden's handbook, four short paragraphs:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>Refunds for annual plans are available within 30 days of purchase.</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>Monthly plans cannot be refunded after the first invoice is paid.</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>Exception: agents may approve a refund after 30 days when the outage lasted more than 4 hours.</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>Submit refund requests through the support console and attach the invoice number.</span></span></code></pre>\n<p>With <code>max_words=30</code> and <code>overlap=1</code>, the function returns three chunks. Each line shows a chunk's word count and the first words of each paragraph in it:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>refunds#0 22 words: Refunds for annual plans / Monthly plans cannot be</span></span>\n<span class=\"line\"><span>refunds#1 28 words: Monthly plans cannot be / Exception: agents may approve</span></span>\n<span class=\"line\"><span>refunds#2 29 words: Exception: agents may approve / Submit refund requests through</span></span></code></pre>\n<p>The 30-day rule for annual plans (paragraph 1) and the agent exception (paragraph 3) never share a chunk. A question about the exception can therefore return a chunk that says nothing about which plans it covers. Raising the overlap to two paragraphs wouldn't fix this: the overlap would no longer fit under the 30-word limit, so the function would start each chunk fresh. A 60-word limit keeps the whole section in one chunk. In production, also prepend the heading path to each chunk, and fetch the neighbors of each match.</p>\n<h4 id=\"context-for-each-chunk\">Context for each chunk</h4>\n<p>Anthropic's contextual retrieval has a model write a short description of where each chunk sits in its document, usually 50 to 100 tokens, and prepends it before embedding. On Anthropic's test sets, this cut the fraction of relevant chunks missed in the top 20 from 5.7% to 3.7%. Adding contextual BM25 brought it to 2.9%. The cost is one model call per chunk at indexing time.</p>\n<h4 id=\"metadata\">Metadata</h4>\n<p>Each chunk carries metadata for four jobs: filters (<code>product = payroll-plus</code>), citations (title, section path, URL), freshness (version date, content hash) and permissions (the groups allowed to read it). Decide these fields first, because changing chunk boundaries later means embedding every chunk again.</p>","headings":[{"id":"chunking","title":"Chunking","depth":3},{"id":"chunking-with-overlap","title":"Chunking with overlap","depth":4},{"id":"context-for-each-chunk","title":"Context for each chunk","depth":4},{"id":"metadata","title":"Metadata","depth":4}]},{"id":"retrieving-and-reranking","title":"Retrieving and reranking","html":"<h3 id=\"retrieving-and-reranking\">Retrieving and reranking</h3>\n<p>At question time, the system rewrites the question, searches in two ways, merges the results and reranks the best candidates.</p>\n<figure class=\"diagram\" data-text=\"Rewritten query -> Keyword search top 50 (BM25); Rewritten query -> Vector search top 50; Keyword search top 50 (BM25) -> Merge by rank (RRF); Vector search top 50 -> Merge by rank (RRF); Merge by rank (RRF) -> Cross-encoder reranks 50 to 5; Cross-encoder reranks 50 to 5 -> Top 5 chunks for the prompt\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 320\" width=\"800\" role=\"img\" aria-label=\"One question, two searches, one merge, one rerank\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"118.7\" y1=\"133\" x2=\"198.7\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"118.7\" y1=\"191\" x2=\"198.7\" y2=\"251\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"278.7\" y1=\"71\" x2=\"358.7\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"278.7\" y1=\"253\" x2=\"358.7\" y2=\"193\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"162\" x2=\"647\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"136\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"158.5\">Rewritten</tspan><tspan x=\"80\" y=\"174.5\">query</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Keyword search</tspan><tspan x=\"240\" y=\"54.5\">top 50 (BM25)</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"278.5\">Vector search</tspan><tspan x=\"240\" y=\"294.5\">top 50</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"158.5\">Merge by rank</tspan><tspan x=\"400\" y=\"174.5\">(RRF)</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">Cross-encoder</tspan><tspan x=\"560\" y=\"174.5\">reranks 50 to 5</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"136\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"158.5\">Top 5 chunks</tspan><tspan x=\"720\" y=\"174.5\">for the prompt</tspan></text></svg></div><figcaption>One question, two searches, one merge, one rerank</figcaption></figure>\n<h4 id=\"query-rewriting\">Query rewriting</h4>\n<p>\"What about annual ones?\" means nothing to a search engine. A small model can rewrite the message into a standalone query, such as \"refund window for annual plans,\" using the conversation so far. Each rewrite costs a model call, so keep it only when evaluation shows a gain.</p>\n<h4 id=\"hybrid-search\">Hybrid search</h4>\n<p>Vector search matches meaning, so \"money back\" finds \"refund,\" but it can miss exact strings such as an error code, ERR-4012. Keyword search with BM25 scores each chunk by the query words it contains, weighted by how rare they are across the corpus. Anthropic describes BM25 as a method that \"uses lexical matching to find precise word or phrase matches.\"</p>\n<p>Merge the two lists with reciprocal rank fusion (RRF), which uses ranks and ignores raw scores. A chunk's score is the sum of 1 ÷ (60 + rank) over each list it appears in. Cormack and colleagues proposed RRF in 2009, fixed the constant at 60, and reported that the choice was not critical. A chunk ranked 2nd by BM25 and 5th by vector search scores 1/62 + 1/65 ≈ 0.0315. A chunk ranked 1st only by vector search scores 1/61 ≈ 0.0164. The chunk both searches agree on wins. <a href=\"/learn/vector-search#hybrid-search\">Vector search</a> covers hybrid search in more depth.</p>\n<h4 id=\"permissions-at-retrieval-time\">Permissions at retrieval time</h4>\n<p>Enforce permissions inside each search, before anything reaches the prompt. A model can repeat any paragraph that reaches its prompt, and an instruction not to share it won't reliably stop it. Filtering a finished list afterward is weaker, because restricted chunks use up candidate slots and any search path that skips the filter leaks. OWASP recommends \"fine-grained access controls and permission-aware vector and embedding stores,\" and it warns that embeddings from one group can be \"inadvertently retrieved in response to queries from another group's LLM.\" Key cached answers by permission groups too.</p>\n<h4 id=\"reranking-with-a-cross-encoder\">Reranking with a cross-encoder</h4>\n<p>The first-stage vector search compares precomputed vectors, so the query and the chunk are never read together. A cross-encoder reads the query and one candidate together and outputs a relevance score. Nogueira and Cho used BERT this way, with the query as sentence A and the passage as sentence B, scoring each passage independently. Re-ranking BM25's top 1,000 results raised mean reciprocal rank at 10 (MRR@10, defined below) on MS MARCO's development set from 16.7 to 36.5 for their BERT-large model. The gain came from reordering the same candidates.</p>\n<p>The cost is one model pass per candidate. With 50 candidates of about 400 tokens each, plus the question repeated with each one, the reranker reads about 21,500 tokens per question. Reranking models are much smaller than the model that writes the answer, but this work still happens on every question and adds latency, so rerank a short list. Anthropic kept the top 150 candidates and reranked them down to 20, which cut its missed-chunk rate from 2.9% to 1.9%.</p>","headings":[{"id":"retrieving-and-reranking","title":"Retrieving and reranking","depth":3},{"id":"query-rewriting","title":"Query rewriting","depth":4},{"id":"hybrid-search","title":"Hybrid search","depth":4},{"id":"permissions-at-retrieval-time","title":"Permissions at retrieval time","depth":4},{"id":"reranking-with-a-cross-encoder","title":"Reranking with a cross-encoder","depth":4}]},{"id":"building-the-prompt","title":"Building the prompt","html":"<h3 id=\"building-the-prompt\">Building the prompt</h3>\n<p>The prompt's contents count against the context window, the most tokens the model reads in one request. They also cost time and memory.</p>\n<h4 id=\"the-token-budget\">The token budget</h4>\n<p>Halden's assumed budget: 600 tokens of instructions, 1,000 of conversation history, five chunks at 400 tokens (2,000), a 50-token question and 700 tokens reserved for the answer. The total is 4,350 tokens, far below the 128,000-token window.</p>\n<p>The real limit is memory. Table 3 of the Llama 3 paper gives the 70B model 80 layers, 64 attention heads over a model dimension of 8,192, and 8 key-value heads, so each head is 128-dimensional (8,192 ÷ 64). The cache holds a key and a value per layer, per key-value head, for each token: 2 × 80 × 8 × 128 × 2 = 327,680 bytes per token, using 2 bytes per BF16 number. The 4,350-token prompt needs 4,350 × 327,680 = 1,425,408,000 bytes, about 1.43 GB, for the whole request.</p>\n<p>Take the running deployment: 70B weights in BF16 on four 80 GB H100 GPUs, which give 320 GB. About 10% stays free for runtime work (an assumption), leaving 288 GB. The 140 GB of weights leave 148 GB for cache, which holds 148 × 10^9 ÷ 1,425,408,000 ≈ 103 requests of this size at once. If each request pasted 100,000 tokens of handbook, one request would take 32.8 GB, leaving room for about four. Keep stable instructions at the start of the prompt, so the engine can reuse their cached work through <a href=\"/learn/inference-engines#prefix-caching\">prefix caching</a>.</p>\n<p>The sketch below estimates tokens as characters divided by 4. Llama 3's tokenizer averaged 3.94 characters per token on a sample of English data, so the estimate suits that tokenizer, and others differ.</p>\n<h4 id=\"order-and-citations\">Order and citations</h4>\n<p>In \"Lost in the Middle,\" Liu and colleagues found a U-shaped curve. Models use information at the start (\"primacy bias\") and at the end (\"recency bias\") of a long input better than information in the middle. With GPT-3.5-Turbo, multi-document question answering accuracy \"can drop by more than 20%.\" In the worst case, with 20 or 30 documents, it fell below the 56.1% that the same model scored with no documents at all. Put the best chunks first, and keep the count small.</p>\n<p>Number the sources and require citations by number. The app maps [2] to a chunk, and it rejects an answer that cites a number it didn't send. Tell the model to say so when the sources don't answer.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> build_prompt</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(question: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, ranked: list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], budget_tokens: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> tuple[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]]:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Keep chunks in rank order until the estimated token budget is spent.</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    Each kept chunk gets a number, which the model must use when it cites.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    kept, lines, used </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [], [], </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> chunk </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ranked:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        cost </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(chunk[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">//</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 4</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # about 4 characters per token</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> used </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cost </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> budget_tokens:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            break</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        used </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cost</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        kept.append(chunk[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        lines.append(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(kept)</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">] </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">chunk[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'title'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}\\n{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">chunk[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'text'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    prompt </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Answer using only the sources below and cite them as [1], [2]. \"</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">              \"If the sources do not answer the question, say so.</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">              +</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(lines) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">Question: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">question</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> prompt, kept</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    ranked </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [{</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"kb#</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"title\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Article </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"word \"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 160</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">} </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    prompt, kept </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> build_prompt(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Can agents approve a late refund?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, ranked, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">budget_tokens</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(kept)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># each chunk is 200 tokens, so the third would push the total past 500</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(prompt.splitlines()[</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # Expected:</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # ['kb#1', 'kb#2']</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # Question: Can agents approve a late refund?</span></span></code></pre>\n<p>A grounded answer to the refund question looks like this:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>Annual plans can be refunded within 30 days of purchase [1]. Agents may approve a refund after 30 days when the outage lasted more than 4 hours [2].</span></span></code></pre>\n<p>Treat retrieved text as data. Wrap each source in clear delimiters, and tell the model to ignore any instructions that appear inside sources. That lowers the risk without removing it. Greshake and colleagues showed that attackers can exploit LLM-integrated applications by \"strategically injecting prompts into data likely to be retrieved\" (<a href=\"/learn/ai-security#prompt-injection\">prompt injection</a> covers defenses).</p>","headings":[{"id":"building-the-prompt","title":"Building the prompt","depth":3},{"id":"the-token-budget","title":"The token budget","depth":4},{"id":"order-and-citations","title":"Order and citations","depth":4}]},{"id":"keeping-the-index-fresh","title":"Keeping the index fresh","html":"<h3 id=\"keeping-the-index-fresh\">Keeping the index fresh</h3>\n<p>Documents change, get deleted and change who may read them, and the index has to follow within a promised time. Suppose Halden promises that an edited policy is answerable within 15 minutes. That number is an assumption, but write the promise down before you design the pipeline.</p>\n<p>Version each document by a content hash and its last-modified time. On a change, re-parse and re-chunk that one document, write its new chunks under the new version, switch queries to that version, and then delete the old chunks from every index and cache. A reader should see one version or the other, never a mix.</p>\n<p>Test deletes explicitly. After a delete event, search for a phrase from the deleted document and expect zero hits from every path. Route changes through a queue, so a bulk import of 1,200 articles can't overwhelm the embedding service (<a href=\"/learn/queues-and-async\">queues and async processing</a> covers the pattern). Watch queue lag, the age of the oldest waiting change, because it shows whether the 15-minute promise holds.</p>\n<p>Permissions change too. Resolve a user's current groups when the query runs, because groups copied at indexing time go stale. Changing the chunker or the embedding model changes every vector, so rebuild the whole index beside the old one, evaluate both on the same eval set, and switch when the new one matches or beats the old (<a href=\"/learn/vector-search#changing-the-embedding-model\">vector search</a> explains why).</p>","headings":[{"id":"keeping-the-index-fresh","title":"Keeping the index fresh","depth":3}]},{"id":"evaluating-retrieval-and-answers","title":"Evaluating retrieval and answers","html":"<h3 id=\"evaluating-retrieval-and-answers\">Evaluating retrieval and answers</h3>\n<p>Measure retrieval and answers separately. If the right chunk never reaches the prompt, no prompt change can fix the answer.</p>\n<p>Build the eval set first. For Halden, assume 200 real agent questions, each with labeled relevant chunks and a reference answer, plus questions the handbook can't answer (<a href=\"/learn/ai-evals-and-observability#building-an-eval-set\">building an eval set</a>).</p>\n<h4 id=\"retrieval-metrics\">Retrieval metrics</h4>\n<p>Recall@k is the share of relevant chunks that appear in the top k results, averaged over questions. It matches a prompt with k chunks: did the evidence reach the model? Mean reciprocal rank (MRR) averages, over questions, 1 divided by the rank of the first relevant result, or 0 if none is returned. NIST's TREC-8 definition uses the same formula. Recall@k rewards finding everything within k, and MRR rewards putting the best result first.</p>\n<p>The sketch computes both on four toy queries. Each has a ranked list from the retriever and a set of chunks labeled relevant.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> recall_at_k</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(ranked: list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], relevant: set[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], k: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Share of the relevant chunks that appear in the top k results.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">set</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(ranked[:k]) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> relevant) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(relevant)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> reciprocal_rank</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(ranked: list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], relevant: set[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"1 divided by the rank of the first relevant result, or 0 if none is ranked.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> rank, item </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> enumerate</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(ranked, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">start</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> item </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> relevant:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> rank</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># (chunk IDs in the retriever's ranked order, chunk IDs labeled relevant)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">runs </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    ([</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c07\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c12\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c03\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c44\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c09\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c12\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    ([</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c13\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c02\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c14\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c20\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c31\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c13\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c14\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    ([</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c50\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c51\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c52\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c53\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c54\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c88\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c88\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    ([</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c61\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c72\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c60\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c73\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c74\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c60\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c61\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        mean </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(recall_at_k(r, rel, k) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> r, rel </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> runs) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(runs)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"recall@</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">k</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> = </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">mean</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"MRR = </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(reciprocal_rank(r, rel) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> r, rel </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> runs) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(runs)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.3f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # Expected:</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # recall@1 = 0.25</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # recall@3 = 0.75</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # recall@5 = 0.75</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # MRR = 0.667</span></span></code></pre>\n<p>Recall@1 is 0.25, recall@3 and recall@5 are 0.75, and MRR is 0.667. The third query's relevant chunk sits at rank 6. Recall@5 counts it as a miss, but MRR still credits it with 1/6. With five chunks in the prompt, report recall@5.</p>\n<h4 id=\"answer-metrics\">Answer metrics</h4>\n<p>Score each answer on correctness against the reference, on faithfulness, and on citation accuracy: does each [n] support its sentence? The RAGAS paper defines faithfulness as an answer whose \"claims that are made in the answer can be inferred from the context.\" A language model usually checks the claims, so calibrate that judge against human labels (<a href=\"/learn/ai-evals-and-observability#ways-to-grade\">ways to grade</a>). Track abstention too.</p>","headings":[{"id":"evaluating-retrieval-and-answers","title":"Evaluating retrieval and answers","depth":3},{"id":"retrieval-metrics","title":"Retrieval metrics","depth":4},{"id":"answer-metrics","title":"Answer metrics","depth":4}]},{"id":"how-rag-fails","title":"How RAG fails","html":"<h3 id=\"how-rag-fails\">How RAG fails</h3>\n<p>Most failures look the same from outside: a wrong answer. The table gives the first check for each symptom.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Symptom</th><th>Likely cause</th><th>First check</th></tr></thead><tbody><tr><td>\"The handbook has no such policy\"</td><td>Never ingested, dropped by the parser, or missed by retrieval</td><td>Search the chunk store for a phrase from the policy; check recall@5 on the question</td></tr><tr><td>Right chunk retrieved, answer ignores it</td><td>Lost in the middle, or too many chunks</td><td>Chunk position; faithfulness by position</td></tr><tr><td>Answer quotes an old policy</td><td>Old and new versions both indexed, or a delete never applied</td><td>Version of each cited chunk; delete events and queue lag</td></tr><tr><td>Sales user sees an HR-only paragraph</td><td>Filter missing from one search path, applied after the prompt, or a shared cache</td><td>Run the query as a user with no access, through each path</td></tr><tr><td>Answer follows an instruction found in an article</td><td>Indirect prompt injection</td><td>Which chunk carried the instruction</td></tr><tr><td>Citation [4] when only three sources were sent</td><td>Citation mapping bug</td><td>Validate citation numbers against the sources sent</td></tr><tr><td>Confident answer to an uncovered question</td><td>No abstention rule; no unanswerable questions in the eval set</td><td>Abstention rate on unanswerable questions</td></tr></tbody></table></div>\n<p>Missing documents are the quietest failure, since a dropped table or skipped folder yields a confident \"not in the handbook.\" Search the chunk store before blaming retrieval. Conflicting sources are a version problem: put version dates in the prompt and tell the model to prefer the newest. Check which chunk carried any injected instruction (<a href=\"/learn/ai-security#prompt-injection\">AI security</a>).</p>","headings":[{"id":"how-rag-fails","title":"How RAG fails","depth":3}]},{"id":"beyond-basic-rag","title":"Beyond basic RAG","html":"<h3 id=\"beyond-basic-rag\">Beyond basic RAG</h3>\n<h4 id=\"agentic-retrieval\">Agentic retrieval</h4>\n<p>Basic RAG searches once. An agent decides what to search for, reads the results and searches again when it needs to. That helps when one search finds half an answer, such as a customer's plan followed by that plan's refund window. Each search is a tool call with its own tokens and latency, so the loop needs a stop rule and a budget (<a href=\"/learn/ai-agents#tool-calling\">AI agents</a>).</p>\n<h4 id=\"graph-based-retrieval\">Graph-based retrieval</h4>\n<p>Top-k retrieval fails on whole-corpus questions such as \"What are the main complaint themes this quarter?\" No single chunk holds that answer. GraphRAG, by Edge and colleagues, \"uses an LLM to build a graph index in two stages.\" It derives an entity knowledge graph from the source documents, then writes community summaries for groups of related entities. The authors state that RAG \"fails on global questions directed at an entire text corpus.\" Indexing calls a language model over the whole corpus, which costs more than embedding the same text. Use graph retrieval for corpus-wide questions only.</p>","headings":[{"id":"beyond-basic-rag","title":"Beyond basic RAG","depth":3},{"id":"agentic-retrieval","title":"Agentic retrieval","depth":4},{"id":"graph-based-retrieval","title":"Graph-based retrieval","depth":4}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Halden adds 2,000 articles of 1,000 words each. Chunks are 300 words, and each starts 250 words after the last. How many chunks are there, and how much memory do 1,024-dimension float32 vectors take?</summary>\n<p>1,000 ÷ 250 = 4 chunks per article, so 8,000 chunks. Each vector takes 1,024 × 4 = 4,096 bytes, so 8,000 vectors take 32,768,000 bytes, about 32.8 MB.</p>\n</details>\n<details>\n<summary>Halden's refund policy changes weekly, and every answer must cite its paragraph. Should the team fine-tune a model on the handbook, use RAG, or paste the whole handbook into a long context window?</summary>\n<p>RAG. Fine-tuning would need retraining for each change, and it stores no source to cite. Ovadia and colleagues found that RAG outperformed fine-tuning for new knowledge. Long context pays for every token on every request, so it suits only small, stable corpora.</p>\n</details>\n<details>\n<summary>Recall@5 is 0.9, but faithfulness is low, and the assistant misses facts its sources contain. The prompt holds 20 chunks, and the relevant one is usually in the middle. What do you change first?</summary>\n<p>Cut the chunk count, and move the best chunks to the start. Retrieval is working, since the evidence arrives. Liu and colleagues found that accuracy drops when relevant information sits in the middle of a long input. Measure faithfulness by chunk position to confirm it on your data.</p>\n</details>\n<details>\n<summary>A sales agent sees a sentence from the HR-only compensation guide. Vector search applies the permission filter, but keyword search doesn't. What is the fix, and how do you prove it?</summary>\n<p>Apply the same filter in both searches, ideally through one shared query builder, and key caches by permission groups. Prove it with a test that runs a user with no HR group through every search path and expects zero compensation-guide chunks.</p>\n</details>\n<details>\n<summary>Three questions have their first correct chunk at rank 1, at rank 4, and nowhere in the top 10. What is MRR, and which query should you investigate first?</summary>\n<p>MRR = (1 + 1/4 + 0) ÷ 3 ≈ 0.417. Investigate the third query. Reranking can't help when nothing relevant was retrieved, so check whether the chunk is indexed and whether the query rewrite dropped a key term.</p>\n</details>\n<details>\n<summary>An \"Exception\" paragraph sits two paragraphs below the rule it modifies, and retrieval returns the exception alone. Name two indexing fixes and one query-time fix.</summary>\n<p>At indexing time, keep whole sections together up to a size limit, and prepend the heading path or a contextual description to each chunk before embedding. At query time, fetch the neighbors of each hit, so the rule travels with the exception.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2005.11401\" target=\"_blank\" rel=\"noopener noreferrer\">Lewis et al., \"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks\" (arXiv 2005.11401)</a>: the paper that introduced RAG.</li>\n<li><a href=\"https://arxiv.org/abs/2307.03172\" target=\"_blank\" rel=\"noopener noreferrer\">Liu et al., \"Lost in the Middle\" (arXiv 2307.03172)</a>: the position effect behind the prompt-ordering advice.</li>\n<li><a href=\"https://www.anthropic.com/news/contextual-retrieval\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic, \"Introducing Contextual Retrieval\"</a>: contextual chunks, contextual BM25 and reranking, with failure rates after each step.</li>\n<li><a href=\"https://arxiv.org/abs/1901.04085\" target=\"_blank\" rel=\"noopener noreferrer\">Nogueira and Cho, \"Passage Re-ranking with BERT\" (arXiv 1901.04085)</a>: the cross-encoder re-ranker, with query and passage read as one input.</li>\n<li><a href=\"https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Cormack, Clarke and Büttcher, \"Reciprocal Rank Fusion\" (SIGIR 2009)</a>: the two-page paper with the RRF formula and the constant 60.</li>\n<li><a href=\"https://arxiv.org/abs/2404.16130\" target=\"_blank\" rel=\"noopener noreferrer\">Edge et al., \"From Local to Global: A Graph RAG Approach\" (arXiv 2404.16130)</a>: GraphRAG, for questions about a whole corpus.</li>\n<li><a href=\"https://arxiv.org/abs/2312.05934\" target=\"_blank\" rel=\"noopener noreferrer\">Ovadia et al., \"Fine-Tuning or Retrieval?\" (arXiv 2312.05934)</a>: the comparison behind the fine-tuning advice.</li>\n<li><a href=\"https://arxiv.org/abs/2309.15217\" target=\"_blank\" rel=\"noopener noreferrer\">Es et al., \"Ragas\" (arXiv 2309.15217)</a>: the faithfulness definition used in the evaluation section.</li>\n<li><a href=\"https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/\" target=\"_blank\" rel=\"noopener noreferrer\">OWASP, \"LLM08:2025 Vector and Embedding Weaknesses\"</a>: permission-aware retrieval and cross-group leaks.</li>\n<li><a href=\"https://arxiv.org/abs/2302.12173\" target=\"_blank\" rel=\"noopener noreferrer\">Greshake et al., \"Not what you've signed up for\" (arXiv 2302.12173)</a>: indirect prompt injection through retrieved data.</li>\n<li><a href=\"https://trec.nist.gov/presentations/TREC8/removed/tsld051.htm\" target=\"_blank\" rel=\"noopener noreferrer\">NIST, TREC-8 slides on reciprocal rank</a>: the MRR definition used here.</li>\n<li><a href=\"https://arxiv.org/abs/2407.21783\" target=\"_blank\" rel=\"noopener noreferrer\">Llama 3 Herd of Models (arXiv 2407.21783)</a>: Table 3 for the architecture numbers, and section 3.2 for the tokenizer's characters per token.</li>\n<li><a href=\"https://www.nvidia.com/en-us/data-center/h100/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA H100</a>: the 80 GB memory figure for the running example.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}