{"id":"ai-security","title":"Security for AI systems","sections":[{"id":"overview","title":"Overview","html":"<p>A parameterized SQL query is safe because the database can tell the query text from the values. A language model cannot make that distinction. It reads one stream of <strong>tokens</strong>, the pieces of text it works with, and that stream holds the developer's instructions, the user's question, a fetched web page and a tool's output. Every rule in the <a href=\"/learn/security\">Security</a> chapter still applies. This chapter covers what the model adds.</p>\n<p>This chapter answers six questions:</p>\n<ul>\n<li>What changes when a model reads text from people the app does not trust?</li>\n<li>What is prompt injection, and why can't it be fully fixed?</li>\n<li>Which defenses reduce the damage, and which only look like defenses?</li>\n<li>How do models leak data, and how should an app handle text they write?</li>\n<li>How far should an agent's tools reach?</li>\n<li>How do jailbreaks, unsafe model files and runaway costs fit in?</li>\n</ul>\n<p>We'll use <strong>Crumb</strong>, the recipe site from the <a href=\"/learn/scalability\">Scalability</a> chapter, which has added a chat assistant. It answers questions about recipes, reads a signed-in user's private drafts when asked, and can email a recipe to a friend.</p>","headings":[]},{"id":"the-new-attack-surface","title":"The new attack surface","html":"<h3 id=\"the-new-attack-surface\">The new attack surface</h3>\n<p>The Security chapter's rule against injection is to keep data separate from code, and a model cannot do that. The UK's National Cyber Security Centre (NCSC) wrote in December 2025 that \"Current large language models (LLMs) simply do not enforce a security boundary between instructions and data inside a prompt.\" The <strong>prompt</strong> is everything the model reads for one call, including the system prompt and any text the app added. <a href=\"/learn/llm-inference\">LLM inference</a> explains tokens and context windows.</p>\n<p>Untrusted text arrives from web pages, emails, comments and tool results. Replies go out as text that the app may treat as HTML, SQL, shell commands or tool arguments. Anything in the prompt can come back out, including through logs and caches. Some model files run code when loaded, and every request spends GPU time or money.</p>\n<p>OWASP, the Open Worldwide Application Security Project, publishes the <a href=\"https://genai.owasp.org/llm-top-10/\" target=\"_blank\" rel=\"noopener noreferrer\">OWASP Top 10 for LLM Applications</a>. When we checked in October 2026, its main page showed the 2025 edition and no newer one. The table lists its ten items and where each is covered here.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Item</th><th>Where this chapter covers it</th></tr></thead><tbody><tr><td>LLM01:2025 Prompt Injection</td><td><a href=\"#prompt-injection\">Prompt injection</a></td></tr><tr><td>LLM02:2025 Sensitive Information Disclosure</td><td><a href=\"#leaking-data\">Leaking data</a></td></tr><tr><td>LLM03:2025 Supply Chain</td><td><a href=\"#the-model-supply-chain\">The model supply chain</a></td></tr><tr><td>LLM04:2025 Data and Model Poisoning</td><td>Briefly, in <a href=\"/learn/training-infrastructure#the-data-pipeline\">the data pipeline</a></td></tr><tr><td>LLM05:2025 Improper Output Handling</td><td><a href=\"#handling-model-output-safely\">Handling model output safely</a></td></tr><tr><td>LLM06:2025 Excessive Agency</td><td><a href=\"#limiting-what-agents-can-do\">Limiting what agents can do</a></td></tr><tr><td>LLM07:2025 System Prompt Leakage</td><td><a href=\"#leaking-data\">Leaking data</a></td></tr><tr><td>LLM08:2025 Vector and Embedding Weaknesses</td><td>Briefly, in <a href=\"/learn/rag\">RAG</a></td></tr><tr><td>LLM09:2025 Misinformation</td><td>Briefly, in <a href=\"/learn/ai-evals-and-observability\">evaluating AI systems</a></td></tr><tr><td>LLM10:2025 Unbounded Consumption</td><td><a href=\"#runaway-costs\">Runaway costs</a></td></tr></tbody></table></div>","headings":[{"id":"the-new-attack-surface","title":"The new attack surface","depth":3}]},{"id":"prompt-injection","title":"Prompt injection","html":"<h3 id=\"prompt-injection\">Prompt injection</h3>\n<p>OWASP defines the first risk as: \"A Prompt Injection Vulnerability occurs when user prompts alter the LLM's behavior or output in unintended ways.\" The user's own prompt is only one way in, since any text the model reads can carry instructions. Simon Willison named the attack in September 2022, after SQL injection: \"I propose that the obvious name for this should be prompt injection.\"</p>\n<figure class=\"diagram\" data-text=\"Developer&#x27;s instructions -> The prompt (one text); User&#x27;s question -> The prompt (one text); Untrusted: recipe page and comments -> The prompt (one text); Private drafts -> The prompt (one text); The prompt (one text) -> Model; Model -> Reply and tool calls\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 445\" width=\"800\" role=\"img\" aria-label=\"One prompt, many sources. The model reads all of it as text.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"105.8\" y1=\"71\" x2=\"212.4\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"188.6\" x2=\"167\" y2=\"194.6\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"255.4\" x2=\"167\" y2=\"249.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"110.2\" y1=\"368\" x2=\"212.4\" y2=\"253\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"222\" x2=\"407\" y2=\"222\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"551\" y1=\"222\" x2=\"647\" y2=\"222\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-note\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">Developer's</tspan><tspan x=\"80\" y=\"54.5\">instructions</tspan></text><rect class=\"dg-node dg-client\" x=\"12\" y=\"136\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"158.5\">User's</tspan><tspan x=\"80\" y=\"174.5\">question</tspan></text><rect class=\"dg-node dg-note\" x=\"12\" y=\"248\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"80\" y=\"270.5\">Untrusted:</tspan><tspan x=\"80\" y=\"286.5\">recipe page and</tspan><tspan x=\"80\" y=\"302.5\">comments</tspan></text><path class=\"dg-node dg-store\" d=\"M12,378 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M12,378 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"401.5\">Private</tspan><tspan x=\"80\" y=\"417.5\">drafts</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"196\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"218.5\">The prompt</tspan><tspan x=\"240\" y=\"234.5\">(one text)</tspan></text><rect class=\"dg-node dg-box\" x=\"412\" y=\"198\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"480\" y=\"226.5\">Model</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"196\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"218.5\">Reply and</tspan><tspan x=\"720\" y=\"234.5\">tool calls</tspan></text></svg></div><figcaption>One prompt, many sources. The model reads all of it as text.</figcaption></figure>\n<p>The developer's instructions, the question and the fetched content become one text. The model has no separate channel for instructions.</p>\n<h4 id=\"direct-and-indirect-injection\">Direct and indirect injection</h4>\n<p>In <strong>direct injection</strong>, the attacker types the instructions into the chat, so the harm usually stays in that user's session. In <strong>indirect injection</strong>, the instructions sit in content that the app later gives to the model, such as a web page, an email or a comment. The victim asks about that content, and the attacker never talks to the model. In 2023, Kai Greshake and colleagues demonstrated this in \"Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.\" They injected prompts \"into data likely to be retrieved,\" against systems that included \"Bing's GPT-4 powered Chat and code-completion engines.\"</p>\n<h4 id=\"a-worked-example-at-crumb\">A worked example at Crumb</h4>\n<p>Priya asks the assistant to summarize a recipe's comments and to check her drafts for the ingredients.</p>\n<ol>\n<li>The app fetches the recipe page and its comments, which strangers wrote.</li>\n<li>One comment contains a sentence written as an instruction, asking the assistant to send the reader's drafts to an address the commenter chose. We describe it rather than quote it, since its wording does not matter here.</li>\n<li>The app joins the developer's instructions, the question, the comments and her drafts into one text. Nothing marks the comment as data that must never be obeyed.</li>\n<li>The model may follow the sentence and call the email tool with the drafts.</li>\n<li>Priya gets an ordinary summary. The stranger already has her drafts.</li>\n</ol>\n<p>The attacker needed only a public comment. Three things had to line up: the text reached the prompt, the model followed it, and a tool could send data out. Each is a place the app can act.</p>\n<h4 id=\"why-there-is-no-complete-fix\">Why there is no complete fix</h4>\n<p>A model follows instructions wherever they appear, and it cannot reliably tell the developer's from an attacker's. The NCSC says that \"it's very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be.\" OWASP says it is \"unclear if there are fool-proof methods of prevention for prompt injection.\" In December 2025, OpenAI wrote that prompt injection is unlikely to ever be fully solved, as TechCrunch reported. The defenses below reduce the risk, but none makes injected text harmless by itself.</p>","headings":[{"id":"prompt-injection","title":"Prompt injection","depth":3},{"id":"direct-and-indirect-injection","title":"Direct and indirect injection","depth":4},{"id":"a-worked-example-at-crumb","title":"A worked example at Crumb","depth":4},{"id":"why-there-is-no-complete-fix","title":"Why there is no complete fix","depth":4}]},{"id":"defending-against-prompt-injection","title":"Defending against prompt injection","html":"<h3 id=\"defending-against-prompt-injection\">Defending against prompt injection</h3>\n<p>Assume some injected text will reach the model, and design so that a successful injection can do little harm. OWASP's LLM01 page lists seven mitigations. Three shape the design most.</p>\n<h4 id=\"enforce-rules-in-code-not-in-the-prompt\">Enforce rules in code, not in the prompt</h4>\n<p>If Crumb's system prompt says \"Only email the signed-in user,\" an injected comment can contradict it. The email tool itself should refuse any address other than the user's own or a contact the user picked. The model can ask to send, and the tool decides. OWASP's LLM07 entry says security controls must be \"enforced independently from the LLM.\"</p>\n<h4 id=\"separate-the-model-that-reads-from-the-model-that-acts\">Separate the model that reads from the model that acts</h4>\n<p>In April 2023, Willison described the <strong>dual LLM</strong> pattern. A <strong>privileged</strong> model takes the user's instructions and holds the tools, but never reads untrusted text. A <strong>quarantined</strong> model reads untrusted text and has no tools, and its output is treated as untrusted. A <strong>controller</strong>, which is plain code, keeps untrusted text in named variables, so the privileged model can refer to it without seeing it.</p>\n<figure class=\"diagram\" data-text=\"Priya -> Controller (plain code) (question); Controller (plain code) <-> Privileged model, tools (placeholders, calls); Controller (plain code) <-> Quarantined model, no tools (text, short answer); Controller (plain code) -> Tools (drafts, email) (approved calls); Recipe page and comments -> Quarantined model, no tools (fetched)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 640 445\" width=\"640\" role=\"img\" aria-label=\"Dual LLM. The model with tools never reads the untrusted page.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"247\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"278.7\" y1=\"73\" x2=\"121.3\" y2=\"191\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"361.3\" y1=\"73\" x2=\"518.7\" y2=\"191\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"320\" y1=\"71\" x2=\"320\" y2=\"366\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"560\" y1=\"373\" x2=\"560\" y2=\"253\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">Priya</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"38.5\">Controller</tspan><tspan x=\"320\" y=\"54.5\">(plain code)</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"196\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"218.5\">Privileged</tspan><tspan x=\"80\" y=\"234.5\">model, tools</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"196\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"218.5\">Quarantined</tspan><tspan x=\"560\" y=\"234.5\">model, no tools</tspan></text><path class=\"dg-node dg-store\" d=\"M252,378 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M252,378 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"401.5\">Tools</tspan><tspan x=\"320\" y=\"417.5\">(drafts, email)</tspan></text><rect class=\"dg-node dg-note\" x=\"492\" y=\"376\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"560\" y=\"398.5\">Recipe page</tspan><tspan x=\"560\" y=\"414.5\">and comments</tspan></text><rect class=\"dg-edge-bg\" x=\"169.2\" y=\"33\" width=\"59.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"199\" y=\"46\" text-anchor=\"middle\">question</text><rect class=\"dg-edge-bg\" x=\"136.1\" y=\"123\" width=\"127.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"200\" y=\"136\" text-anchor=\"middle\">placeholders, calls</text><rect class=\"dg-edge-bg\" x=\"379.2\" y=\"123\" width=\"121.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"440\" y=\"136\" text-anchor=\"middle\">text, short answer</text><rect class=\"dg-edge-bg\" x=\"271.6\" y=\"209.5\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"222.5\" text-anchor=\"middle\">approved calls</text><rect class=\"dg-edge-bg\" x=\"533.3\" y=\"304\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"317\" text-anchor=\"middle\">fetched</text></svg></div><figcaption>Dual LLM. The model with tools never reads the untrusted page.</figcaption></figure>\n<p>Willison wrote that \"The social engineering aspects also mean that this isn't a 100% reliable solution.\" The 2025 CaMeL paper, \"Defeating Prompt Injections by Design,\" by Edoardo Debenedetti and colleagues, goes further. It extracts the control and data flows from the trusted query, so retrieved data cannot change which steps run. On the AgentDojo benchmark, CaMeL solved 77% of tasks with provable security, against 84% for an undefended system. That 7-point gap is the price of the guarantee.</p>\n<h4 id=\"mark-untrusted-content-and-constrain-what-comes-back\">Mark untrusted content and constrain what comes back</h4>\n<p>OWASP's advice to \"segregate and identify external content\" points the same way: keep retrieved text distinct, and label it as data. These labels are soft controls, since the model can still obey a sentence inside them. Willison's 2022 translation example showed this: a prompt that warned the model about such text still failed. Constrained outputs are harder to abuse. If a quarantined model may answer only \"recipe\", \"comment\" or \"spam\", code has nothing else to act on. If the model proposes a URL to fetch, check its host against an allowlist (a list of permitted values) first, as the <a href=\"/learn/security#server-side-request-forgery-ssrf\">SSRF section</a> of the Security chapter explains.</p>\n<p>Filters catch known patterns, so treat them as one layer. Keep poisoned pages in your evaluation set, and log every denied tool call. <a href=\"/learn/ai-evals-and-observability\">Evaluating AI systems</a> explains how to build an eval set.</p>","headings":[{"id":"defending-against-prompt-injection","title":"Defending against prompt injection","depth":3},{"id":"enforce-rules-in-code-not-in-the-prompt","title":"Enforce rules in code, not in the prompt","depth":4},{"id":"separate-the-model-that-reads-from-the-model-that-acts","title":"Separate the model that reads from the model that acts","depth":4},{"id":"mark-untrusted-content-and-constrain-what-comes-back","title":"Mark untrusted content and constrain what comes back","depth":4}]},{"id":"leaking-data","title":"Leaking data","html":"<h3 id=\"leaking-data\">Leaking data</h3>\n<p>OWASP's LLM02 entry opens: \"Sensitive information can affect both the LLM and its application context.\" Its examples include personal identifiable information (PII), financial details, health records and security credentials.</p>\n<h4 id=\"put-only-permitted-data-in-the-prompt\">Put only permitted data in the prompt</h4>\n<p>The model cannot keep a secret from the person it talks to, except by obeying instructions, which can be overridden. So the app decides, before the model runs, what enters the prompt. Priya's drafts go into Priya's prompt and nobody else's. Retrieval must filter by permission before the model sees a document, as <a href=\"/learn/rag#retrieving-and-reranking\">RAG</a> describes.</p>\n<p>System prompts are not secret either. OWASP's LLM07 entry says the system prompt \"should not be considered a secret, nor should it be used as a security control.\" A coupon code in the system prompt is a leak waiting for the right question.</p>\n<h4 id=\"data-leaves-through-the-reply-the-tools-and-the-browser\">Data leaves through the reply, the tools and the browser</h4>\n<p>Data can leave in the reply, in a tool call such as an email, or in a link or image that the browser loads. The last channel is easy to miss. A URL can carry data in its path or query string, so a model that writes a URL can send data to whichever server it names. Sending data to someone who should not receive it is called <strong>exfiltration</strong>.</p>\n<p>A 2025 paper, \"EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System,\" describes this in a product. \"Zero-click\" means the victim does nothing. Its abstract describes a Microsoft 365 Copilot vulnerability that \"enabled remote, unauthenticated data exfiltration via a single crafted email.\" Its public CVE identifier is CVE-2025-32711.</p>\n<h4 id=\"logs-traces-and-caches\">Logs, traces and caches</h4>\n<p>Logs and traces copy prompts and replies, which often contain personal data. Redact when you log, keep retention short and limit who can read them. OWASP's guidance includes \"Tokenization and Redaction.\"</p>\n<p>A model server can also reuse the attention data for a repeated prompt prefix, the <a href=\"/learn/llm-inference#the-kv-cache\">KV cache</a>, through <a href=\"/learn/inference-engines#prefix-caching\">prefix caching</a>. That saves compute, but a cache shared across customers can leak. In a 2025 NDSS Symposium paper, \"I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving,\" Guanlong Wu and colleagues describe sharing \"the Key-Value (KV) cache for identical token sequences among multiple users, saving both memory and computation,\" which \"may lead to new side channel attack vectors.\" A <strong>side channel</strong> is a measurable effect, such as response time, that reveals something the system meant to hide. The paper names SGLang and vLLM as systems that use this mechanism. Keep prefix caches inside one <strong>tenant</strong>, meaning one customer or team.</p>\n<h4 id=\"most-leaks-need-no-attacker\">Most leaks need no attacker</h4>\n<p>In May 2023, The Register reported that Samsung had temporarily banned generative AI tools, after Korean media reports that staff had typed confidential material into ChatGPT while debugging code and writing up meeting notes. Data that leaves the company's systems cannot be pulled back. Prevention is policy and tooling: an approved tool, a list of data that must never be pasted, and controlled retention.</p>","headings":[{"id":"leaking-data","title":"Leaking data","depth":3},{"id":"put-only-permitted-data-in-the-prompt","title":"Put only permitted data in the prompt","depth":4},{"id":"data-leaves-through-the-reply-the-tools-and-the-browser","title":"Data leaves through the reply, the tools and the browser","depth":4},{"id":"logs-traces-and-caches","title":"Logs, traces and caches","depth":4},{"id":"most-leaks-need-no-attacker","title":"Most leaks need no attacker","depth":4}]},{"id":"handling-model-output-safely","title":"Handling model output safely","html":"<h3 id=\"handling-model-output-safely\">Handling model output safely</h3>\n<p>OWASP's LLM05 entry defines the risk as \"insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed downstream to other components and systems.\" The model should be treated as \"any other user, adopting a zero-trust approach.\" So the Security chapter's rules apply to model output as they apply to form fields.</p>\n<h4 id=\"escape-for-html\">Escape for HTML</h4>\n<p>If a reply reaches the page as raw HTML, a <code>&#x3C;script></code> tag in it runs in the reader's browser. That is stored cross-site scripting (XSS), with the model standing in for the commenter. Escape the text on the way out. This sketch does that, and it runs as written.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> html</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> render_reply</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(model_text: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Turn a model reply into HTML that shows the text and nothing else.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    escaped </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> html.escape(model_text, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">quote</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># &#x26; &#x3C; > \" ' become entities</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    paragraphs </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> escaped.split(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p.strip()]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"&#x3C;p>\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p.replace(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"&#x3C;br>\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"&#x3C;/p>\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> paragraphs)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    reply </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> 'Your refund is on its way.</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">&#x3C;script>alert(\"hi\")&#x3C;/script>'</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    out </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> render_reply(reply)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(out)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # &#x3C;p>Your refund is on its way.&#x3C;/p>&#x3C;p>&#x26;lt;script&#x26;gt;alert(&#x26;quot;hi&#x26;quot;)&#x26;lt;/script&#x26;gt;&#x3C;/p></span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    assert</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"&#x3C;script>\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> out</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    assert</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> out.startswith(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"&#x3C;p>Your refund is on its way.&#x3C;/p>\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"ok\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<p>The <code>&#x3C;script></code> tag arrives as harmless text, and the quotes are escaped too, so the text is also safe inside an attribute. Escaping covers text only. Markdown needs a sanitizer with an allowlist of tags, and a URL in an <code>href</code> or <code>src</code> needs a scheme check, because a <code>javascript:</code> URL runs code with no tag at all.</p>\n<h4 id=\"build-queries-from-allowlists\">Build queries from allowlists</h4>\n<p>If the assistant filters recipes by cuisine, the model should not write SQL. Ask it for a JSON object such as <code>{\"cuisine\": \"thai\"}</code>, check the value against a fixed list, and pass it as a query parameter. Shell commands follow the same rule: the model chooses from named operations, and the code builds the argument array.</p>\n<h4 id=\"images-and-links-are-a-data-channel\">Images and links are a data channel</h4>\n<p>When the chat panel renders Markdown, the browser loads each image without a click, and the URL the model wrote can point anywhere. Do not load external images from model output. If you need them, allow only your own hosts through your proxy, and set a Content Security Policy with a tight <code>img-src</code>.</p>","headings":[{"id":"handling-model-output-safely","title":"Handling model output safely","depth":3},{"id":"escape-for-html","title":"Escape for HTML","depth":4},{"id":"build-queries-from-allowlists","title":"Build queries from allowlists","depth":4},{"id":"images-and-links-are-a-data-channel","title":"Images and links are a data channel","depth":4}]},{"id":"limiting-what-agents-can-do","title":"Limiting what agents can do","html":"<h3 id=\"limiting-what-agents-can-do\">Limiting what agents can do</h3>\n<p>An agent is a model in a loop that calls tools, as the <a href=\"/learn/ai-agents\">AI agents</a> chapter explains. Each <strong>tool</strong>, a function the model can ask the app to run, is a permission. OWASP's LLM06 entry names \"excessive functionality,\" \"excessive permissions\" and \"excessive autonomy\" as the causes of excessive agency. Its defenses include human-in-the-loop control, so a person approves high-impact actions before they happen.</p>\n<h4 id=\"the-lethal-trifecta\">The lethal trifecta</h4>\n<p>In June 2025, Simon Willison named three conditions that together make an agent dangerous:</p>\n<ol>\n<li>\"Access to your private data\"</li>\n<li>\"Exposure to untrusted content\"</li>\n<li>\"The ability to externally communicate in a way that could be used to steal your data\"</li>\n</ol>\n<p>\"If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker,\" he wrote. His advice is to \"avoid that lethal trifecta combination entirely.\"</p>\n<p>Crumb's assistant has all three. Remove one leg wherever you can, and gate the rest. The NCSC is blunter: \"Therefore don't let an LLM processing emails from random external people have access to privileged tools.\" For Crumb, keep the drafts, which are the feature, and the comments, which are the content, then gate the send. Reading leaks nothing outside the session by itself, and a send is the one way out.</p>\n<figure class=\"diagram\" data-text=\"Private drafts and saved lists -> Assistant (model + tools) (reads); Comments and recipe pages from strangers -> Assistant (model + tools) (reads); Assistant (model + tools) -> Send email to anyone (sends)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 480 440\" width=\"480\" role=\"img\" aria-label=\"The lethal trifecta. Remove a leg, or gate the last one.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"110.2\" y1=\"76\" x2=\"212.4\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"367.1\" y1=\"79\" x2=\"267.6\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"240\" y1=\"251\" x2=\"240\" y2=\"371\" marker-end=\"url(#dg-arrow)\"></line><path class=\"dg-node dg-store\" d=\"M12,18 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M12,18 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"41.5\">Private drafts</tspan><tspan x=\"80\" y=\"57.5\">and saved lists</tspan></text><rect class=\"dg-node dg-note\" x=\"332\" y=\"8\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"400\" y=\"30.5\">Comments and</tspan><tspan x=\"400\" y=\"46.5\">recipe pages</tspan><tspan x=\"400\" y=\"62.5\">from strangers</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"196\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"218.5\">Assistant</tspan><tspan x=\"240\" y=\"234.5\">(model + tools)</tspan></text><rect class=\"dg-node dg-client\" x=\"172\" y=\"376\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"398.5\">Send email</tspan><tspan x=\"240\" y=\"414.5\">to anyone</tspan></text><rect class=\"dg-edge-bg\" x=\"140.8\" y=\"124.5\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"161.3\" y=\"137.5\" text-anchor=\"middle\">reads</text><rect class=\"dg-edge-bg\" x=\"296.8\" y=\"126\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"317.3\" y=\"139\" text-anchor=\"middle\">reads</text><rect class=\"dg-edge-bg\" x=\"219.5\" y=\"302\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"315\" text-anchor=\"middle\">sends</text></svg></div><figcaption>The lethal trifecta. Remove a leg, or gate the last one.</figcaption></figure>\n<h4 id=\"give-each-tool-the-smallest-scope\">Give each tool the smallest scope</h4>\n<p><code>read_drafts</code> returns only the signed-in user's drafts. <code>search_recipes</code> queries the public index. <code>save_to_list</code> writes only to the user's list. <code>send_email</code> reaches only the user or a contact the user picked, with a length limit and no attachments. Crumb has no tool that runs arbitrary SQL or fetches any URL, the open-ended tools that OWASP advises against where possible.</p>\n<h4 id=\"ask-a-person-before-a-risky-action\">Ask a person before a risky action</h4>\n<p>A risky action has effects outside the session, such as sending, paying, deleting or changing settings. A session with untrusted content in it is <strong>tainted</strong>, and in a tainted session every write is risky, because the model's next choice may come from that content. The approval screen must show the exact action, including the recipient and full body. A summary written by the model can be wrong or shaped by injected text.</p>\n<p>The gate sits between the model's request and the tool:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> dataclasses </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> dataclass</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> typing </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> Callable</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Each tool is read, write or external. Unlisted tools are denied.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TOOLS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"read_drafts\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"read\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"search_recipes\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"read\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"save_to_list\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"write\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"send_email\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"external\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">@dataclass</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> ToolCall</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    name: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    args: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> allow</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(call: ToolCall, tainted: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">          approve: Callable[[ToolCall], </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Decide whether a model-requested tool call may run.</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    tainted: untrusted text (a page, comment or email) is in the context.</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    approve: shows the exact call to a person; returns True or False.</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    kind </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> TOOLS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.get(call.name)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> kind </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> False</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">           # unknown tool: deny by default</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> kind </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"read\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> True</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">            # reads change nothing outside the session</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> kind </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"write\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> and</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tainted:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> True</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">            # a clean session may save to the user's list</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> approve(call)       </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># tainted writes, and every send, need a person</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    never </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> lambda</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> call: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     # a person who always says no</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    always </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> lambda</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> call: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     # a person who always says yes</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    cases </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        (ToolCall(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"search_recipes\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"q\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"rice\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, never),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        (ToolCall(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"save_to_list\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">7</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, never),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        (ToolCall(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"save_to_list\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">7</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, never),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        (ToolCall(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"send_email\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"to\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"a@example.com\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, never),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        (ToolCall(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"send_email\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"to\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"a@example.com\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, always),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        (ToolCall(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"delete_account\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {}), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, always),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    ]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> call, tainted, approve </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cases:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        result </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> allow(call, tainted, approve)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">call.name</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:15</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> tainted=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">tainted</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">!s:5</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">result</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<p>Running it prints:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>search_recipes  tainted=True  -> True</span></span>\n<span class=\"line\"><span>save_to_list    tainted=False -> True</span></span>\n<span class=\"line\"><span>save_to_list    tainted=True  -> False</span></span>\n<span class=\"line\"><span>send_email      tainted=False -> False</span></span>\n<span class=\"line\"><span>send_email      tainted=True  -> True</span></span>\n<span class=\"line\"><span>delete_account  tainted=False -> False</span></span></code></pre>\n<p>Unknown tools are denied, so a new tool does nothing until someone classifies it. Reads run freely. Writes run freely in a clean session but need a person in a tainted one, and every send needs a person. Ask only for actions with effects, because if the assistant asks before every search, people will click yes without reading.</p>\n<p>If an agent runs code, run it in a sandbox with no network by default, no secrets, and limits on time and memory. OWASP's LLM10 entry lists sandbox techniques among its mitigations.</p>","headings":[{"id":"limiting-what-agents-can-do","title":"Limiting what agents can do","depth":3},{"id":"the-lethal-trifecta","title":"The lethal trifecta","depth":4},{"id":"give-each-tool-the-smallest-scope","title":"Give each tool the smallest scope","depth":4},{"id":"ask-a-person-before-a-risky-action","title":"Ask a person before a risky action","depth":4}]},{"id":"abuse-and-content-safety","title":"Abuse and content safety","html":"<h3 id=\"abuse-and-content-safety\">Abuse and content safety</h3>\n<p>A <strong>jailbreak</strong> is a prompt that gets a model to ignore its safety training. OWASP calls it \"a form of prompt injection where the attacker provides inputs that cause the model to disregard its safety protocols entirely.\"</p>\n<p>Banned-phrase lists cannot keep up. In a 2023 paper, Andy Zou and colleagues used an automated search to find suffixes, strings of text added to a request, that made models produce objectionable content. Their abstract says the suffixes are \"quite transferable, including to black-box, publicly released LLMs,\" and that one worked on the public interfaces of ChatGPT, Bard and Claude.</p>\n<p>A <strong>moderation classifier</strong> is a separate model that labels text or images against categories. OpenAI's guide says its models \"detect harmful content in text and images,\" and recommends using the scores as signals for your own policy. Two limits matter for agents. The endpoint \"doesn't cover tool names, tool descriptions, tool schemas, or response-format schemas.\" And \"if you stream a generated response, moderation scores arrive after the full generated output is available.\" You can wait for the whole reply, which delays the first visible token, or stream and replace a flagged reply. A general chat app can stream. An app for children should hold output until the check passes.</p>\n<p>Misuse at scale is mostly a rate and identity problem, which <a href=\"/learn/rate-limiting\">rate limiting</a> covers.</p>","headings":[{"id":"abuse-and-content-safety","title":"Abuse and content safety","depth":3}]},{"id":"the-model-supply-chain","title":"The model supply chain","html":"<h3 id=\"the-model-supply-chain\">The model supply chain</h3>\n<p>OWASP's LLM03 entry warns that supply chain weaknesses \"can affect the integrity of training data, models, and deployment platforms.\" For some model file formats, the file is code as well as data.</p>\n<h4 id=\"why-pickle-files-can-run-code\">Why pickle files can run code</h4>\n<p>Python's <code>pickle</code> format stores an object as instructions for a small <strong>stack machine</strong>, a program that pushes and pops values. The Python documentation warns: \"The pickle module is not secure. Only unpickle data you trust.\" Hugging Face's documentation explains that some instructions import a name from a module, and <code>REDUCE</code> calls it with arguments, so loading the file runs code. Pickle is \"the default format for PyTorch model weights.\"</p>\n<p>PyTorch changed its default for this reason. Since version 2.6, <code>torch.load</code> uses <code>weights_only=True</code> unless you pass a <code>pickle_module</code>. Its <a href=\"https://docs.pytorch.org/docs/2.6/notes/serialization.html\" target=\"_blank\" rel=\"noopener noreferrer\">serialization notes</a> warn that <code>weights_only=False</code> \"can result in arbitrary code execution.\" Load any community checkpoint in an isolated worker with no database access, no network and no secrets. That is <a href=\"/learn/security#the-principle-of-least-privilege\">least privilege</a> applied to model loading.</p>\n<h4 id=\"prefer-safetensors\">Prefer safetensors</h4>\n<p>Safetensors is a format \"for storing tensors safely (as opposed to pickle),\" according to its <a href=\"https://huggingface.co/docs/safetensors/index\" target=\"_blank\" rel=\"noopener noreferrer\">documentation</a>. A <strong>tensor</strong> is a multi-dimensional array of numbers, such as a weight matrix. Its layout is an 8-byte little-endian integer giving the size of a JSON header, then the header, which maps each tensor name to its data type, shape and byte range, then the raw bytes. Nothing in that layout asks a loader to import or call anything. The <a href=\"https://github.com/huggingface/safetensors\" target=\"_blank\" rel=\"noopener noreferrer\">README</a> marks pickle as \"Unsafe, runs arbitrary code.\"</p>\n<p>A safe format does not make a model trustworthy. OWASP warns that \"Models are binary black boxes and unlike open source, static inspection can offer little to security assurances.\"</p>\n<h4 id=\"scan-verify-and-pin\">Scan, verify and pin</h4>\n<p>Hugging Face's Hub scans pickle files and lists the modules they import, using <code>pickletools.genops</code>, which \"allows us to read the file without executing potentially dangerous code.\" The page says the scan is \"not 100% foolproof.\" OWASP advises that you \"only use models from verifiable sources and use third-party model integrity checks with signing and file hashes.\" Pin the exact commit you reviewed, and check each file's <strong>hash</strong>, a fingerprint of its bytes, in your deploy pipeline.</p>\n<h4 id=\"a-research-demonstration\">A research demonstration</h4>\n<p>In February 2024, HiddenLayer published \"Silent Sabotage: Hijacking Safetensors Conversion on Hugging Face.\" It described how Hugging Face's conversion bot, SFConvertbot, which turned pickle models into safetensors through pull requests, could be abused. A hijacked model submitted for conversion could persist malicious code \"inside the service so that models are hijacked automatically.\" HiddenLayer says it told Hugging Face before publishing. The lesson is general: a service that converts, hosts or scans model files is part of your supply chain, and it can be attacked like any other build step.</p>","headings":[{"id":"the-model-supply-chain","title":"The model supply chain","depth":3},{"id":"why-pickle-files-can-run-code","title":"Why pickle files can run code","depth":4},{"id":"prefer-safetensors","title":"Prefer safetensors","depth":4},{"id":"scan-verify-and-pin","title":"Scan, verify and pin","depth":4},{"id":"a-research-demonstration","title":"A research demonstration","depth":4}]},{"id":"runaway-costs","title":"Runaway costs","html":"<h3 id=\"runaway-costs\">Runaway costs</h3>\n<p>OWASP's LLM10 entry describes Denial of Wallet: attackers who send heavy volumes exploit usage-based billing, \"leading to unsustainable financial burdens on the provider and risking financial ruin.\" A bug that loops over a tool call a thousand times burns the same money.</p>\n<h4 id=\"where-the-money-goes\">Where the money goes</h4>\n<p>API bills count input tokens, everything sent in the prompt, and output tokens, everything written back. An agent resends its growing history at every step, so a run's cost grows with its steps. Assume a model API costs $3 per million input tokens. That is an assumption, since real prices vary widely by provider and commitment.</p>\n<ul>\n<li>One chat turn with a 2,000-token prompt costs 2,000 × $3 ÷ 1,000,000 = $0.006.</li>\n<li>An agent run with 20 steps starts at 10,000 tokens and adds 1,000 tokens of tool results per step. The total is 20 × 10,000 + 1,000 × 190 = 390,000 input tokens, which costs $1.17, about 195 chat turns' worth.</li>\n</ul>\n<p>Caching a stable prompt prefix can make repeated text cheaper, as <a href=\"/learn/ai-agents#managing-context\">Managing context</a> explains, but caps are still needed.</p>\n<p>Self-hosting moves the cost from tokens to GPU time. Assume $3.00 per H100-hour. The 70B deployment in <a href=\"/learn/ai-estimation#memory-for-the-kv-cache\">AI estimation</a> uses four H100s, so 4 × $3.00 = $12.00 an hour, or about $8,760 a month at 730 hours. With a fixed fleet, an attacker cannot raise that bill, but can fill the capacity. If the fleet autoscales, the attacker can do both. Its 148 GB of KV cache holds 148 × 10^9 ÷ 327,680 ≈ 451,660 tokens. Each 4,096-token conversation takes 4,096 × 327,680 bytes ≈ 1.34 GB, so about 110 long conversations fill the cache, and other users wait.</p>\n<h4 id=\"put-limits-at-several-levels\">Put limits at several levels</h4>\n<ul>\n<li><strong>Per request:</strong> cap input and output tokens.</li>\n<li><strong>Per agent run:</strong> cap steps, tool calls and total input tokens.</li>\n<li><strong>Per user or key:</strong> use token-based limits, estimating before a call and settling after it. <a href=\"/learn/llm-gateways#token-based-rate-limits\">Token-based rate limits</a> covers how.</li>\n<li><strong>Per tenant:</strong> cap concurrent requests, so one customer cannot take all the GPUs.</li>\n<li><strong>Per team:</strong> set a budget with alerts, and a switch that stops the feature.</li>\n</ul>\n<p>The sketch enforces the per-run limit and computes the two costs above.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> run_cost</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(start_tokens: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, tokens_per_step: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, steps: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">             usd_per_million: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Input cost of an agent run whose prompt grows each step.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(start_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tokens_per_step </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(steps))</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> usd_per_million </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1_000_000</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> Budget</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Stop an agent run before it passes its step or input-token limit.\"\"\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, max_steps: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, max_input_tokens: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_steps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_steps</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_input_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_input_tokens</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.steps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.input_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> charge</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, context_tokens: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"\"\"Call before each model call, with the size of the prompt.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.steps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.input_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> context_tokens</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.steps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_steps:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"step limit of </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_steps</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> reached\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.input_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_input_tokens:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input-token limit of </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_input_tokens</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> reached\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    turn </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> run_cost(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3.00</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)        </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># one 2,000-token chat turn</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    run </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> run_cost(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">20</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3.00</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 20 steps, prompt grows 1,000 each</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"one turn: $</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">turn</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.3f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)           </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># one turn: $0.006</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"20-step run: $</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">run</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">run </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> turn</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.0f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> times one turn\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # 20-step run: $1.17, 195 times one turn</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_steps, max_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">50_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">25_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)]:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        budget </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> Budget(max_steps, max_tokens)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> _ </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                budget.charge(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)         </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># four calls of 10,000 tokens</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        except</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> err:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">            print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"stopped:\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, err)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # stopped: step limit of 3 reached</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # stopped: input-token limit of 25000 reached</span></span></code></pre>\n<p><code>Budget.charge</code> runs before each model call and raises an error when a limit would be passed, so the call that goes over is never sent. The hard stop lives in code the model cannot change.</p>","headings":[{"id":"runaway-costs","title":"Runaway costs","depth":3},{"id":"where-the-money-goes","title":"Where the money goes","depth":4},{"id":"put-limits-at-several-levels","title":"Put limits at several levels","depth":4}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>A Crumb assistant run starts with a 6,000-token prompt and adds 800 tokens of tool results at each of 15 steps. At $3 per million input tokens, what does one run cost, and how many 2,000-token chat turns cost the same?</summary>\n<p>Step i (counting from 0) processes 6,000 + 800 × i tokens, so the run processes 15 × 6,000 + 800 × 105 = 174,000 tokens, or $0.522. A 2,000-token turn costs $0.006, so the run costs as much as 87 turns. A step or input-token cap would stop it early.</p>\n</details>\n<details>\n<summary>The 70B deployment on four H100s keeps about 148 GB for the KV cache. How many 4,096-token conversations fit, and what happens when 150 arrive together?</summary>\n<p>The cache holds about 451,660 tokens, so about 110 conversations fit. The other 40 wait or are rejected, and everyone's latency rises. A per-tenant cap on concurrent requests stops one customer from taking all the capacity.</p>\n</details>\n<details>\n<summary>Crumb's assistant reads public comments, sees each user's private drafts and can email recipes. Which leg of the lethal trifecta do you remove or gate, and why?</summary>\n<p>Keep the comments and the drafts, since they are the content and the feature. Gate the send. Reading leaks nothing outside the session by itself, and sending is the one way out. A send can show the exact recipient and body in a preview, and it can be limited to the user's own address.</p>\n</details>\n<details>\n<summary>After Crumb enables Markdown rendering, a reply makes the reader's browser request an image from a domain Crumb does not own, and the URL includes text from the user's last message. What is happening, and what two changes fix it?</summary>\n<p>The reply contains an image whose URL carries data, and the browser loads images without a click. Stop loading external images from model output, or allow only your own hosts through your proxy. Also add a Content Security Policy with a strict <code>img-src</code>.</p>\n</details>\n<details>\n<summary>A developer puts live coupon codes in the system prompt with the line \"Never reveal these codes.\" What is wrong, and where should the codes live?</summary>\n<p>The system prompt is not a secret. It is part of the model's context, and a clever question can get the model to repeat it. OWASP's LLM07 entry says it \"should not be considered a secret, nor should it be used as a security control.\" The coupon service should check codes on the server, and the model should only ask it to apply one.</p>\n</details>\n<details>\n<summary>A teammate wants to load a 7B community model whose files are PyTorch pickle checkpoints. What do you ask for, and what do you check first?</summary>\n<p>Ask for safetensors, whose layout loads without running code. If only the pickle exists, load it in an isolated worker with no database access, no network and no secrets, using <code>weights_only=True</code>. Check the source and any signature, pin the exact commit, and compare file hashes. Then run your evaluation set, since a safe format does not show good behavior.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://genai.owasp.org/llm-top-10/\" target=\"_blank\" rel=\"noopener noreferrer\">OWASP Top 10 for LLM Applications, 2025</a>: the ten risks in the table. Start with LLM01, LLM06 and LLM10.</li>\n<li>Simon Willison: his <a href=\"https://simonwillison.net/2022/Sep/12/prompt-injection/\" target=\"_blank\" rel=\"noopener noreferrer\">2022 post on prompt injection</a>, <a href=\"https://simonwillison.net/2023/Apr/25/dual-llm-pattern/\" target=\"_blank\" rel=\"noopener noreferrer\">2023 post on the dual LLM pattern</a> and <a href=\"https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/\" target=\"_blank\" rel=\"noopener noreferrer\">2025 post on the lethal trifecta</a>. Read them in that order.</li>\n<li><a href=\"https://arxiv.org/abs/2302.12173\" target=\"_blank\" rel=\"noopener noreferrer\">Greshake et al., arXiv 2302.12173</a>: the indirect prompt injection study.</li>\n<li><a href=\"https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection\" target=\"_blank\" rel=\"noopener noreferrer\">NCSC, \"Prompt injection is not SQL injection (it may be worse)\"</a>: why the SQL fix does not carry over.</li>\n<li><a href=\"https://arxiv.org/abs/2503.18813\" target=\"_blank\" rel=\"noopener noreferrer\">Debenedetti et al., arXiv 2503.18813</a>: CaMeL, with its AgentDojo results.</li>\n<li><a href=\"https://arxiv.org/abs/2509.10540\" target=\"_blank\" rel=\"noopener noreferrer\">Reddy et al., arXiv 2509.10540</a>: EchoLeak, a zero-click exfiltration in a production product.</li>\n<li><a href=\"https://www.ndss-symposium.org/ndss-paper/i-know-what-you-asked-prompt-leakage-via-kv-cache-sharing-in-multi-tenant-llm-serving/\" target=\"_blank\" rel=\"noopener noreferrer\">Wu et al., \"I Know What You Asked\" (NDSS)</a>: prompt leakage through shared KV caches.</li>\n<li><a href=\"https://arxiv.org/abs/2307.15043\" target=\"_blank\" rel=\"noopener noreferrer\">Zou et al., arXiv 2307.15043</a>: automated jailbreak suffixes that transfer across models.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/moderation\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI's moderation guide</a>: categories, limits and the streaming caveat.</li>\n<li><a href=\"https://www.theregister.com/2023/05/02/samsung_generative_ai_ban/\" target=\"_blank\" rel=\"noopener noreferrer\">The Register on Samsung (May 2023)</a>: a leak that needed no attacker.</li>\n<li><a href=\"https://docs.python.org/3/library/pickle.html\" target=\"_blank\" rel=\"noopener noreferrer\">Python's pickle documentation</a>: the security warning.</li>\n<li><a href=\"https://docs.pytorch.org/docs/2.6/notes/serialization.html\" target=\"_blank\" rel=\"noopener noreferrer\">PyTorch serialization notes, version 2.6</a>: the <code>weights_only</code> default.</li>\n<li><a href=\"https://huggingface.co/docs/hub/security-pickle\" target=\"_blank\" rel=\"noopener noreferrer\">Hugging Face Hub, pickle scanning</a>: how pickle files run code, and what the scanner misses.</li>\n<li><a href=\"https://huggingface.co/docs/safetensors/index\" target=\"_blank\" rel=\"noopener noreferrer\">Safetensors documentation</a> and <a href=\"https://github.com/huggingface/safetensors\" target=\"_blank\" rel=\"noopener noreferrer\">README</a>: purpose and layout.</li>\n<li><a href=\"https://hiddenlayer.com/research/silent-sabotage/\" target=\"_blank\" rel=\"noopener noreferrer\">HiddenLayer, \"Silent Sabotage\"</a>: the 2024 research on the conversion bot.</li>\n<li><a href=\"https://techcrunch.com/2025/12/22/openai-says-ai-browsers-may-always-be-vulnerable-to-prompt-injection-attacks/\" target=\"_blank\" rel=\"noopener noreferrer\">TechCrunch on OpenAI and prompt injection (December 2025)</a>: OpenAI's statement that prompt injection is unlikely to be fully solved.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}