{"id":"llm-gateways","title":"LLM gateways","sections":[{"id":"overview","title":"Overview","html":"<p>A <strong>model provider</strong> sells access to a language model through an HTTP API. A <strong>gateway</strong> sits between your applications and those providers. Each application calls one internal endpoint, and the gateway chooses which provider or model answers. An LLM gateway is a specialized <a href=\"/learn/reverse-proxy-and-api-gateway#reverse-proxy-vs-api-gateway\">API gateway</a>. That chapter covers TLS, authentication and per-key limits, so this one covers what changes for model traffic:</p>\n<ul>\n<li>Answers <strong>stream for many seconds</strong>, so connections stay open far longer than typical web requests.</li>\n<li>Cost is counted in <strong>tokens</strong>, the pieces of text a model reads and writes (see <a href=\"/learn/llm-inference#tokens\">Tokens</a>), and the output count is known only at the end.</li>\n<li>Providers set <strong>their own limits</strong> in requests and tokens per minute.</li>\n<li>Models are <strong>not interchangeable</strong>. A fallback can change the answer, its format, or whether the request is accepted at all.</li>\n</ul>\n<p>This chapter answers five questions:</p>\n<ul>\n<li>What does a gateway add beyond calling a provider directly?</li>\n<li>How should a gateway choose a model, and when should it fall back?</li>\n<li>Why reserve tokens before the answer exists, and settle them afterward?</li>\n<li>What can a gateway cache, and how can a cache return a wrong answer?</li>\n<li>How do streams pass through a gateway, and how do you track cost per team?</li>\n</ul>\n<p>We'll follow <strong>Crumb</strong>, the recipe site from earlier chapters. Its recipe helper answers cooking questions with a hosted model. The search team wants a second provider as a backup, and the data team runs a small <strong>open-weight</strong> model (one whose weights are published for download) on its own GPUs. Without a gateway, each service would need its own keys, retries, token counts and logs. The <a href=\"/learn/design-llm-gateway\">LLM gateway design question</a> sizes the same parts for a company with 200 apps.</p>","headings":[]},{"id":"what-a-gateway-adds","title":"What a gateway adds","html":"<h3 id=\"what-a-gateway-adds\">What a gateway adds</h3>\n<p>Without a gateway, each Crumb service calls a provider directly, holds its own key, retries in its own way and writes its own logs. Rotating a key means changing every service. With a gateway, each service makes one kind of call to one internal address:</p>\n<figure class=\"diagram\" data-text=\"Recipe helper -> LLM gateway keys, limits, logs (one API); Search team -> LLM gateway keys, limits, logs; Nightly job -> LLM gateway keys, limits, logs; LLM gateway keys, limits, logs -> Provider A hosted model; LLM gateway keys, limits, logs -> Provider B hosted model; LLM gateway keys, limits, logs -> Self-hosted open-weight model\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 720 325\" width=\"720\" role=\"img\" aria-label=\"One gateway sits between every app and every model endpoint\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"134\" y1=\"69\" x2=\"258\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"247\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"134\" y1=\"255\" x2=\"258\" y2=\"193\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"391\" y1=\"135.4\" x2=\"567\" y2=\"69.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"391\" y1=\"162\" x2=\"567\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"391\" y1=\"188.6\" x2=\"567\" y2=\"254.6\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">Recipe helper</tspan></text><rect class=\"dg-node dg-client\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Search team</tspan></text><rect class=\"dg-node dg-client\" x=\"12\" y=\"258\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"286.5\">Nightly job</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"158.5\">LLM gateway</tspan><tspan x=\"320\" y=\"174.5\">keys, limits, logs</tspan></text><rect class=\"dg-node dg-box\" x=\"572\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"640\" y=\"38.5\">Provider A</tspan><tspan x=\"640\" y=\"54.5\">hosted model</tspan></text><rect class=\"dg-node dg-box\" x=\"572\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"640\" y=\"158.5\">Provider B</tspan><tspan x=\"640\" y=\"174.5\">hosted model</tspan></text><path class=\"dg-node dg-store\" d=\"M572,258 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M572,258 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"640\" y=\"281.5\">Self-hosted</tspan><tspan x=\"640\" y=\"297.5\">open-weight model</tspan></text><rect class=\"dg-edge-bg\" x=\"169.3\" y=\"91\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"196\" y=\"104\" text-anchor=\"middle\">one API</text></svg></div><figcaption>One gateway sits between every app and every model endpoint</figcaption></figure>\n<p>The gateway does three jobs:</p>\n<ul>\n<li><strong>One request format.</strong> vLLM, a serving engine that runs models on GPUs, lists a Chat Completions endpoint at <code>/v1/chat/completions</code> for text models with a chat template. Agent Router, formerly Envoy AI Gateway, describes its aim as \"a stable, OpenAI-compatible interface to models.\" The gateway translates to a provider's native format when it must.</li>\n<li><strong>Keys.</strong> Applications authenticate to the gateway with a key the gateway issues, so provider keys never leave it. LiteLLM calls these virtual keys, and each can belong to a person or a team.</li>\n<li><strong>Routing, limits, caching and logs,</strong> covered below.</li>\n</ul>\n<p>The gateway adds a network hop and some processing time, which you should measure. It cannot make a weak model answer well or stop a malicious prompt on its own. It is one layer among several.</p>","headings":[{"id":"what-a-gateway-adds","title":"What a gateway adds","depth":3}]},{"id":"routing-and-fallbacks","title":"Routing and fallbacks","html":"<h3 id=\"routing-and-fallbacks\">Routing and fallbacks</h3>\n<p><strong>Routing</strong> chooses a <strong>target</strong> (a provider, a model and a key) for each request. A <strong>fallback</strong> is the next target to try. Apps request an <strong>alias</strong> such as <code>chat-large</code>, a name the gateway maps to a list of targets, so moving traffic off a provider means editing that list.</p>\n<p>Let the status code decide how to react:</p>\n<ol>\n<li><strong>429</strong> means a limit was reached. Wait the seconds in <code>Retry-After</code> if present. Anthropic's rate-limit page says to \"wait that many seconds before retrying.\" Without the header, OpenAI's error page recommends exponential backoff with jitter. If the 429 persists, fall back.</li>\n<li><strong>500, 502, 503, 504 and 529</strong> mean the provider is failing or busy. Retry briefly, then fall back. Anthropic describes 529 as \"The API is temporarily overloaded.\"</li>\n<li><strong>Timeouts and connection failures</strong> justify a fallback.</li>\n<li><strong>400</strong> means the request is wrong for that target. Anthropic's error page lists many model-specific rules, such as several recent models rejecting forced tool use. Check a target's features before routing to it, so fallbacks do not become a chain of 400s.</li>\n</ol>\n<p>Three retry rules matter most:</p>\n<ul>\n<li><strong>Never retry a spend-cap 429,</strong> which signals a limit on total spending. Anthropic's errors page says a spend-cap 429 has no <code>retry-after</code> header and that retrying \"fails until access resumes.\" OpenAI's page says \"Retrying billing, spend, or quota errors won't restore API access.\" Mark that account unavailable and alert its owner.</li>\n<li><strong>Don't stack retries.</strong> Anthropic's SDK retries transient failures \"twice by default.\" Three gateway attempts, each with three SDK attempts, can reach a provider nine times: 3 × 3 = 9.</li>\n<li><strong>Cool down failing targets.</strong> LiteLLM can cool down one deployment right after a 429 and route around it.</li>\n</ul>\n<p>The first block is a fallback loop. It retries transient errors with backoff, honors <code>Retry-After</code>, and moves on when retrying cannot help.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> random</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> time</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">RETRYABLE</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">429</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">502</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">503</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">504</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">529</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># limits, overload, server errors</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> ProviderError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">Exception</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, status: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, retry_after: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                 spend_cap: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        super</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">().</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">__init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"HTTP </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">status</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.status </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> status</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.retry_after </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> retry_after   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># seconds, from the Retry-After header</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.spend_cap </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> spend_cap       </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># an account budget stop</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> call_with_fallback</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(targets, request, base_delay</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, max_tries</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Try (name, call) pairs in order. Retry transient errors on each one.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> name, call </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> targets:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> attempt </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(max_tries):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> name, call(request)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            except</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ProviderError </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> err:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> err.status </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">not</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RETRYABLE</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> or</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> err.spend_cap:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                    break</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                # 400, 401 and spend caps: move on</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> attempt </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> &#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_tries:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                    jitter </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> random.uniform(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                    wait </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> err.retry_after </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">or</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> base_delay </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">**</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">attempt </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> jitter</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                    time.sleep(wait)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"every target failed\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<p>The second block runs the loop on fake targets, so the output is fixed.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> fake_provider</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(statuses, answer):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"A fake target that fails with each status in turn, then answers.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    pending </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> list</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(statuses)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> call</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(request):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pending:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            raise</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ProviderError(pending.pop(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">answer</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">request</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> call</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">spare </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> fake_provider([], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"spare\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">overloaded </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> fake_provider([</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">503</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">503</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">503</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"primary\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># fails on every try</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">chain </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"big\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, overloaded), (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"small\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, spare)]</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(call_with_fallback(chain, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"hi\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">base_delay</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.01</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">limited </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> fake_provider([</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">429</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"limited\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)                </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># one 429, then fine</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(call_with_fallback([(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"big\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, limited)], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"hi\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">base_delay</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.01</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">strict </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> fake_provider([</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">400</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"strict\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)                  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># rejects this shape</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">chain </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"strict\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, strict), (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"flexible\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, spare)]</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(call_with_fallback(chain, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"hi\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">base_delay</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.01</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">only </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"only\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, fake_provider([</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">401</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"x\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    call_with_fallback(only, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"hi\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">base_delay</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.01</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">except</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> err:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(err)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Output:</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># ('small', 'spare: hi')</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># ('big', 'limited: hi')</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># ('flexible', 'spare: hi')</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># every target failed</span></span></code></pre>\n<p>A different model is not a drop-in replacement, since context windows, tool calling, JSON rules and tone differ. Keep a capability table per target, route only to targets that support the request, and run the evaluation set on each fallback first (see <a href=\"/learn/ai-evals-and-observability#running-evals-before-release\">running evals before release</a>). Ramp fallback traffic up gradually too: Anthropic warns that a sharp rise in usage can cause 429 errors from \"acceleration limits.\"</p>","headings":[{"id":"routing-and-fallbacks","title":"Routing and fallbacks","depth":3}]},{"id":"token-based-rate-limits","title":"Token-based rate limits","html":"<h3 id=\"token-based-rate-limits\">Token-based rate limits</h3>\n<p>Providers limit traffic per minute in requests and in tokens. OpenAI measures requests (RPM) and tokens (TPM). Anthropic measures requests, input tokens (ITPM) and output tokens (OTPM). OpenAI's rate-limit page warns that \"Rate limits can be hit across any of the options depending on what occurs first.\"</p>\n<p>Requests alone miss the point, because a 20-token question and a 4,000-token document both count as one request. Anthropic says its API uses \"the token bucket algorithm,\" which refills capacity continuously. The <a href=\"/learn/rate-limiting#token-bucket\">rate limiting chapter</a> explains the algorithm, and the gateway uses the same idea with tokens.</p>\n<p>The hard part is timing. When a request arrives, the gateway knows the prompt size but not the reply length, which is capped only by <code>max_tokens</code> if the app sets one. So the gateway works in two steps:</p>\n<ol>\n<li><strong>Reserve</strong> an estimate at admission: the prompt plus <code>max_tokens</code> (the worst case), or plus an expected reply.</li>\n<li><strong>Settle</strong> when the response ends: replace the reservation with the provider's real count, returning unused tokens or charging the extra.</li>\n</ol>\n<p>Providers count differently, so the gateway must match each rule. OpenAI's rate-limit page says a request counts the larger of <code>max_tokens</code> and an estimate from its characters, so the provider already counts the full <code>max_tokens</code>. Anthropic estimates input at the start and adjusts it during the request. It counts output \"in real time as output tokens are produced,\" and says <code>max_tokens</code> does not factor into its output limit. A gateway that uses one rule for both will either exceed a limit and collect 429s, or refuse work the provider would accept.</p>\n<p>Reservation size is a trade-off. Assume a key allows 100,000 tokens a minute, for this example only. Each request has a 2,000-token prompt and <code>max_tokens</code> of 2,000, and replies average 800 tokens.</p>\n<ul>\n<li>A worst-case reservation of 2,000 + 2,000 = 4,000 tokens admits 100,000 ÷ 4,000 = 25 requests a minute. Real use is 25 × 2,800 = 70,000 tokens, so 30,000 go unused.</li>\n<li>A prompt-only reservation of 2,000 admits 50 requests a minute. Real use would be 50 × 2,800 = 140,000 tokens, which is over the limit. The key runs out after about 100,000 ÷ 2,800 ≈ 35 requests, and 429s follow.</li>\n</ul>\n<p>The next block is a single-process token budget. <code>_refill</code> adds tokens in proportion to elapsed time, so no timer is needed.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> threading</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> time</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> TokenBudget</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"A token bucket that holds estimated tokens and settles them later.\"\"\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, tokens_per_minute: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, capacity: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.rate </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tokens_per_minute </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 60</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   # tokens refilled per second</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.capacity </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> capacity             </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the largest burst allowed</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.level </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> capacity                </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># tokens available right now</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.updated </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> time.monotonic()</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.lock </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> threading.Lock()</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> _refill</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        now </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> time.monotonic()</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        refilled </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.level </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (now </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.updated) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.rate</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.level </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.capacity, refilled)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.updated </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> now</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> reserve</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, estimate: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"\"\"Hold `estimate` tokens. Return 0.0 if held, else seconds to wait.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> estimate </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.capacity:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> ValueError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"estimate exceeds the bucket capacity\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        with</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.lock:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">            self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">._refill()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.level </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> estimate:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">                self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.level </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> estimate</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (estimate </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.level) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.rate</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> settle</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, reserved: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, actual: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"\"\"Swap the reservation for the real count. The level can go negative.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        with</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.lock:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">            self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">._refill()</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">            self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.level </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> reserved </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> actual</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">bucket </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> TokenBudget(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">tokens_per_minute</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">60_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">capacity</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">6_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Request 1 reserves 4,000: a 2,000-token prompt plus max_tokens of 2,000.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(bucket.reserve(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))                  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 0.0</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">bucket.settle(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">reserved</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">actual</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_200</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the reply was 200 tokens</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">round</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(bucket.level))                    </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 3800</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Request 2 reserves 2,500: the prompt plus an expected reply of 500.</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">bucket.reserve(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">bucket.settle(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">reserved</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">actual</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">6_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the reply ran to 4,000 tokens</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">round</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(bucket.level))                    </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># -2200: the bucket is in debt</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Request 3 waits for the debt to clear, at 1,000 tokens a second.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">round</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(bucket.reserve(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))        </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 4.7</span></span></code></pre>\n<p>The demo shows three cases. A short reply returns tokens. A long reply drives the level below zero, which the code calls debt, and the next request waits for the debt to clear. That wait is the value to send in <code>Retry-After</code>. An estimate larger than the bucket can never fit, so it is rejected at once.</p>\n<p>Three more rules matter in production:</p>\n<ul>\n<li><strong>Take both measures together.</strong> Reserve a request slot and a token allowance, and return the first if the second fails.</li>\n<li><strong>Share the counters.</strong> Per-instance counters let a client get N times the limit across N instances. Redis, a shared fast data store, is the usual fix. LiteLLM's docs say that without Redis, the proxy \"keeps enforcing limits per instance.\" The <a href=\"/learn/rate-limiting#distributed-counters-in-redis\">distributed counters section</a> covers the atomic updates. LiteLLM can return 503 when it cannot verify its counters, the fail-closed choice.</li>\n<li><strong>Spread limits across keys and workspaces.</strong> One team's key can run dry while another's has room. Anthropic's workspace limits sit under the organization's limit, which divides one account's capacity between teams.</li>\n</ul>","headings":[{"id":"token-based-rate-limits","title":"Token-based rate limits","depth":3}]},{"id":"caching-responses","title":"Caching responses","html":"<h3 id=\"caching-responses\">Caching responses</h3>\n<p>A gateway can save work at three levels, each with its own risks.</p>\n<p><strong>An exact-match cache</strong> stores full replies. The key must include everything that can change the answer, plus the <strong>tenant</strong> (the customer or team that owns the data), so one customer never receives another's reply. LiteLLM's docs show a custom key built from the model, messages, temperature and <code>logit_bias</code>.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> cache_key</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(tenant: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, request: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Exact-match key: every field that can change the reply goes in.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    fields </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {k: request.get(k) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">              (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"model\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"messages\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"temperature\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"max_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tools\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)}</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    canonical </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json.dumps(fields, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">sort_keys</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">separators</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\",\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\":\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    digest </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib.sha256(canonical.encode()).hexdigest()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">tenant</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">digest</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">base </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"model\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"small-chat\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"temperature\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"messages\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [{</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"role\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"user\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"content\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Store hours?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}]}</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">hotter </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">**</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">base, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"temperature\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(cache_key(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"team-a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, base) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cache_key(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"team-a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(base)))  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># True</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(cache_key(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"team-a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, base) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cache_key(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"team-b\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, base))        </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># False: tenant</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(cache_key(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"team-a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, base) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cache_key(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"team-a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, hotter))      </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># False: temperature</span></span></code></pre>\n<p>Set lifetimes per use case: opening hours can be cached for an hour, but one customer's order status should never be. LiteLLM's per-call controls include <code>ttl</code>, <code>no-cache</code> and <code>no-store</code>.</p>\n<figure class=\"diagram\" data-text=\"Application -> Gateway exact-match cache (request); Gateway exact-match cache <-> Redis or memory (lookup, store); Gateway exact-match cache -> Provider prompt cache (on a miss)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 720 323\" width=\"720\" role=\"img\" aria-label=\"Two caches with different owners: the gateway&#x27;s and the provider&#x27;s\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"102\" x2=\"247\" y2=\"102\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"320\" y1=\"133\" x2=\"320\" y2=\"248\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"391\" y1=\"102\" x2=\"567\" y2=\"102\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"78\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"106.5\">Application</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"76\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"98.5\">Gateway</tspan><tspan x=\"320\" y=\"114.5\">exact-match cache</tspan></text><path class=\"dg-node dg-store\" d=\"M252,260 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M252,260 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"289.5\">Redis or memory</tspan></text><rect class=\"dg-node dg-box\" x=\"572\" y=\"76\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"640\" y=\"98.5\">Provider</tspan><tspan x=\"640\" y=\"114.5\">prompt cache</tspan></text><rect class=\"dg-edge-bg\" x=\"172.3\" y=\"93\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"199\" y=\"106\" text-anchor=\"middle\">request</text><rect class=\"dg-edge-bg\" x=\"274.7\" y=\"181.5\" width=\"90.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"194.5\" text-anchor=\"middle\">lookup, store</text><rect class=\"dg-edge-bg\" x=\"446.1\" y=\"93\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"106\" text-anchor=\"middle\">on a miss</text></svg></div><figcaption>Two caches with different owners: the gateway's and the provider's</figcaption></figure>\n<p><strong>A provider prompt cache</strong> reuses the processed state of a prompt's start, called the <strong>prefix</strong>. Its rules are stricter than most people expect.</p>\n<ul>\n<li><strong>Matching is by prefix.</strong> Anthropic's docs require \"100% identical prompt segments\" up to the marked block, and changing any block at or before it \"produces a different hash.\" OpenAI's docs say \"Cache reuse requires the entire rendered prefix to match.\" Put stable content first, and put per-request values such as timestamps last.</li>\n<li><strong>Short prompts are skipped silently.</strong> Anthropic's minimum runs from 512 to 4,096 tokens depending on the model. Shorter prompts \"will be processed without caching, and no error is returned.\" Log cached-token counts so the misses show up.</li>\n<li><strong>Lifetimes and prices differ.</strong> Anthropic's default lifetime is 5 minutes, refreshed on each use. OpenAI's newest models keep a prefix eligible for reuse for 30 minutes after its last write or reuse. As of October 2026, Anthropic lists cache reads at 0.1 times the base input price for most models and 5-minute cache writes at 1.25 times. OpenAI says cached input is \"discounted up to 95%.\" These are ratios from the docs at the time of writing, so check the current pages before you forecast.</li>\n<li><strong>Scope caches per customer.</strong> Anthropic isolates caches per workspace and never shares them across organizations. OpenAI's docs say separate cache keys \"help prevent cache-hit probing across users,\" since a faster response can reveal that someone recently sent the same prefix.</li>\n</ul>\n<p>The gateway's job is to keep prompts stable and to report cache hit rates for each team.</p>\n<p><strong>A semantic cache</strong> matches on meaning. It turns the question into an <strong>embedding</strong>, a list of numbers that represents the text's meaning (see <a href=\"/learn/vector-search\">vector search</a>), and returns a stored reply when a similar question clears a threshold. Hit rates can be high, and so can wrong answers. In a November 2025 InfoQ article, a production system matched \"I don't want this business account anymore\" to automatic payment cancellation steps at 84.9% confidence. The team reported a 99% false-positive rate for some models at a threshold of 0.7. Start with exact matches. Add semantic caching only for narrow, frequent questions, tuned on labeled pairs (see <a href=\"/learn/ai-evals-and-observability#building-an-eval-set\">building an eval set</a>) and scoped per tenant.</p>","headings":[{"id":"caching-responses","title":"Caching responses","depth":3}]},{"id":"streaming-through-a-gateway","title":"Streaming through a gateway","html":"<h3 id=\"streaming-through-a-gateway\">Streaming through a gateway</h3>\n<p>Most chat products stream: the provider sends small events over server-sent events (SSE). OpenAI's streaming guide describes \"HTTP streaming (stream=true) over server-sent events (SSE),\" and Anthropic's docs use <code>\"stream\": true</code>. Users feel the <strong>time to first token</strong>, the wait before the first word appears, and then read as the rest arrives (see <a href=\"/learn/llm-inference#latency-and-throughput-metrics\">latency and throughput metrics</a> and <a href=\"/learn/realtime#server-sent-events-sse\">SSE</a>).</p>\n<figure class=\"diagram\" data-text=\"Provider streams events -> Gateway forwards, counts (SSE events); Gateway forwards, counts -> Browser shows text (as they arrive); Gateway forwards, counts -> Token ledger (settle when done)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 323\" width=\"800\" role=\"img\" aria-label=\"The gateway forwards each event as it arrives and settles at the end\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"102\" x2=\"327\" y2=\"102\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"102\" x2=\"647\" y2=\"102\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"400\" y1=\"131\" x2=\"400\" y2=\"248\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"76\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"98.5\">Provider</tspan><tspan x=\"80\" y=\"114.5\">streams events</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"76\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"98.5\">Gateway</tspan><tspan x=\"400\" y=\"114.5\">forwards, counts</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"76\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"98.5\">Browser</tspan><tspan x=\"720\" y=\"114.5\">shows text</tspan></text><path class=\"dg-node dg-store\" d=\"M332,260 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,260 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"289.5\">Token ledger</tspan></text><rect class=\"dg-edge-bg\" x=\"203\" y=\"93\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"239\" y=\"106\" text-anchor=\"middle\">SSE events</text><rect class=\"dg-edge-bg\" x=\"510.6\" y=\"93\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"559\" y=\"106\" text-anchor=\"middle\">as they arrive</text><rect class=\"dg-edge-bg\" x=\"345.4\" y=\"180.5\" width=\"109.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"400\" y=\"193.5\" text-anchor=\"middle\">settle when done</text></svg></div><figcaption>The gateway forwards each event as it arrives and settles at the end</figcaption></figure>\n<p>Streaming creates three problems for a proxy:</p>\n<ul>\n<li><strong>Buffering.</strong> NGINX buffers responses by default. With <code>proxy_buffering off</code>, its docs say the response \"is passed to a client synchronously, immediately as it is received.\" Otherwise users see nothing, then everything at once. A provider can also send <code>X-Accel-Buffering: no</code> for one response.</li>\n<li><strong>Idle timeouts.</strong> An idle timeout caps how long a connection may stay silent. NGINX's <code>proxy_read_timeout</code> defaults to 60 seconds \"between two successive read operations,\" not for the whole response, so a steady stream is safe. Envoy's route timeout defaults to 15 seconds, which its docs call \"typically a problem for streaming responses,\" so streaming routes need an idle timeout instead. Forward the <code>ping</code> events some providers send during long pauses, since they count as activity.</li>\n<li><strong>Errors after the first byte.</strong> Once the first event is out, the status is fixed at 200, yet Anthropic's errors page says an error \"can occur after the API returns a 200 response.\" The gateway can end the stream with its own error event, retry on another target only if nothing reached the user, or resume from the partial text, as Anthropic's recovery guidance describes. Resuming can repeat or contradict text already read, so use it only where the product tolerates that.</li>\n</ul>\n<p>Usage arrives inside the stream too. Anthropic's <code>message_start</code> carries the input token count, and each <code>message_delta</code> carries a cumulative output count. The relay below forwards events as they arrive and settles in a <code>finally</code> block, so a client that disconnects still releases its budget.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> relay</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(events, budget, reserved, send):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Forward streamed events as they arrive, then settle the reservation.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    actual </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> reserved                  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># cut short: charge all of it</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    tokens_in </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> event </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> events:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> event[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"message_start\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                tokens_in </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> event[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"message\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">][</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"usage\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">][</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            elif</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> event[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"message_delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                # The output count is cumulative, so the last one wins.</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                actual </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tokens_in </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> event[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"usage\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">][</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"output_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            send(event)                </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># no buffering: the client sees it now</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    finally</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        budget.settle(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">reserved</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">reserved, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">actual</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">actual)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">events </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"message_start\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"message\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"usage\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}}},</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"content_block_delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Hello\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}},</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"message_delta\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"usage\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"output_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">15</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}},</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"message_stop\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">},</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">budget </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> TokenBudget(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">tokens_per_minute</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">60_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">capacity</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">20_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">budget.reserve(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)                  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 500 prompt tokens plus max_tokens of 2,000</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">relay(events, budget, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">send</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=lambda</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> e: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(e[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]))  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># prints four names</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">round</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(budget.level))             </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 19485</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">budget.reserve(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">relay(events[:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], budget, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2_500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">send</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=lambda</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> e: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># cut off before message_delta</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">round</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(budget.level))             </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 16985: the whole reservation is kept</span></span></code></pre>\n<p>The relay must pass unknown event types along, since Anthropic says code \"should handle unknown event types gracefully.\" Streams also hold connections open. By <a href=\"/learn/performance-and-latency#littles-law\">Little's law</a>, 100 new streams a second that each last 20 seconds keep 100 × 20 = 2,000 streams open at once, so size the gateway's connection limits for that.</p>","headings":[{"id":"streaming-through-a-gateway","title":"Streaming through a gateway","depth":3}]},{"id":"tracking-cost","title":"Tracking cost","html":"<h3 id=\"tracking-cost\">Tracking cost</h3>\n<p>Cost comes from the provider's usage report. The gateway's estimate only drives admission. The formula is:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>cost = uncached input × input price + cache reads × read price</span></span>\n<span class=\"line\"><span>     + cache writes × write price + output × output price</span></span></code></pre>\n<p>Assume $3.00 per million input tokens and $15.00 per million output tokens. These are assumptions, not quotes, and real prices vary by provider, model and commitment. A cache read costs 0.1 times the input price, or $0.30 per million. A request with 1,000 uncached input tokens, 5,000 cache-read tokens and 500 output tokens costs:</p>\n<ul>\n<li>Uncached input: 1,000 × $3.00 ÷ 1,000,000 = $0.0030</li>\n<li>Cache reads: 5,000 × $0.30 ÷ 1,000,000 = $0.0015</li>\n<li>Output: 500 × $15.00 ÷ 1,000,000 = $0.0075</li>\n<li>Total: $0.0120</li>\n</ul>\n<p>Without the cache, the 6,000 input tokens would cost $0.0180, so the cache cut the input cost by 75%.</p>\n<p>Write one ledger row per request, with tokens by kind, team, target, status and a <code>price_version</code>, so old reports can be reproduced after prices change.</p>\n<p>Budgets need reservations too. Without one, 50 requests in flight can overshoot a team's budget by the cost of 50 requests. LiteLLM supports budgets per key, user and team, reset by a <code>budget_duration</code>. Its docs warn that the default <code>max_budget</code> is null and \"is not checked for keys,\" so test that setting before you rely on it. Charging each team from the ledger every month is <strong>chargeback</strong>: billing each team for what it used.</p>","headings":[{"id":"tracking-cost","title":"Tracking cost","depth":3}]},{"id":"logging-and-guardrails","title":"Logging and guardrails","html":"<h3 id=\"logging-and-guardrails\">Logging and guardrails</h3>\n<p>A gateway sees every prompt and reply, which makes its logs sensitive. Log what you need to operate it: the gateway's request ID and the provider's (Anthropic's errors page says every response carries a <code>request-id</code> header), plus team, alias, target, fallback path, token counts, cache hits, time to first token and error type. Do not log full prompts by default. If you must keep them, encrypt them, keep them briefly and redact them first.</p>\n<p><strong>Redaction</strong> removes personal data before a prompt leaves your network. Names, email addresses, card numbers and phone numbers become placeholders. Microsoft Presidio is an MIT-licensed, open-source SDK whose README describes \"fast identification and anonymization modules for private entities in text.\" Detectors miss some items and flag others, so test on your own traffic. The <a href=\"/learn/ai-security#leaking-data\">security chapter</a> covers the risk.</p>\n<p><strong>Guardrails</strong> check both directions: size limits, injection screening and policy (which teams may use which models) on the way in, and moderation and format checks on the way out. OWASP's entry on unbounded consumption recommends input size limits, rate limits, quotas and timeouts. Run cheap checks first, and measure their latency apart from the model's time to first token (see <a href=\"/learn/ai-security#abuse-and-content-safety\">abuse and content safety</a>).</p>","headings":[{"id":"logging-and-guardrails","title":"Logging and guardrails","depth":3}]},{"id":"build-or-buy","title":"Build or buy","html":"<h3 id=\"build-or-buy\">Build or buy</h3>\n<ul>\n<li><strong>Build a thin layer:</strong> auth, virtual keys, token budgets, one adapter per provider and a ledger. You get control, but you own every provider change, so each adapter needs tests against the live API.</li>\n<li><strong>Run an open-source proxy:</strong> LiteLLM provides virtual keys, budgets, per-minute limits, routing, fallbacks and caching. It needs Redis for shared counters, and some features, such as per-model budgets, need a commercial enterprise license.</li>\n<li><strong>Run Agent Router:</strong> formerly Envoy AI Gateway, it joined the Agentic AI Foundation in September 2026. Its site calls it \"powered by Envoy,\" and version 1.2 is the first release under the new name. Check its documentation for token-based limits and cost tracking in the version you adopt.</li>\n<li><strong>Buy a managed gateway:</strong> Cloudflare's AI Gateway reports requests, tokens and cost, and offers caching, rate limiting, retries and fallbacks. Cloudflare hosts it, so prompts pass through its network. Check the data terms first.</li>\n</ul>\n<p>A rough rule: build when you have one or two providers and a small team. Adopt an open-source proxy with several providers, budgeted teams and people who can run Redis. Buy when the data rules allow it and nobody wants to run a proxy. Either way, keep the interface the apps call stable, so the gateway can change later.</p>","headings":[{"id":"build-or-buy","title":"Build or buy","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Your key allows 100,000 tokens a minute (assumed). Requests have a 2,000-token prompt and max_tokens of 2,000, and replies average 800 tokens. How many requests a minute does a worst-case reservation admit, and how many does a prompt-only reservation admit? Which is safe?</summary>\n<p>A worst-case reservation holds 4,000 tokens, so the key admits 100,000 ÷ 4,000 = 25 requests a minute. A prompt-only reservation admits 50, but each request really uses about 2,800 tokens, so 50 × 2,800 = 140,000 is over the limit. The worst case is the safe one, and settlement returns its unused part.</p>\n</details>\n<details>\n<summary>A provider returns 429 with error type rate_limit_error and no retry-after header. The gateway retries three times with backoff, then fails the request. What went wrong?</summary>\n<p>The 429 is probably a spend cap. Anthropic's errors page says a spend-cap 429 has no retry-after header and that retrying fails until access resumes. OpenAI's error page says billing and quota errors are not fixed by retrying. The gateway should recognize the cap, stop sending to that account, alert its owner and fall back.</p>\n</details>\n<details>\n<summary>A streamed answer has already shown 40 tokens when the provider sends an overloaded error event, and a fallback target exists. What are the options, and what does each cost?</summary>\n<p>The status is already 200, so it cannot signal the error. The gateway can end the stream with its own error event, which leaves a partial answer. It can restart on the fallback, which repeats text the user already saw. Or it can resume from the partial text, which risks repeating or contradicting it. For structured output such as JSON, ending the stream and letting the client retry is usually safer.</p>\n</details>\n<details>\n<summary>Assume $3.00 per million input tokens, $15.00 per million output tokens, and cache reads at 0.1 times the input price. A request has 1,000 uncached input tokens, 5,000 cache-read tokens and 500 output tokens. What does it cost?</summary>\n<p>Uncached input: 1,000 × $3.00 ÷ 1,000,000 = $0.0030. Cache reads: 5,000 × $0.30 ÷ 1,000,000 = $0.0015. Output: 500 × $15.00 ÷ 1,000,000 = $0.0075. Total: $0.0120.</p>\n</details>\n<details>\n<summary>A semantic cache with a threshold of 0.7 sends \"I don't want this business account anymore\" to the steps for cancelling automatic payments. What happened, and what would you change?</summary>\n<p>Embedding similarity measures closeness of wording and topic, not intent, so the two questions cleared the threshold. Tune the threshold on labeled pairs that should and should not match, scope the cache per tenant, and use exact matches alone where a wrong answer is costly.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://platform.claude.com/docs/en/api/rate-limits\" target=\"_blank\" rel=\"noopener noreferrer\">Rate limits (Claude Platform docs)</a>: the token bucket, input and output token limits, and spend-cap 429s, as of October 2026.</li>\n<li><a href=\"https://platform.claude.com/docs/en/build-with-claude/prompt-caching\" target=\"_blank\" rel=\"noopener noreferrer\">Prompt caching (Claude Platform docs)</a>: breakpoints, lifetimes, minimum lengths, price ratios and workspace isolation.</li>\n<li><a href=\"https://platform.claude.com/docs/en/build-with-claude/streaming\" target=\"_blank\" rel=\"noopener noreferrer\">Streaming messages (Claude Platform docs)</a>: event order, cumulative usage, ping and error events, and recovery.</li>\n<li><a href=\"https://platform.claude.com/docs/en/api/errors\" target=\"_blank\" rel=\"noopener noreferrer\">Errors (Claude Platform docs)</a>: which status codes to retry, and the SDK's retry defaults.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/rate-limits\" target=\"_blank\" rel=\"noopener noreferrer\">Rate limits (OpenAI developer docs)</a>: RPM and TPM, and how a request's token count is estimated.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/prompt-caching\" target=\"_blank\" rel=\"noopener noreferrer\">Prompt caching (OpenAI developer docs)</a>: minimum lengths, prefix matching, lifetimes and cache-hit probing.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/error-codes\" target=\"_blank\" rel=\"noopener noreferrer\">Error codes (OpenAI developer docs)</a>: what 429 and 503 mean, and which errors retrying cannot fix.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/streaming-responses\" target=\"_blank\" rel=\"noopener noreferrer\">Streaming responses (OpenAI developer docs)</a>: HTTP streaming over server-sent events.</li>\n<li><a href=\"https://docs.vllm.ai/en/latest/serving/online_serving/\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM OpenAI-compatible server</a>: the Chat Completions endpoint.</li>\n<li><a href=\"https://docs.litellm.ai/docs/routing\" target=\"_blank\" rel=\"noopener noreferrer\">LiteLLM routing</a> and <a href=\"https://docs.litellm.ai/docs/proxy/users\" target=\"_blank\" rel=\"noopener noreferrer\">virtual keys and budgets</a>: routing, fallbacks, cooldowns and budgets.</li>\n<li><a href=\"https://docs.litellm.ai/docs/caching/all_caches\" target=\"_blank\" rel=\"noopener noreferrer\">LiteLLM caching</a>: cache backends, per-call controls and similarity thresholds.</li>\n<li><a href=\"https://theagentrouter.ai/\" target=\"_blank\" rel=\"noopener noreferrer\">Agent Router</a> and <a href=\"https://aaif.io/blog/agent-router-joins-aaif\" target=\"_blank\" rel=\"noopener noreferrer\">the foundation's announcement</a>: the rename and its first release under the new name.</li>\n<li><a href=\"https://developers.cloudflare.com/ai-gateway/\" target=\"_blank\" rel=\"noopener noreferrer\">Cloudflare AI Gateway</a>: a managed gateway's analytics, caching and fallbacks.</li>\n<li><a href=\"https://www.envoyproxy.io/docs/envoy/latest/faq/configuration/timeouts\" target=\"_blank\" rel=\"noopener noreferrer\">Envoy timeouts</a> and <a href=\"https://nginx.org/en/docs/http/ngx_http_proxy_module.html\" target=\"_blank\" rel=\"noopener noreferrer\">NGINX proxy module</a>: timeouts and buffering for streams.</li>\n<li><a href=\"https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/\" target=\"_blank\" rel=\"noopener noreferrer\">OWASP LLM10: Unbounded Consumption (2025)</a>: denial of wallet, meaning runaway spending, and its mitigations.</li>\n<li><a href=\"https://github.com/microsoft/presidio\" target=\"_blank\" rel=\"noopener noreferrer\">Microsoft Presidio</a>: an open-source, MIT-licensed SDK for finding and anonymizing private data.</li>\n<li><a href=\"https://www.infoq.com/articles/reducing-false-positives-retrieval-augmented-generation/\" target=\"_blank\" rel=\"noopener noreferrer\">InfoQ on false positives in retrieval (November 2025)</a>: confident wrong matches from similarity search.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}