System Design Handbook · llms.txt
A model provider sells access to a language model through an HTTP API. A gateway sits between your applications and those providers. Each application calls one internal endpoint, and the gateway chooses which provider or model answers. An LLM gateway is a specialized API gateway. That chapter covers TLS, authentication and per-key limits, so this one covers what changes for model traffic:
This chapter answers five questions:
We'll follow Crumb, the recipe site from earlier chapters. Its recipe helper answers cooking questions with a hosted model. The search team wants a second provider as a backup, and the data team runs a small open-weight model (one whose weights are published for download) on its own GPUs. Without a gateway, each service would need its own keys, retries, token counts and logs. The LLM gateway design question sizes the same parts for a company with 200 apps.
Without a gateway, each Crumb service calls a provider directly, holds its own key, retries in its own way and writes its own logs. Rotating a key means changing every service. With a gateway, each service makes one kind of call to one internal address:
The gateway does three jobs:
/v1/chat/completions for text models with a chat template. Agent Router, formerly Envoy AI Gateway, describes its aim as "a stable, OpenAI-compatible interface to models." The gateway translates to a provider's native format when it must.The gateway adds a network hop and some processing time, which you should measure. It cannot make a weak model answer well or stop a malicious prompt on its own. It is one layer among several.
Routing chooses a target (a provider, a model and a key) for each request. A fallback is the next target to try. Apps request an alias such as chat-large, a name the gateway maps to a list of targets, so moving traffic off a provider means editing that list.
Let the status code decide how to react:
Retry-After if present. Anthropic's rate-limit page says to "wait that many seconds before retrying." Without the header, OpenAI's error page recommends exponential backoff with jitter. If the 429 persists, fall back.Three retry rules matter most:
retry-after header and that retrying "fails until access resumes." OpenAI's page says "Retrying billing, spend, or quota errors won't restore API access." Mark that account unavailable and alert its owner.The first block is a fallback loop. It retries transient errors with backoff, honors Retry-After, and moves on when retrying cannot help.
import random
import time
RETRYABLE = {429, 500, 502, 503, 504, 529} # limits, overload, server errors
class ProviderError(Exception):
def __init__(self, status: int, retry_after: float = 0.0,
spend_cap: bool = False):
super().__init__(f"HTTP {status}")
self.status = status
self.retry_after = retry_after # seconds, from the Retry-After header
self.spend_cap = spend_cap # an account budget stop
def call_with_fallback(targets, request, base_delay=0.5, max_tries=3):
"""Try (name, call) pairs in order. Retry transient errors on each one."""
for name, call in targets:
for attempt in range(max_tries):
try:
return name, call(request)
except ProviderError as err:
if err.status not in RETRYABLE or err.spend_cap:
break # 400, 401 and spend caps: move on
if attempt + 1 < max_tries:
jitter = random.uniform(0.5, 1)
wait = err.retry_after or base_delay * 2**attempt * jitter
time.sleep(wait)
raise RuntimeError("every target failed")
The second block runs the loop on fake targets, so the output is fixed.
def fake_provider(statuses, answer):
"""A fake target that fails with each status in turn, then answers."""
pending = list(statuses)
def call(request):
if pending:
raise ProviderError(pending.pop(0))
return f"{answer}: {request}"
return call
spare = fake_provider([], "spare")
overloaded = fake_provider([503, 503, 503], "primary") # fails on every try
chain = [("big", overloaded), ("small", spare)]
print(call_with_fallback(chain, "hi", base_delay=0.01))
limited = fake_provider([429], "limited") # one 429, then fine
print(call_with_fallback([("big", limited)], "hi", base_delay=0.01))
strict = fake_provider([400], "strict") # rejects this shape
chain = [("strict", strict), ("flexible", spare)]
print(call_with_fallback(chain, "hi", base_delay=0.01))
only = [("only", fake_provider([401], "x"))]
try:
call_with_fallback(only, "hi", base_delay=0.01)
except RuntimeError as err:
print(err)
# Output:
# ('small', 'spare: hi')
# ('big', 'limited: hi')
# ('flexible', 'spare: hi')
# every target failed
A different model is not a drop-in replacement, since context windows, tool calling, JSON rules and tone differ. Keep a capability table per target, route only to targets that support the request, and run the evaluation set on each fallback first (see running evals before release). Ramp fallback traffic up gradually too: Anthropic warns that a sharp rise in usage can cause 429 errors from "acceleration limits."
Providers limit traffic per minute in requests and in tokens. OpenAI measures requests (RPM) and tokens (TPM). Anthropic measures requests, input tokens (ITPM) and output tokens (OTPM). OpenAI's rate-limit page warns that "Rate limits can be hit across any of the options depending on what occurs first."
Requests alone miss the point, because a 20-token question and a 4,000-token document both count as one request. Anthropic says its API uses "the token bucket algorithm," which refills capacity continuously. The rate limiting chapter explains the algorithm, and the gateway uses the same idea with tokens.
The hard part is timing. When a request arrives, the gateway knows the prompt size but not the reply length, which is capped only by max_tokens if the app sets one. So the gateway works in two steps:
max_tokens (the worst case), or plus an expected reply.Providers count differently, so the gateway must match each rule. OpenAI's rate-limit page says a request counts the larger of max_tokens and an estimate from its characters, so the provider already counts the full max_tokens. Anthropic estimates input at the start and adjusts it during the request. It counts output "in real time as output tokens are produced," and says max_tokens does not factor into its output limit. A gateway that uses one rule for both will either exceed a limit and collect 429s, or refuse work the provider would accept.
Reservation size is a trade-off. Assume a key allows 100,000 tokens a minute, for this example only. Each request has a 2,000-token prompt and max_tokens of 2,000, and replies average 800 tokens.
The next block is a single-process token budget. _refill adds tokens in proportion to elapsed time, so no timer is needed.
import threading
import time
class TokenBudget:
"""A token bucket that holds estimated tokens and settles them later."""
def __init__(self, tokens_per_minute: float, capacity: float):
self.rate = tokens_per_minute / 60 # tokens refilled per second
self.capacity = capacity # the largest burst allowed
self.level = capacity # tokens available right now
self.updated = time.monotonic()
self.lock = threading.Lock()
def _refill(self) -> None:
now = time.monotonic()
refilled = self.level + (now - self.updated) * self.rate
self.level = min(self.capacity, refilled)
self.updated = now
def reserve(self, estimate: int) -> float:
"""Hold `estimate` tokens. Return 0.0 if held, else seconds to wait."""
if estimate > self.capacity:
raise ValueError("estimate exceeds the bucket capacity")
with self.lock:
self._refill()
if self.level >= estimate:
self.level -= estimate
return 0.0
return (estimate - self.level) / self.rate
def settle(self, reserved: int, actual: int) -> None:
"""Swap the reservation for the real count. The level can go negative."""
with self.lock:
self._refill()
self.level += reserved - actual
bucket = TokenBudget(tokens_per_minute=60_000, capacity=6_000)
# Request 1 reserves 4,000: a 2,000-token prompt plus max_tokens of 2,000.
print(bucket.reserve(4_000)) # 0.0
bucket.settle(reserved=4_000, actual=2_200) # the reply was 200 tokens
print(round(bucket.level)) # 3800
# Request 2 reserves 2,500: the prompt plus an expected reply of 500.
bucket.reserve(2_500)
bucket.settle(reserved=2_500, actual=6_000) # the reply ran to 4,000 tokens
print(round(bucket.level)) # -2200: the bucket is in debt
# Request 3 waits for the debt to clear, at 1,000 tokens a second.
print(round(bucket.reserve(2_500), 1)) # 4.7
The demo shows three cases. A short reply returns tokens. A long reply drives the level below zero, which the code calls debt, and the next request waits for the debt to clear. That wait is the value to send in Retry-After. An estimate larger than the bucket can never fit, so it is rejected at once.
Three more rules matter in production:
A gateway can save work at three levels, each with its own risks.
An exact-match cache stores full replies. The key must include everything that can change the answer, plus the tenant (the customer or team that owns the data), so one customer never receives another's reply. LiteLLM's docs show a custom key built from the model, messages, temperature and logit_bias.
import hashlib
import json
def cache_key(tenant: str, request: dict) -> str:
"""Exact-match key: every field that can change the reply goes in."""
fields = {k: request.get(k) for k in
("model", "messages", "temperature", "max_tokens", "tools")}
canonical = json.dumps(fields, sort_keys=True, separators=(",", ":"))
digest = hashlib.sha256(canonical.encode()).hexdigest()
return f"{tenant}:{digest}"
base = {"model": "small-chat", "temperature": 0,
"messages": [{"role": "user", "content": "Store hours?"}]}
hotter = {**base, "temperature": 0.9}
print(cache_key("team-a", base) == cache_key("team-a", dict(base))) # True
print(cache_key("team-a", base) == cache_key("team-b", base)) # False: tenant
print(cache_key("team-a", base) == cache_key("team-a", hotter)) # False: temperature
Set lifetimes per use case: opening hours can be cached for an hour, but one customer's order status should never be. LiteLLM's per-call controls include ttl, no-cache and no-store.
A provider prompt cache reuses the processed state of a prompt's start, called the prefix. Its rules are stricter than most people expect.
The gateway's job is to keep prompts stable and to report cache hit rates for each team.
A semantic cache matches on meaning. It turns the question into an embedding, a list of numbers that represents the text's meaning (see vector search), and returns a stored reply when a similar question clears a threshold. Hit rates can be high, and so can wrong answers. In a November 2025 InfoQ article, a production system matched "I don't want this business account anymore" to automatic payment cancellation steps at 84.9% confidence. The team reported a 99% false-positive rate for some models at a threshold of 0.7. Start with exact matches. Add semantic caching only for narrow, frequent questions, tuned on labeled pairs (see building an eval set) and scoped per tenant.
Most chat products stream: the provider sends small events over server-sent events (SSE). OpenAI's streaming guide describes "HTTP streaming (stream=true) over server-sent events (SSE)," and Anthropic's docs use "stream": true. Users feel the time to first token, the wait before the first word appears, and then read as the rest arrives (see latency and throughput metrics and SSE).
Streaming creates three problems for a proxy:
proxy_buffering off, its docs say the response "is passed to a client synchronously, immediately as it is received." Otherwise users see nothing, then everything at once. A provider can also send X-Accel-Buffering: no for one response.proxy_read_timeout defaults to 60 seconds "between two successive read operations," not for the whole response, so a steady stream is safe. Envoy's route timeout defaults to 15 seconds, which its docs call "typically a problem for streaming responses," so streaming routes need an idle timeout instead. Forward the ping events some providers send during long pauses, since they count as activity.Usage arrives inside the stream too. Anthropic's message_start carries the input token count, and each message_delta carries a cumulative output count. The relay below forwards events as they arrive and settles in a finally block, so a client that disconnects still releases its budget.
def relay(events, budget, reserved, send):
"""Forward streamed events as they arrive, then settle the reservation."""
actual = reserved # cut short: charge all of it
tokens_in = 0
try:
for event in events:
if event["type"] == "message_start":
tokens_in = event["message"]["usage"]["input_tokens"]
elif event["type"] == "message_delta":
# The output count is cumulative, so the last one wins.
actual = tokens_in + event["usage"]["output_tokens"]
send(event) # no buffering: the client sees it now
finally:
budget.settle(reserved=reserved, actual=actual)
events = [
{"type": "message_start", "message": {"usage": {"input_tokens": 500}}},
{"type": "content_block_delta", "delta": {"text": "Hello"}},
{"type": "message_delta", "usage": {"output_tokens": 15}},
{"type": "message_stop"},
]
budget = TokenBudget(tokens_per_minute=60_000, capacity=20_000)
budget.reserve(2_500) # 500 prompt tokens plus max_tokens of 2,000
relay(events, budget, 2_500, send=lambda e: print(e["type"])) # prints four names
print(round(budget.level)) # 19485
budget.reserve(2_500)
relay(events[:2], budget, 2_500, send=lambda e: None) # cut off before message_delta
print(round(budget.level)) # 16985: the whole reservation is kept
The relay must pass unknown event types along, since Anthropic says code "should handle unknown event types gracefully." Streams also hold connections open. By Little's law, 100 new streams a second that each last 20 seconds keep 100 × 20 = 2,000 streams open at once, so size the gateway's connection limits for that.
Cost comes from the provider's usage report. The gateway's estimate only drives admission. The formula is:
cost = uncached input × input price + cache reads × read price
+ cache writes × write price + output × output price
Assume $3.00 per million input tokens and $15.00 per million output tokens. These are assumptions, not quotes, and real prices vary by provider, model and commitment. A cache read costs 0.1 times the input price, or $0.30 per million. A request with 1,000 uncached input tokens, 5,000 cache-read tokens and 500 output tokens costs:
Without the cache, the 6,000 input tokens would cost $0.0180, so the cache cut the input cost by 75%.
Write one ledger row per request, with tokens by kind, team, target, status and a price_version, so old reports can be reproduced after prices change.
Budgets need reservations too. Without one, 50 requests in flight can overshoot a team's budget by the cost of 50 requests. LiteLLM supports budgets per key, user and team, reset by a budget_duration. Its docs warn that the default max_budget is null and "is not checked for keys," so test that setting before you rely on it. Charging each team from the ledger every month is chargeback: billing each team for what it used.
A gateway sees every prompt and reply, which makes its logs sensitive. Log what you need to operate it: the gateway's request ID and the provider's (Anthropic's errors page says every response carries a request-id header), plus team, alias, target, fallback path, token counts, cache hits, time to first token and error type. Do not log full prompts by default. If you must keep them, encrypt them, keep them briefly and redact them first.
Redaction removes personal data before a prompt leaves your network. Names, email addresses, card numbers and phone numbers become placeholders. Microsoft Presidio is an MIT-licensed, open-source SDK whose README describes "fast identification and anonymization modules for private entities in text." Detectors miss some items and flag others, so test on your own traffic. The security chapter covers the risk.
Guardrails check both directions: size limits, injection screening and policy (which teams may use which models) on the way in, and moderation and format checks on the way out. OWASP's entry on unbounded consumption recommends input size limits, rate limits, quotas and timeouts. Run cheap checks first, and measure their latency apart from the model's time to first token (see abuse and content safety).
A rough rule: build when you have one or two providers and a small team. Adopt an open-source proxy with several providers, budgeted teams and people who can run Redis. Buy when the data rules allow it and nobody wants to run a proxy. Either way, keep the interface the apps call stable, so the gateway can change later.
A worst-case reservation holds 4,000 tokens, so the key admits 100,000 ÷ 4,000 = 25 requests a minute. A prompt-only reservation admits 50, but each request really uses about 2,800 tokens, so 50 × 2,800 = 140,000 is over the limit. The worst case is the safe one, and settlement returns its unused part.
The 429 is probably a spend cap. Anthropic's errors page says a spend-cap 429 has no retry-after header and that retrying fails until access resumes. OpenAI's error page says billing and quota errors are not fixed by retrying. The gateway should recognize the cap, stop sending to that account, alert its owner and fall back.
The status is already 200, so it cannot signal the error. The gateway can end the stream with its own error event, which leaves a partial answer. It can restart on the fallback, which repeats text the user already saw. Or it can resume from the partial text, which risks repeating or contradicting it. For structured output such as JSON, ending the stream and letting the client retry is usually safer.
Uncached input: 1,000 × $3.00 ÷ 1,000,000 = $0.0030. Cache reads: 5,000 × $0.30 ÷ 1,000,000 = $0.0015. Output: 500 × $15.00 ÷ 1,000,000 = $0.0075. Total: $0.0120.
Embedding similarity measures closeness of wording and topic, not intent, so the two questions cleared the threshold. Tune the threshold on labeled pairs that should and should not match, scope the cache per tenant, and use exact matches alone where a wrong answer is costly.
Previous: Running inference at scale · Next: Embeddings and vector search