System Design Handbook · llms.txt
A chat assistant answers each message with a reply that appears word by word. The first words should arrive within about a second, and the rest keep coming while the reader follows along. The interface looks like a messaging app, but each reply needs a large model to read the whole conversation and generate new tokens, and the GPUs that run the model are the scarce, expensive resource.
The chapter builds on the handbook's AI part: how LLM inference works, inference serving engines, running inference at scale and LLM gateways. For history, devices and sync, see the chat system chapter.
Functional requirements:
Non-functional requirements:
These invented assumptions describe a large consumer app:
Messages. Free users send 18 million messages a day and paid users send 8 million, 26 million in all. That is 26,000,000 ÷ 86,400 s ≈ 301 messages a second on average and about 903 at peak. At six messages per conversation, there are about 4.3 million conversations a day.
Model mix. Paid users send 75% of their messages to the Large model, and free users send 25%. Large gets 6 million + 4.5 million = 10.5 million messages a day, and Fast gets 15.5 million. At peak, Large takes 3 × 10.5 million ÷ 86,400 ≈ 365 messages a second, and Fast takes 538.
Prompt size. For the k-th message in a conversation, the earlier turns are k − 1 exchanges of 60 + 350 = 410 tokens. Averaged over six messages, the history is 2.5 × 410 = 1,025 tokens, so the average prompt is 1,200 + 1,025 + 60 = 2,285 tokens.
Token rates at peak. Large generates 365 × 350 ≈ 128,000 tokens a second and Fast generates 538 × 350 ≈ 188,000, about 316,000 output tokens a second in all. Without caching, prefill would need 903 × 2,285 ≈ 2.06 million prompt tokens a second.
Cache hits at peak. Prefix caching skips the prefill for a prompt's start when the replica still holds it. The running example's 148 GB cache holds 148 × 10⁹ ÷ 327,680 ≈ 451,660 tokens, about 196 conversations of 2,300 tokens. At peak each Large replica takes about 8 new messages a second, so an idle conversation's cache survives about 196 ÷ 8 ≈ 24 seconds. Assume most users pause longer than that, so only a quarter of history tokens hit at peak, while the system prompt always hits. That gives 1,200 + 0.25 × 1,025 ≈ 1,456 cached tokens (64% of the prompt) and 829 uncached tokens to prefill.
Replicas. A replica is one copy of a model on its own GPUs, and a pool is a group of identical replicas. A Large replica runs the 70B model on four H100s. Its 140 GB of 16-bit weights leave 148 GB for the cache. A Fast replica runs the 8B model on one H100, leaving 56 GB of its 72 GB usable. A stream averages about 2,500 tokens of context while decoding, so it costs 0.82 GB of cache on Large and 0.33 GB on Fast. Memory caps a replica at 180 Large streams and 170 Fast streams, but at peak decode time and prefill are the tighter limits.
At 50% of peak math, prefill takes 140 GFLOP ÷ (4 × 989 TFLOPS × 0.5) = 70.8 µs per token on Large and 32.4 µs on Fast, so the 829 uncached tokens take about 59 ms and 27 ms. A decode step with B streams reads the weights and every stream's cache, which takes (140 + 0.82B) ÷ 13,400 s on Large and (16 + 0.33B) ÷ 3,350 s on Fast. If prefill takes a share p of a replica's time, decode must produce the rest of the output: (1 − p) × B ÷ step(B) tokens a second. Solving that for the smallest pool that keeps 30 tokens a second per stream at peak gives this:
| Pool | Replicas provisioned | Share of time on prefill | Streams per replica | Tokens/s per stream | Tokens/s per stream with one replica lost |
|---|---|---|---|---|---|
| Large (70B, 4 H100s each) | 44 | 49% | 90 | 32 | 31 (43 replicas) |
| Fast (8B, 1 H100 each) | 40 | 36% | 126 | 37 | 33 (39 replicas) |
Each provisioned count is one more than the smallest pool that meets the target, so losing a replica still leaves enough. The design needs 44 × 4 + 40 = 216 H100s. Little's law agrees with the solver: 365 × (0.3 s + 350 ÷ 32.2) ≈ 4,070 Large streams, about 92 per replica, and 538 × (0.15 s + 350 ÷ 37.4) ≈ 5,120 Fast streams, about 128 per replica. Both are within 3% of the solver.
The cache assumption dominates, so here is the sizing at two other peak hit rates:
| History hits at peak | Uncached prompt tokens per message | Large replicas | Fast replicas | H100s |
|---|---|---|---|---|
| 50% | about 570 | 34 | 35 | 171 |
| 25% (this design) | 829 | 44 | 40 | 216 |
| 0% | 1,085 | 54 | 45 | 261 |
Cost and storage. 216 H100s at an assumed $3 an hour cost 216 × $3 × 730 hours = $473,040 a month. Over 780 million messages a month, that is about $0.0006 a message, or $0.0036 for a six-message conversation. Average load is about a third of peak, and cold starts limit how closely a fleet can follow the daily curve. An exchange takes about 2 KB, so history grows by about 52 GB a day, or 19 TB a year (57 TB with three copies). The chat system chapter sized 300,000 writes a second, while this design writes 903 at peak, so the GPUs set the size of this system. Titles come from one short Fast call per conversation, about 4.3 million calls a day of roughly 400 prompt tokens each, which adds about 5% to the Fast pool's prompt volume.
Sending a message is a POST whose response is a server-sent event (SSE) stream, one HTTP response carrying text events from server to client. A browser's EventSource object only issues GET requests, so the app reads the POST response with fetch. A reconnect is a GET with a Last-Event-ID header. Stopping a reply is an ordinary request.
| Call | Purpose |
|---|---|
POST /v1/conversations/{id}/messages | Send a message and stream the reply. The body names the model, the parent message and any attachments. |
GET /v1/generations/{gid}/stream | Resume a stream after a dropped connection, from the last event ID the client saw. |
POST /v1/generations/{gid}/cancel | Stop a reply. A second call has no further effect. |
POST /v1/conversations/{id}/messages with regenerate_of | Make a new reply to the same user message. |
GET /v1/conversations, PATCH and DELETE on /v1/conversations/{id} | List (with a cursor), rename and delete conversations. |
A send, abridged. Each event's id is a token offset, so a resume can pick up where the stream stopped.
POST /v1/conversations/c_8f2d/messages HTTP/1.1
Authorization: Bearer <session token>
Idempotency-Key: 7b1e0c52-3a9f-4d1e-9c0b-2f6a8e41d5b7
Accept: text/event-stream
Content-Type: application/json
{"content": "Summarize this contract in five bullets.",
"model": "large", "parent_id": "m_1042", "attachments": ["f_93a1"]}
HTTP/1.1 200 OK
Content-Type: text/event-stream
event: start
data: {"message_id": "m_1043", "generation_id": "g_5530"}
event: delta
id: 17
data: {"text": "Five points"}
event: end
data: {"finish_reason": "end_turn", "input_tokens": 2310, "cached_tokens": 1880, "output_tokens": 298}
The Idempotency-Key header makes a send safe to retry, as the idempotency keys section describes. The server keeps the key with its message and generation IDs for 24 hours. A retry rejoins the running generation or receives the stored reply, and it never creates a second user message.
A normal send names the active leaf as its parent_id. Each message points to the message it answers, so a conversation is a tree, and the active leaf is the last message on the branch the user is viewing. Regenerating adds a sibling reply and moves the active leaf to it. A send whose parent is no longer the active leaf gets 409 Conflict, which stops two devices from interleaving writes unnoticed.
Every query reads one user's data, so the data is sharded by user_id, and a user's conversations and messages share one shard.
CREATE TABLE messages (
user_id BIGINT NOT NULL, -- shard key
conv_id UUID NOT NULL,
msg_id BIGINT NOT NULL, -- time-ordered, see unique IDs
parent_id BIGINT, -- the message this one answers
role VARCHAR(9) NOT NULL, -- 'user' or 'assistant'
content TEXT NOT NULL,
model VARCHAR(16), -- 'large' or 'fast' on replies
finish_reason VARCHAR(16), -- end_turn, max_tokens, cancelled, error
input_tokens INT,
cached_tokens INT,
output_tokens INT,
created_at TIMESTAMP NOT NULL,
PRIMARY KEY (user_id, conv_id, msg_id)
);
A conversations row holds the title, model preference, active leaf, rolling summary and the last message that summary covers. A new message and its conversation update commit in one transaction on that shard. The unique IDs chapter covers msg_id.
Attachments go to object storage, and the context builder extracts their text later. Image attachments need a model that accepts images, which the sizing above leaves out. Generation records hold a stream's state and token offset, expire within minutes, and live in a fast key-value store. The gateway writes one usage row per request to an append-only log for billing.
Step by step:
parent_id is the active leaf. It writes the user message and a generation record, then starts the stream.The window is 32,768 tokens, and each request reserves 1,024 for the reply. The system prompt takes 1,200 of the remaining 31,744, which leaves 30,544 for memory, the summary and the turns. Each prompt has a fixed layout:
[ system prompt 1,200 tokens shared by every request ]
[ memory block up to 500 per user, changes rarely ]
[ summary of old turns up to 800 per conversation, changes ]
[ recent turns oldest first a new exchange is appended ]
[ new user message about 60 always uncached ]
The layout is chosen for the cache. The system prompt and memory come first, so they match for every message from a user. The summary changes only when turns move into it, and recent turns change only when turns are dropped, so a new message appends to a prefix the engine has already processed.
An average conversation never reaches the limit, since the budget holds about 30,544 ÷ 410 ≈ 74 exchanges. Longer ones need two rules. Drop old turns in blocks of four, because dropping one per message changes the start of the history on every request, and the cache for those turns never hits. Summarize what leaves: a background job on the Fast pool reads the old summary plus four turns (about 2,440 tokens) and writes at most 400, so the cost is paid once per four turns.
The function below keeps the newest whole turns that fit.
def fit_history(turns, room, count):
"""Return (kept, dropped): the newest whole turns that fit in `room` tokens.
Old turns leave in blocks of four, so the cached prefix changes once every
four messages instead of on every message.
"""
kept, used = [], 0
for turn in reversed(turns): # newest first
size = count(turn)
if used + size > room:
break
kept.append(turn)
used += size
dropped = len(turns) - len(kept)
if dropped:
dropped = -(-dropped // 4) * 4 # round up to a whole block of four
return turns[dropped:], dropped
turns = [f"turn {i}" for i in range(10)] # ten exchanges, 100 tokens each
kept, dropped = fit_history(turns, room=550, count=lambda t: 100)
print(len(kept), dropped, kept[0]) # 2 8 turn 8
The caller passes room as the 30,544-token budget minus the memory, the summary and the new message. It passes count as the served tokenizer run on the rendered template, since counting four characters per token misses badly on code and on many languages.
Each hop (engine, gateway, edge and app) must forward events as they arrive, or a buffering proxy hides the stream (see streaming through a gateway). The edge sends a ping every 10 seconds when no tokens are flowing, so a long prefill does not look like a dead connection.
A reply ends in one of three ways:
cancelled. Tokens produced so far are billed, and later turns include the partial text, so the model sees what the user already read.Last-Event-ID inside that window replays the missed events, and the reply continues from the same token. Otherwise the edge cancels, as in the second case.Mobile connections often drop for a few seconds, and a restarted reply would prefill again and could read differently, since sampling is random. At 32 tokens a second, the grace period wastes at most about 320 tokens per dropped stream.
Product limits describe the plan. Free users get 10 Large messages a day and paid users get 200. Fast messages are capped at 40 per three hours for free users and 400 for paid users. Token limits protect the GPUs, and the gateway enforces them with the reserve-then-settle method from token-based rate limits. Each message reserves its prompt plus max_tokens (1,024 for free users, 4,096 for paid), and the unused part is settled afterwards. No user may have more than two generations in flight.
At peak a Large replica holds about 90 streams, and demand the pools cannot absorb waits in a queue. The gateway estimates each pool's wait from its waiting requests and completion rate, as the fleet chapter describes. When the free-tier estimate passes two seconds, free requests get 429 Too Many Requests with Retry-After, and paid requests move ahead, following the priorities and fairness rules. Overload also degrades in steps: free Large requests fall back to Fast with a notice, and free max_tokens drops to 1,024.
Input checks run before any pool sees a request: size limits, tier policy, and screening of text extracted from uploads, which can carry prompt injection. The reply is harder to check. OpenAI's moderation guide notes that for a streamed reply, classifier scores arrive only once the full output exists.
This design checks the stream in windows. Each window of 256 generated tokens, about 8 seconds of text, goes to a classifier that runs beside the stream. A flagged window makes the relay send an error event with a replacement notice, cancel the generation and keep the text for review. A final check covers the whole reply. The trade-off is that one window can show before a flag lands. Waiting for the whole reply would delay the first visible token by its full length, about 11 seconds for a 350-token reply. The classifier reads every output token, so its pool is sized from the peak rate of about 316,000 tokens a second.
Each user has a home region, chosen at sign-up, where their conversations are written. A second region keeps an asynchronous copy within seconds, and each region runs its own pools.
Failover has three parts. If the home region is unreachable, the second region serves the user from its copy. Messages sent in the seconds before the failure may be missing from that copy, so the app sends its last known message ID, and the server works out the gap. The idempotency key keeps any resend safe.
Capacity is the second part. If each region carries half the users, losing one doubles the other's peak. Either each region is sized for the full peak, which doubles the cost, or the survivor sheds free traffic until the failed region returns.
The third part is model failover. The Large alias can list a hosted provider after the pool, under the routing and fallbacks rules. An hour of peak Large traffic is about 1.31 million messages. At the assumed $3 and $15 per million input and output tokens, with 2,300 input and 350 output tokens each, that hour costs about $15,900. The Large pool costs $528 an hour (176 H100s at $3), so the fallback costs about 30 times as much. A spend cap and an alert belong in the design before the first outage.
Track four groups of numbers for each pool and tier. Speed: time to first token (TTFT, the wait before the first token) and time per output token (TPOT, the average gap between later tokens), both at p50 and p99, plus queue wait. Alert when TPOT rises above its healthy baseline, as the metrics to watch section advises. Load: the prefix hit ratio (cached input tokens over input tokens), the prefill share and preemptions. The hit ratio matters most, since the sizing depends on it. Quality and safety: finish reasons, cancels per 1,000 streams and moderation flags. Cost: spend per tier and per conversation.
Each message is one trace, with child spans for the history load, the context build (tokens in, cached tokens, whether a summary was used), routing, the engine call, moderation and storage. The engine span carries the GenAI attributes, such as gen_ai.usage.cache_read.input_tokens. Message text stays out of traces by default, under the content and privacy rules.
Cost per conversation comes from the ledger. Each usage row records tokens by kind and the pool that served it, and each pool's hourly cost is spread over the messages it served. At this volume that is about $0.0036 for a six-message conversation.
Some products remember facts across conversations. Keep that memory apart from the history, and keep it small.
Each message would need 2,285 − 1,200 = 1,085 uncached tokens, up from 829. The pools would need 54 Large and 45 Fast replicas, including the spares, or 261 H100s instead of 216. The hit rate sizes the fleet, so measure it first after launch.
Within the 10-second grace period, the edge replays the events after token 212, and the reply continues from the same token with no new prefill and no new sample. After 40 seconds the generation has been cancelled, so the resume call returns 410 Gone. The app loads the stored partial reply, saved with the finish reason cancelled, and offers to continue or regenerate.
A cached prefix stops matching at the first block that changes, so everything after a dropped turn must be processed again. Dropping one turn per message changes the start of the history on every request, so those turns never hit the cache. Dropping four at a time moves the prefix every fourth message, and the cache works for the three messages in between.
Unchanged TPOT means decoding is healthy, so the wait is before the first token. Check the prefix hit ratio first. A falling ratio raises the uncached prefill per message and pushes the prefill share above its designed 49%. Restoring the ratio helps most, by keeping histories short and routing each conversation to the replica that holds it. Chunked prefill, more replicas and a separate prefill pool come after that.
The hour carries about 1.31 million messages. Input costs 1.31 million × 2,300 × $3 ÷ 1,000,000 ≈ $9,040, and output costs 1.31 million × 350 × $15 ÷ 1,000,000 ≈ $6,880. The total is about $15,900, against $528 an hour for the pool, so a spend cap on the fallback route should exist before the first outage.
Waiting for the whole reply delays the first visible token by its full length, about 11 seconds for a 350-token reply at 32 tokens a second. Windows keep text flowing, at the cost that one flagged window, about 8 seconds of text, can show before the stream stops. Smaller windows shrink that exposure and add classifier calls, so the window size is a product decision to test on real traffic.
Last-Event-ID.Previous: Training and fine-tuning infrastructure · Next: Design a model inference platform (serve many LLMs to many customers)