System Design Handbook · llms.txt
An image generation service turns a prompt such as "a lighthouse in a storm, ink wash style" into a grid of four pictures. It also edits uploaded images, repaints masked regions and enlarges a chosen picture. Consumers use a web and mobile app, and developers call the same engine through an API.
An image model spends seconds of GPU work and has nothing usable until the end, so the design is a job queue with GPU workers behind it. It builds on GPUs for AI, running inference at scale, estimating AI workloads and queues and async work. Multi-tenant admission control is covered in the model inference platform question.
Unlike the token-by-token models in the LLM chapters, most image models are diffusion models. They start from random noise and remove some of it in each of many denoising steps, guided by the prompt. Five ideas are enough for the design:
These numbers are invented. There are 2 million daily active users, 10% of them paid. Free users run 2 jobs a day and paid users 15, and the API adds 1.4 million.
The model class is an SDXL-sized latent diffusion model of about 3.5 billion parameters, about 7 GB of weights in 16-bit numbers. The timing is an assumption to replace with profiling: on one H100 at 1024×1024, a denoising step costs 8 ms of fixed overhead plus 17 ms per evaluation in the batch. The VAE decode takes 60 ms per image and the text encoder 10 ms per request (also assumptions).
A full request has 4 images × 2 evaluations (CFG) = 8 evaluations per step. One step takes 8 + 17 × 8 = 144 ms, and 30 steps take 4,320 ms. Adding 4 × 60 = 240 ms of decode and 10 ms of text encoding gives 4.57 GPU-seconds per request, 1.14 per image. At $3 an hour that is $0.00095 per image if the GPU never idles. In general, GPU time ≈ steps × evaluations per step × time per evaluation, plus decode. Turning guidance off cuts a request to 2.53 GPU-seconds, a 4-step distilled model without guidance to 0.55, and time per evaluation grows roughly with pixel count, so a 512×512 draft costs about a quarter as much per evaluation.
An edit at strength 0.6 runs the last 60% of the schedule (18 steps) after encoding the source image (40 ms, assumed): 18 × 144 + 240 + 40 + 10 = 2,882 ms. A 2× upscale enlarges the image and refines it with 15 steps over four 1024×1024 tiles batched with CFG: 15 × 144 + 240 = 2,400 ms. The average job is 0.8 × 4.57 + 0.1 × 2.88 + 0.1 × 2.40 ≈ 4.18 GPU-seconds.
If fast capacity tracks the daily curve at 75% utilization, it uses 140 ÷ 0.75 × 24 ≈ 4,494 GPU-hours a day, $13,480 at $3 an hour, against $33,700 for holding 469 GPUs all day. Relaxed mode uses 247 ÷ 0.9 × 24 ≈ 6,586 GPU-hours, $7,900 at an assumed spot price of $1.20 an hour. Per image, fast costs $13,480 ÷ 10.73 million ≈ $0.0013 and relaxed $7,900 ÷ 18.87 million ≈ $0.0004, about $640,000 a month in all.
Assumed sizes: a 1024×1024 PNG is 1.5 MB, a WebP display copy 250 KB and a thumbnail 20 KB. An upscaled 2048×2048 image is 6 MB as PNG and 1 MB as WebP.
A submit returns 202 Accepted with a job ID at once. The app follows progress over a WebSocket, and API clients poll or receive a webhook.
| Call | Purpose |
|---|---|
POST /v1/images/generations | Text to image. Returns job_id, status and an estimated wait. |
POST /v1/images/edits | Image to image or inpainting, with a source image, optional mask and strength. |
POST /v1/images/{image_id}/upscale | A 2× upscale of one image. |
GET /v1/jobs/{job_id} and POST /v1/jobs/{job_id}/cancel | State, progress and image URLs; cancel a queued or running job. |
GET /v1/images?cursor=... and DELETE /v1/images/{image_id} | Gallery listing and deletion. |
POST /v1/images/generations HTTP/1.1
Authorization: Bearer <api key>
Idempotency-Key: 4f0c9a1e-77d2-4b8e-a0c3-91e5d2b6f7a8
Content-Type: application/json
{"prompt": "a lighthouse in a storm, ink wash style", "model": "base-v3",
"size": "1024x1024", "n": 4, "style": "ink-wash", "seed": 81723,
"guidance": 6, "mode": "fast", "webhook_url": "https://example.com/hooks/images"}
The idempotency key stops a retried submit from running or charging twice. A webhook is a POST signed with an HMAC (a keyed hash) of its timestamp and body, carrying job_id, a unique event_id and the final state. It is delivered at least once, with backoff for up to 24 hours, so receivers skip an event_id they have seen.
400 with a policy code and costs no GPU time.202.succeeded, settles credits and emits the event behind the WebSocket push or webhook. Clients load images from the CDN, as in the video streaming design.Nothing is finished until the decoder runs, about 4.6 seconds in. A request held open that long ties up a connection through every proxy, and a dropped connection loses the result. A job ID lets the client disconnect, switch devices or use a webhook, and lets the server retry a failed worker unseen, the pattern in telling the user the job is done.
A worker takes a job by lease: it marks the job running with its ID and an expiry 30 seconds ahead, and renews the lease every few steps. If the worker dies, the lease expires and the job returns to the queue, like the visibility timeout in when a worker fails halfway. The third failure ends in failed with credits refunded. Writes are keyed by job ID and image index, so a repeated attempt overwrites the same objects. If only some images are flagged, the job succeeds and those show a notice.
Previews. Every 5 steps the worker decodes the current latent with a small approximate decoder (assumed 3 ms for four images) and publishes a 2×2 grid as one JPEG of about 30 KB. In fast mode the first preview comes 5 × 144 ms ≈ 0.7 seconds after the job starts. Six previews per job is about 1.4 TB a day, pushed over a WebSocket and never stored. A user who sees a bad composition at step 10 can cancel and free the GPU.
A batch shares one tensor, so its jobs must match on model, style, size, step count and guidance. At 1024×1024 the assumed timing gives little gain past one request, because 8 evaluations already keep the GPU busy with arithmetic. A step costs 21 ms per evaluation at 2 evaluations, 18 at 8, 17.5 at 16 and 17.25 at 32.
Running a request's four images one at a time takes 4 × 30 × 42 + 240 = 5,280 ms, against 4,570 as one batch, so a request's images always share a batch. Adding a second request lifts throughput from 0.875 to 0.899 images per GPU-second, under 3%, and nearly doubles latency to 8.9 seconds. Fast mode runs one request per batch, and relaxed mode may pair two. At 512×512 the fixed 8 ms matters more (12.3 ms for one evaluation, 4.5 ms each at 32), so drafts are batched across many requests.
from dataclasses import dataclass
@dataclass(frozen=True)
class Job:
job_id: str
model: str # model version, e.g. "base-v3" or "turbo-v3"
style: str # LoRA id, "" for none
width: int
height: int
steps: int
images: int = 4
cfg: bool = True # classifier-free guidance doubles the evaluations
def shape_key(j):
"""Jobs with equal keys can share one batch tensor and one step loop."""
return (j.model, j.style, j.width, j.height, j.steps, j.cfg)
def evals(j):
return j.images * (2 if j.cfg else 1)
def next_batch(waiting, max_evals):
"""`waiting` is in scheduling order. Take the head, then later jobs with the same key."""
head = waiting[0]
batch, used = [head], evals(head)
for j in waiting[1:]:
if shape_key(j) == shape_key(head) and used + evals(j) <= max_evals:
batch.append(j)
used += evals(j)
return batch, used
waiting = [
Job("a", "turbo-v3", "", 512, 512, 4, cfg=False),
Job("b", "base-v3", "", 1024, 1024, 30),
Job("c", "turbo-v3", "", 512, 512, 4, cfg=False),
Job("d", "turbo-v3", "ink-wash", 512, 512, 4, cfg=False),
]
batch, used = next_batch(waiting, max_evals=32)
print([j.job_id for j in batch], used) # ['a', 'c'] 8
The head always runs, so batching never makes the oldest job wait. Mixing shapes would need padding: a 1024×1024 job and a 1344×768 job padded to a shared 1344×1024 do 31% and 33% more work than their own pixels need, and the padding changes the picture. That is one reason the API allows only five sizes.
An H100 has about 72 GB usable after 10% headroom. With assumed sizes, four 7 GB model variants (current base, previous version, inpainting and distilled) take 28 GB, an upscaler 3 GB, activations for 8 evaluations at about 2 GB each 16 GB, one VAE decode at 1024 about 5 GB, and 50 style adapters at 150 MB each 7.5 GB. That is 59.5 GB. A 70B language model needs four H100s for its weights, while one H100 holds five image models, so compute is the scarce resource and every worker serves every variant. Newer models are larger (FLUX.1 has about 12 billion parameters, 24 GB in 16-bit), so redo the budget per model.
A style is a LoRA adapter, a small set of weight changes trained for one look, as in serving many models. At an assumed 150 MB, one loads from local NVMe in 0.03 seconds and from the regional store over 10 Gbps in 0.12. Merging it into the denoiser takes about a second (assumed) and then costs nothing per step, while applying it unmerged adds a little per step but lets one batch mix styles. Assume the top 50 styles cover 80% of styled jobs, so those stay on every worker. A rare style is mapped by consistent hashing onto 8 workers, where it stays warm, and its jobs go there.
Each tier has a queue, and each job gets a deadline at submission: 15 seconds for fast, 600 for paid relaxed and 1,800 for free. Workers pop in strict tier order, except that a paid relaxed job within 60 seconds of its deadline goes ahead of fast work, so a busy evening cannot starve it.
import heapq
import itertools
TARGET_S = {"fast": 15, "relaxed_paid": 600, "relaxed_free": 1800}
GUARD_S = 60 # a paid relaxed job this close to its deadline jumps the fast queue
class TieredQueue:
def __init__(self):
self.heaps = {tier: [] for tier in TARGET_S}
self.seq = itertools.count()
def push(self, tier, job, now):
deadline = now + TARGET_S[tier]
heapq.heappush(self.heaps[tier], (deadline, next(self.seq), job))
def pop(self, now):
paid = self.heaps["relaxed_paid"]
if paid and paid[0][0] - now <= GUARD_S:
return heapq.heappop(paid)[2]
for tier in ("fast", "relaxed_paid", "relaxed_free"):
if self.heaps[tier]:
return heapq.heappop(self.heaps[tier])[2]
return None
q = TieredQueue()
q.push("relaxed_free", "free-1", now=0)
q.push("relaxed_paid", "paid-1", now=0)
q.push("fast", "fast-1", now=545)
q.push("fast", "fast-2", now=545)
print(q.pop(now=545), q.pop(now=545), q.pop(now=546), q.pop(now=547))
# paid-1 fast-1 fast-2 free-1
429 with a suggested time. Refusing at the door costs nothing, while dropping an accepted job breaks a promise, as back pressure explains.Assume the quietest hour runs at 0.2 of the daily average. Fast mode then needs 140 × 0.2 ÷ 0.75 ≈ 37 GPUs at night and 469 at the evening peak. Scale on queue wait and waiting jobs per worker, since GPU utilization reads near 100% on any saturated worker whatever the queue length.
The fleet chapter's cold start for a 141 GB model on a 10 Gbps link was about 209 seconds on a fresh node and 88 on a warm one. With the same 10 GB container image and link but 7 GB of weights, the download falls from 113 seconds to 5.6 and the disk load from 28 seconds to 1.4. Assume 45 seconds of engine start, including cached compiled kernels and a warm-up image. A cold node is ready in 8 + 5.6 + 1.4 + 45 = 60 seconds and a warm one in about 46, so start-up work now dominates. A new cloud machine adds minutes, so keep a small warm pool and scale ahead of the evening rise.
The relaxed tier runs on spot GPUs, spare capacity the provider can reclaim at short notice. At $1.20 an hour instead of $3 (assumed), its 6,586 GPU-hours save about $11,860 a day. Providers warn from about 30 seconds to 2 minutes ahead, longer than a batch, so the worker stops leasing and finishes. A machine that vanishes unwarned loses on average half a job (about 2.3 GPU-seconds), and the expired leases send its jobs elsewhere. Fast mode stays on on-demand GPUs, because a reclaim wave at peak would break its target.
torch.compile or TensorRT. Gains of 20% to 40% per step are plausible, an assumption to measure. Each shape needs its own compiled graph, a second reason for five sizes, and stored artifacts let new workers skip compiling.CSAM follows a separate path. In many jurisdictions, including the United States, a provider that becomes aware of apparent CSAM must report it to a designated body (in the US, NCMEC's CyberTipline) and preserve it as the law specifies. Counsel defines the process. Engineering provides a restricted pipeline, holds the material from automatic deletion for the preservation period, and acts on the account.
Rate limits apply per account, device and network, and an account with repeated blocks slows down, then loses fast mode, then is suspended (see abuse and content safety). Unlike the windowed checks on streamed text in the chat assistant, each image is checked whole before release.
Each file is stored under a content-addressed key, the SHA-256 hash of its bytes, such as img/3f/3fa9c2…e1.webp. The bytes behind a key never change, so the CDN caches it with Cache-Control: public, max-age=31536000, immutable (see HTTP caching headers). The metadata database maps each image_id to its keys, job, owner, seed and visibility.
Private images use signed URLs, with an expiry and a signature the CDN checks at the edge. The signature stays out of the cache key, or every viewer would miss. Thumbnails and WebP copies are written with the PNG, and a pull CDN fits, since most images are seen only by their creator.
Deletion marks the row, which hides the image at once. A background job then deletes the objects (unless another row points at the same hash, as when a seeded job is rerun) and purges the CDN paths. Signed URLs expire within an hour, which bounds how long a copied link works, and backups age out on their own schedule.
These signals can show that a marked image came from this service. None can prove that an unmarked image was made by a person.
The seed fixes the starting noise. Each image stores its seed (the request seed plus the image index) and the full recipe: model version, scheduler, steps, guidance, size, style version and source image. Reusing a seed with a small prompt change keeps a similar composition. Reruns are not bit-exact, because another GPU type, batch size, library version or nondeterministic kernel can change pixels slightly. Promise reproduction only on the same model version, keep old versions available as long as that promise lasts, and serve the stored file when an exact copy matters.
Trace each job with a span per stage, as in tracing model calls, and log only prompt hashes.
A step falls from 144 ms to 8 + 17 × 4 = 76 ms, and a request from 4.57 to 2.53 GPU-seconds, about 45% less, but images follow the prompt less closely. A distilled model with guidance built in saves more without that loss, once it passes a quality eval.
2.9 million × 4.18 ÷ 86,400 ≈ 140 busy GPUs on average, 351 at peak, so 351 ÷ 0.75 ≈ 469. Tracking the daily curve takes about 4,494 GPU-hours, $13,480, over 10.7 million images: about $0.0013 each.
Nothing usable exists until about 4.6 seconds of GPU work are done, and relaxed jobs may wait minutes first. A long request holds a connection through every proxy and loses the result if it drops. A job ID lets clients reconnect, poll or take a webhook, and lets the server retry a failed worker unseen.
At most one batch, about 4.6 seconds, or 8.9 if two requests were paired. Pausing is possible, since a latent is only 131 KB per image, but rebuilding the batch would save only seconds.
On the warning, the worker stops leasing and finishes its batch, which is shorter than the notice. Without one, the leases expire after 30 seconds and other workers rerun the jobs, overwriting the same objects because writes are keyed by job ID and image index. The expected loss is half a job.
Run the watermark detector, which may still find the payload after resizing or recompression, and compare the image's perceptual hash with stored output hashes. A match points to a job. No match proves nothing, because heavy edits or regeneration can remove both signals.
Previous: Design a vector database (a managed service for storing and searching embeddings) · Next: Numbers to know