{"id":"design-image-generation","title":"Design an image generation service (text to image, like Midjourney or DALL·E)","sections":[{"id":"overview","title":"Overview","html":"<p>An image generation service turns a prompt such as \"a lighthouse in a storm, ink wash style\" into a grid of four pictures. It also edits uploaded images, repaints masked regions and enlarges a chosen picture. Consumers use a web and mobile app, and developers call the same engine through an API.</p>\n<p>An image model spends seconds of GPU work and has nothing usable until the end, so the design is a job queue with GPU workers behind it. It builds on <a href=\"/learn/gpus-for-ai\">GPUs for AI</a>, <a href=\"/learn/inference-fleets\">running inference at scale</a>, <a href=\"/learn/ai-estimation\">estimating AI workloads</a> and <a href=\"/learn/queues-and-async\">queues and async work</a>. Multi-tenant admission control is covered in the <a href=\"/learn/design-inference-platform#admission-control-and-fairness\">model inference platform</a> question.</p>","headings":[]},{"id":"how-a-diffusion-model-makes-an-image","title":"How a diffusion model makes an image","html":"<h3 id=\"how-a-diffusion-model-makes-an-image\">How a diffusion model makes an image</h3>\n<p>Unlike the token-by-token models in the LLM chapters, most image models are <strong>diffusion models</strong>. They start from random noise and remove some of it in each of many <strong>denoising steps</strong>, guided by the prompt. Five ideas are enough for the design:</p>\n<ul>\n<li><strong>Latent space and the VAE decoder.</strong> The steps run on a compressed copy of the image called a latent, a 128×128 grid with 4 channels for a 1024×1024 image. A variational autoencoder (VAE) converts between pixels and latents, and its decoder turns the final latent into pixels once, at the end.</li>\n<li><strong>The text encoder</strong> turns the prompt into vectors that every step reads. It runs once per request.</li>\n<li><strong>The denoiser</strong> is the large network that predicts the noise in the current latent. One call of it is an <strong>evaluation</strong>.</li>\n<li><strong>Classifier-free guidance (CFG)</strong> evaluates the denoiser twice per step, with and without the prompt, and pushes the result away from the unprompted answer by a guidance scale such as 6. Images follow the prompt more closely, and each step costs twice as much.</li>\n<li><strong>Distilled few-step models</strong> imitate the full model in 1 to 4 steps, usually with guidance built in, at some cost in quality and variety.</li>\n</ul>","headings":[{"id":"how-a-diffusion-model-makes-an-image","title":"How a diffusion model makes an image","depth":3}]},{"id":"requirements-and-estimates","title":"Requirements and estimates","html":"<h3 id=\"requirements-and-estimates\">Requirements and estimates</h3>\n<h4 id=\"scope\">Scope</h4>\n<ul>\n<li><strong>Functional:</strong> text to image at 1024×1024 (plus four other shapes of about one megapixel), four images per request, previews while they form, image-to-image edits and inpainting (repainting a masked area), 2× upscale, style presets, a gallery with history, and a developer API with webhooks.</li>\n<li><strong>Tiers:</strong> free users get relaxed mode only, a slower queue, with a daily cap. Paid users get monthly fast credits plus unlimited relaxed mode. API customers pay per image and use the fast queue.</li>\n<li><strong>Safety and provenance:</strong> a check on every prompt before GPU work and on every image before anyone sees it, plus a signed C2PA manifest and an invisible watermark on each output.</li>\n<li><strong>Non-functional:</strong> fast mode returns four images within 15 seconds at p95 (the 95th percentile), with a first preview soon after the job starts. Paid relaxed jobs finish within 10 minutes at p95. Free jobs are best effort and may be refused at peak. An accepted job is never lost.</li>\n<li><strong>Out of scope:</strong> training, video, and ranking for a public feed.</li>\n</ul>\n<h4 id=\"traffic\">Traffic</h4>\n<p>These numbers are invented. There are 2 million daily active users, 10% of them paid. Free users run 2 jobs a day and paid users 15, and the API adds 1.4 million.</p>\n<ul>\n<li>Free 1.8 million × 2 = 3.6 million, paid 200,000 × 15 = 3.0 million, API 1.4 million: <strong>8 million jobs a day</strong>, 80% text to image, 10% edits and 10% upscales.</li>\n<li>8,000,000 ÷ 86,400 ≈ 92.6 jobs a second on average. With an assumed peak factor of 2.5, about <strong>231 a second</strong> at peak.</li>\n<li>Half of paid jobs use fast mode, so fast carries 1.5 + 1.4 = <strong>2.9 million jobs a day</strong> and relaxed carries 3.6 + 1.5 = <strong>5.1 million</strong>.</li>\n</ul>\n<h4 id=\"gpu-time-per-image\">GPU time per image</h4>\n<p>The model class is an SDXL-sized latent diffusion model of about 3.5 billion parameters, about 7 GB of weights in 16-bit numbers. The timing is an <strong>assumption</strong> to replace with profiling: on one H100 at 1024×1024, a denoising step costs 8 ms of fixed overhead plus 17 ms per evaluation in the batch. The VAE decode takes 60 ms per image and the text encoder 10 ms per request (also assumptions).</p>\n<p>A full request has 4 images × 2 evaluations (CFG) = 8 evaluations per step. One step takes 8 + 17 × 8 = 144 ms, and 30 steps take 4,320 ms. Adding 4 × 60 = 240 ms of decode and 10 ms of text encoding gives <strong>4.57 GPU-seconds per request, 1.14 per image</strong>. At $3 an hour that is $0.00095 per image if the GPU never idles. In general, GPU time ≈ steps × evaluations per step × time per evaluation, plus decode. Turning guidance off cuts a request to 2.53 GPU-seconds, a 4-step distilled model without guidance to 0.55, and time per evaluation grows roughly with pixel count, so a 512×512 draft costs about a quarter as much per evaluation.</p>\n<p>An edit at strength 0.6 runs the last 60% of the schedule (18 steps) after encoding the source image (40 ms, assumed): 18 × 144 + 240 + 40 + 10 = 2,882 ms. A 2× upscale enlarges the image and refines it with 15 steps over four 1024×1024 tiles batched with CFG: 15 × 144 + 240 = 2,400 ms. The average job is 0.8 × 4.57 + 0.1 × 2.88 + 0.1 × 2.40 ≈ <strong>4.18 GPU-seconds</strong>.</p>\n<h4 id=\"gpus-and-cost\">GPUs and cost</h4>\n<ul>\n<li>All jobs: 8 million × 4.18 ≈ 33.5 million GPU-seconds a day, about 387 GPUs busy on average and 969 at peak.</li>\n<li>Fast mode: 2.9 million × 4.18 ÷ 86,400 ≈ 140 busy GPUs on average, 351 at peak. Planned at 75% utilization to leave room for bursts and failures, the peak needs 351 ÷ 0.75 ≈ <strong>469 H100s</strong>.</li>\n<li>Relaxed mode: 5.1 million × 4.18 ÷ 86,400 ≈ 247 busy GPUs on average, 617 at peak. Its jobs can wait, so it runs near 90% utilization on spot GPUs.</li>\n<li>Output: 6.4 million × 4 + 0.8 million × 4 + 0.8 million × 1 = <strong>29.6 million images a day</strong>.</li>\n</ul>\n<p>If fast capacity tracks the daily curve at 75% utilization, it uses 140 ÷ 0.75 × 24 ≈ 4,494 GPU-hours a day, $13,480 at $3 an hour, against $33,700 for holding 469 GPUs all day. Relaxed mode uses 247 ÷ 0.9 × 24 ≈ 6,586 GPU-hours, $7,900 at an assumed spot price of $1.20 an hour. Per image, fast costs $13,480 ÷ 10.73 million ≈ <strong>$0.0013</strong> and relaxed $7,900 ÷ 18.87 million ≈ <strong>$0.0004</strong>, about $640,000 a month in all.</p>\n<h4 id=\"storage-and-egress\">Storage and egress</h4>\n<p>Assumed sizes: a 1024×1024 PNG is 1.5 MB, a WebP display copy 250 KB and a thumbnail 20 KB. An upscaled 2048×2048 image is 6 MB as PNG and 1 MB as WebP.</p>\n<ul>\n<li>New data: 28.8 million × 1.77 MB + 0.8 million × 7.02 MB ≈ 51.0 + 5.6 = <strong>56.6 TB a day</strong>, 20.7 PB a year if everything is kept.</li>\n<li>Retention: keep every PNG for 30 days (1.3 PB at any time), then only for the 10% of images a user downloaded, upscaled, favourited or published (1.58 PB a year). Keep WebP copies, thumbnails and upscales for all (4.89 PB a year). After one year the store holds about <strong>7.8 PB</strong>, about $155,000 a month at an assumed $0.02 per GB-month, against $413,000 if every PNG were kept. It then grows by 0.54 PB a month, so old images belong in a colder storage class.</li>\n<li>Egress: assume each image is viewed twice at display size and ten times as a thumbnail (700 KB), and each upscale is downloaded once. That is 28.8 million × 0.7 MB + 0.8 million × 6 MB ≈ <strong>25 TB a day</strong>, about $15,000 a month at an assumed $0.02 per GB.</li>\n</ul>","headings":[{"id":"requirements-and-estimates","title":"Requirements and estimates","depth":3},{"id":"scope","title":"Scope","depth":4},{"id":"traffic","title":"Traffic","depth":4},{"id":"gpu-time-per-image","title":"GPU time per image","depth":4},{"id":"gpus-and-cost","title":"GPUs and cost","depth":4},{"id":"storage-and-egress","title":"Storage and egress","depth":4}]},{"id":"api-design","title":"API design","html":"<h3 id=\"api-design\">API design</h3>\n<p>A submit returns <code>202 Accepted</code> with a job ID at once. The app follows progress over a WebSocket, and API clients poll or receive a webhook.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Call</th><th>Purpose</th></tr></thead><tbody><tr><td><code>POST /v1/images/generations</code></td><td>Text to image. Returns <code>job_id</code>, <code>status</code> and an estimated wait.</td></tr><tr><td><code>POST /v1/images/edits</code></td><td>Image to image or inpainting, with a source image, optional mask and strength.</td></tr><tr><td><code>POST /v1/images/{image_id}/upscale</code></td><td>A 2× upscale of one image.</td></tr><tr><td><code>GET /v1/jobs/{job_id}</code> and <code>POST /v1/jobs/{job_id}/cancel</code></td><td>State, progress and image URLs; cancel a queued or running job.</td></tr><tr><td><code>GET /v1/images?cursor=...</code> and <code>DELETE /v1/images/{image_id}</code></td><td>Gallery listing and deletion.</td></tr></tbody></table></div>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">POST</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> /v1/images/generations </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">HTTP</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.1</span></span>\n<span class=\"line\"><span style=\"color:#22863A;--shiki-dark:#85E89D\">Authorization</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> Bearer &#x3C;api key></span></span>\n<span class=\"line\"><span style=\"color:#22863A;--shiki-dark:#85E89D\">Idempotency-Key</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> 4f0c9a1e-77d2-4b8e-a0c3-91e5d2b6f7a8</span></span>\n<span class=\"line\"><span style=\"color:#22863A;--shiki-dark:#85E89D\">Content-Type</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> application/json</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">{</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"prompt\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"a lighthouse in a storm, ink wash style\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"model\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"base-v3\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> \"size\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"1024x1024\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"n\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"style\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"ink-wash\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"seed\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">81723</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> \"guidance\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">6</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"mode\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fast\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"webhook_url\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"https://example.com/hooks/images\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>The <a href=\"/learn/reliability-patterns#idempotency-keys\">idempotency key</a> stops a retried submit from running or charging twice. A webhook is a POST signed with an HMAC (a keyed hash) of its timestamp and body, carrying <code>job_id</code>, a unique <code>event_id</code> and the final state. It is delivered at least once, with backoff for up to 24 hours, so receivers skip an <code>event_id</code> they have seen.</p>","headings":[{"id":"api-design","title":"API design","depth":3}]},{"id":"high-level-design","title":"High-level design","html":"<h3 id=\"high-level-design\">High-level design</h3>\n<figure class=\"diagram\" data-text=\"App and API clients <-> Job API auth, credits (submit, WebSocket); Job API auth, credits -> Prompt check before GPU (prompt); Job API auth, credits -> Jobs DB recipe, state (job row); Job API auth, credits -> Tiered queues fast, relaxed (enqueue); Tiered queues fast, relaxed -> GPU workers batch, denoise (batch); GPU workers batch, denoise -> Image check after decode (images); GPU workers batch, denoise -> Object storage PNG, WebP (store); GPU workers batch, denoise -> Progress fan-out (previews); Progress fan-out -> Job API auth, credits (push); Object storage PNG, WebP -> CDN signed URLs (origin)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 380\" width=\"800\" role=\"img\" aria-label=\"Submitting is cheap. GPU work happens behind a queue.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"153\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"240\" y1=\"133\" x2=\"240\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"278.7\" y1=\"133\" x2=\"352\" y2=\"78\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"560\" y1=\"133\" x2=\"560\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"162\" x2=\"647\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"534.2\" y1=\"191\" x2=\"427.6\" y2=\"311\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"374.2\" y1=\"313\" x2=\"267.6\" y2=\"193\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"720\" y1=\"196\" x2=\"720\" y2=\"311\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"136\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"158.5\">App and</tspan><tspan x=\"80\" y=\"174.5\">API clients</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"158.5\">Job API</tspan><tspan x=\"240\" y=\"174.5\">auth, credits</tspan></text><rect class=\"dg-node dg-note\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Prompt check</tspan><tspan x=\"240\" y=\"54.5\">before GPU</tspan></text><path class=\"dg-node dg-store\" d=\"M332,18 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,18 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"41.5\">Jobs DB</tspan><tspan x=\"400\" y=\"57.5\">recipe, state</tspan></text><rect class=\"dg-node dg-queue\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"452\" y1=\"144\" x2=\"452\" y2=\"180\"></line><line class=\"dg-line\" x1=\"442\" y1=\"144\" x2=\"442\" y2=\"180\"></line><line class=\"dg-line\" x1=\"432\" y1=\"144\" x2=\"432\" y2=\"180\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"386\" y=\"158.5\">Tiered queues</tspan><tspan x=\"386\" y=\"174.5\">fast, relaxed</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">GPU workers</tspan><tspan x=\"560\" y=\"174.5\">batch, denoise</tspan></text><rect class=\"dg-node dg-note\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Image check</tspan><tspan x=\"560\" y=\"54.5\">after decode</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"316\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"338.5\">Progress</tspan><tspan x=\"400\" y=\"354.5\">fan-out</tspan></text><path class=\"dg-node dg-store\" d=\"M652,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"161.5\">Object storage</tspan><tspan x=\"720\" y=\"177.5\">PNG, WebP</tspan></text><rect class=\"dg-node dg-box\" x=\"652\" y=\"316\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"338.5\">CDN</tspan><tspan x=\"720\" y=\"354.5\">signed URLs</tspan></text><rect class=\"dg-edge-bg\" x=\"102.3\" y=\"114\" width=\"115.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"160\" y=\"127\" text-anchor=\"middle\">submit, WebSocket</text><rect class=\"dg-edge-bg\" x=\"216.4\" y=\"94\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"107\" text-anchor=\"middle\">prompt</text><rect class=\"dg-edge-bg\" x=\"288.6\" y=\"96.5\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"315.3\" y=\"109.5\" text-anchor=\"middle\">job row</text><rect class=\"dg-edge-bg\" x=\"292.3\" y=\"114\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"127\" text-anchor=\"middle\">enqueue</text><rect class=\"dg-edge-bg\" x=\"458.5\" y=\"114\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"127\" text-anchor=\"middle\">batch</text><rect class=\"dg-edge-bg\" x=\"536.4\" y=\"94\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"107\" text-anchor=\"middle\">images</text><rect class=\"dg-edge-bg\" x=\"618.5\" y=\"109\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"639\" y=\"122\" text-anchor=\"middle\">store</text><rect class=\"dg-edge-bg\" x=\"451.1\" y=\"242\" width=\"59.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"480.9\" y=\"255\" text-anchor=\"middle\">previews</text><rect class=\"dg-edge-bg\" x=\"303.5\" y=\"244\" width=\"34.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320.9\" y=\"257\" text-anchor=\"middle\">push</text><rect class=\"dg-edge-bg\" x=\"696.4\" y=\"244.5\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"720\" y=\"257.5\" text-anchor=\"middle\">origin</text></svg></div><figcaption>Submitting is cheap. GPU work happens behind a queue.</figcaption></figure>\n<ol>\n<li>The API authenticates the caller, checks the idempotency key, the tier's limits and the user's running jobs (at most 3 fast ones for a paid user), and reserves credits.</li>\n<li>The prompt check runs. A refused prompt gets <code>400</code> with a policy code and costs no GPU time.</li>\n<li>The API writes the job row with the full recipe, enqueues the job ID and returns <code>202</code>.</li>\n<li>A worker pulls a batch of compatible jobs, runs the steps and publishes a preview every 5 steps.</li>\n<li>Each decoded image passes the image check, gets its C2PA manifest and watermark, and is written to object storage as PNG, WebP and thumbnail.</li>\n<li>The worker marks the job <code>succeeded</code>, settles credits and emits the event behind the WebSocket push or webhook. Clients load images from the CDN, as in the <a href=\"/learn/design-video-streaming#cdn-delivery-and-cost\">video streaming</a> design.</li>\n</ol>","headings":[{"id":"high-level-design","title":"High-level design","depth":3}]},{"id":"why-this-is-a-job-queue","title":"Why this is a job queue","html":"<h3 id=\"why-this-is-a-job-queue\">Why this is a job queue</h3>\n<p>Nothing is finished until the decoder runs, about 4.6 seconds in. A request held open that long ties up a connection through every proxy, and a dropped connection loses the result. A job ID lets the client disconnect, switch devices or use a webhook, and lets the server retry a failed worker unseen, the pattern in <a href=\"/learn/queues-and-async#telling-the-user-the-job-is-done\">telling the user the job is done</a>.</p>\n<figure class=\"diagram\" data-text=\"Submit -> queued (accepted); Submit -> rejected when shedding; queued <-> running step k of n (lease, expiry); running step k of n -> succeeded (images pass); running step k of n -> blocked by image check (all flagged); running step k of n -> failed after 3 tries; queued -> cancelled by the user; running step k of n -> cancelled by the user\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 380\" width=\"800\" role=\"img\" aria-label=\"Job states. Only the worker holding the lease moves a running job.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"80\" y1=\"189\" x2=\"80\" y2=\"311\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"313\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"135.4\" x2=\"647\" y2=\"69.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"647\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"188.6\" x2=\"647\" y2=\"254.6\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"264\" y1=\"189\" x2=\"372.4\" y2=\"311\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"400\" y1=\"191\" x2=\"400\" y2=\"311\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Submit</tspan></text><rect class=\"dg-node dg-queue\" x=\"172\" y=\"138\" width=\"136\" height=\"48\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"292\" y1=\"146\" x2=\"292\" y2=\"178\"></line><line class=\"dg-line\" x1=\"282\" y1=\"146\" x2=\"282\" y2=\"178\"></line><line class=\"dg-line\" x1=\"272\" y1=\"146\" x2=\"272\" y2=\"178\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"226\" y=\"166.5\">queued</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"158.5\">running</tspan><tspan x=\"400\" y=\"174.5\">step k of n</tspan></text><rect class=\"dg-node dg-note\" x=\"652\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"720\" y=\"46.5\">succeeded</tspan></text><rect class=\"dg-node dg-note\" x=\"652\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"720\" y=\"158.5\">blocked</tspan><tspan x=\"720\" y=\"174.5\">by image check</tspan></text><rect class=\"dg-node dg-note\" x=\"652\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"720\" y=\"278.5\">failed</tspan><tspan x=\"720\" y=\"294.5\">after 3 tries</tspan></text><rect class=\"dg-node dg-note\" x=\"332\" y=\"316\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"400\" y=\"338.5\">cancelled</tspan><tspan x=\"400\" y=\"354.5\">by the user</tspan></text><rect class=\"dg-node dg-note\" x=\"12\" y=\"316\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"80\" y=\"338.5\">rejected</tspan><tspan x=\"80\" y=\"354.5\">when shedding</tspan></text><rect class=\"dg-edge-bg\" x=\"129.2\" y=\"116\" width=\"59.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"129\" text-anchor=\"middle\">accepted</text><rect class=\"dg-edge-bg\" x=\"274.7\" y=\"114\" width=\"90.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"127\" text-anchor=\"middle\">lease, expiry</text><rect class=\"dg-edge-bg\" x=\"519.9\" y=\"93.4\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"559\" y=\"106.4\" text-anchor=\"middle\">images pass</text><rect class=\"dg-edge-bg\" x=\"519.9\" y=\"153\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"559\" y=\"166\" text-anchor=\"middle\">all flagged</text></svg></div><figcaption>Job states. Only the worker holding the lease moves a running job.</figcaption></figure>\n<p>A worker takes a job by <strong>lease</strong>: it marks the job <code>running</code> with its ID and an expiry 30 seconds ahead, and renews the lease every few steps. If the worker dies, the lease expires and the job returns to the queue, like the visibility timeout in <a href=\"/learn/queues-and-async#acknowledgements-and-visibility-timeouts\">when a worker fails halfway</a>. The third failure ends in <code>failed</code> with credits refunded. Writes are keyed by job ID and image index, so a repeated attempt overwrites the same objects. If only some images are flagged, the job succeeds and those show a notice.</p>\n<p><strong>Previews.</strong> Every 5 steps the worker decodes the current latent with a small approximate decoder (assumed 3 ms for four images) and publishes a 2×2 grid as one JPEG of about 30 KB. In fast mode the first preview comes 5 × 144 ms ≈ 0.7 seconds after the job starts. Six previews per job is about 1.4 TB a day, pushed over a <a href=\"/learn/realtime#websockets\">WebSocket</a> and never stored. A user who sees a bad composition at step 10 can cancel and free the GPU.</p>","headings":[{"id":"why-this-is-a-job-queue","title":"Why this is a job queue","depth":3}]},{"id":"gpu-workers-and-batching","title":"GPU workers and batching","html":"<h3 id=\"gpu-workers-and-batching\">GPU workers and batching</h3>\n<h4 id=\"batching-jobs-of-the-same-shape\">Batching jobs of the same shape</h4>\n<p>A batch shares one tensor, so its jobs must match on model, style, size, step count and guidance. At 1024×1024 the assumed timing gives little gain past one request, because 8 evaluations already keep the GPU busy with arithmetic. A step costs 21 ms per evaluation at 2 evaluations, 18 at 8, 17.5 at 16 and 17.25 at 32.</p>\n<p>Running a request's four images one at a time takes 4 × 30 × 42 + 240 = 5,280 ms, against 4,570 as one batch, so a request's images always share a batch. Adding a second request lifts throughput from 0.875 to 0.899 images per GPU-second, under 3%, and nearly doubles latency to 8.9 seconds. Fast mode runs one request per batch, and relaxed mode may pair two. At 512×512 the fixed 8 ms matters more (12.3 ms for one evaluation, 4.5 ms each at 32), so drafts are batched across many requests.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> dataclasses </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> dataclass</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">@dataclass</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">frozen</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> Job</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    job_id: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    model: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">        # model version, e.g. \"base-v3\" or \"turbo-v3\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    style: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">        # LoRA id, \"\" for none</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    width: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    height: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    steps: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    images: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 4</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    cfg: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> True</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # classifier-free guidance doubles the evaluations</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> shape_key</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(j):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Jobs with equal keys can share one batch tensor and one step loop.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (j.model, j.style, j.width, j.height, j.steps, j.cfg)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> evals</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(j):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j.images </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j.cfg </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> next_batch</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(waiting, max_evals):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"`waiting` is in scheduling order. Take the head, then later jobs with the same key.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    head </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> waiting[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    batch, used </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [head], evals(head)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> waiting[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:]:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> shape_key(j) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> shape_key(head) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> used </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> evals(j) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_evals:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            batch.append(j)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            used </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> evals(j)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> batch, used</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">waiting </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    Job(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"turbo-v3\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">512</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">512</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">cfg</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    Job(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"b\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"base-v3\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1024</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1024</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">30</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    Job(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"turbo-v3\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">512</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">512</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">cfg</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    Job(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"d\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"turbo-v3\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"ink-wash\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">512</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">512</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">cfg</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">batch, used </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> next_batch(waiting, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">max_evals</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">32</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">([j.job_id </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> batch], used)   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># ['a', 'c'] 8</span></span></code></pre>\n<p>The head always runs, so batching never makes the oldest job wait. Mixing shapes would need padding: a 1024×1024 job and a 1344×768 job padded to a shared 1344×1024 do 31% and 33% more work than their own pixels need, and the padding changes the picture. That is one reason the API allows only five sizes.</p>\n<h4 id=\"memory-on-one-gpu\">Memory on one GPU</h4>\n<p>An H100 has about 72 GB usable after 10% headroom. With assumed sizes, four 7 GB model variants (current base, previous version, inpainting and distilled) take 28 GB, an upscaler 3 GB, activations for 8 evaluations at about 2 GB each 16 GB, one VAE decode at 1024 about 5 GB, and 50 style adapters at 150 MB each 7.5 GB. That is 59.5 GB. A 70B language model needs four H100s for its weights, while one H100 holds five image models, so compute is the scarce resource and every worker serves every variant. Newer models are larger (FLUX.1 has about 12 billion parameters, 24 GB in 16-bit), so redo the budget per model.</p>\n<h4 id=\"styles-as-lora-adapters\">Styles as LoRA adapters</h4>\n<p>A style is a LoRA adapter, a small set of weight changes trained for one look, as in <a href=\"/learn/inference-fleets#serving-many-models\">serving many models</a>. At an assumed 150 MB, one loads from local NVMe in 0.03 seconds and from the regional store over 10 Gbps in 0.12. Merging it into the denoiser takes about a second (assumed) and then costs nothing per step, while applying it unmerged adds a little per step but lets one batch mix styles. Assume the top 50 styles cover 80% of styled jobs, so those stay on every worker. A rare style is mapped by <a href=\"/learn/consistent-hashing\">consistent hashing</a> onto 8 workers, where it stays warm, and its jobs go there.</p>","headings":[{"id":"gpu-workers-and-batching","title":"GPU workers and batching","depth":3},{"id":"batching-jobs-of-the-same-shape","title":"Batching jobs of the same shape","depth":4},{"id":"memory-on-one-gpu","title":"Memory on one GPU","depth":4},{"id":"styles-as-lora-adapters","title":"Styles as LoRA adapters","depth":4}]},{"id":"tiered-queues-and-scheduling","title":"Tiered queues and scheduling","html":"<h3 id=\"tiered-queues-and-scheduling\">Tiered queues and scheduling</h3>\n<p>Each tier has a queue, and each job gets a deadline at submission: 15 seconds for fast, 600 for paid relaxed and 1,800 for free. Workers pop in strict tier order, except that a paid relaxed job within 60 seconds of its deadline goes ahead of fast work, so a busy evening cannot starve it.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> heapq</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> itertools</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TARGET_S</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fast\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">15</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"relaxed_paid\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">600</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"relaxed_free\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1800</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">GUARD_S</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 60</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">      # a paid relaxed job this close to its deadline jumps the fast queue</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> TieredQueue</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.heaps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {tier: [] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tier </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> TARGET_S</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.seq </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> itertools.count()</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> push</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, tier, job, now):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        deadline </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> now </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> TARGET_S</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[tier]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        heapq.heappush(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.heaps[tier], (deadline, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">next</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.seq), job))</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> pop</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, now):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        paid </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.heaps[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"relaxed_paid\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> paid </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> paid[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">][</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> now </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> GUARD_S</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> heapq.heappop(paid)[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tier </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fast\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"relaxed_paid\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"relaxed_free\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.heaps[tier]:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> heapq.heappop(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.heaps[tier])[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">q </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> TieredQueue()</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">q.push(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"relaxed_free\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"free-1\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">now</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">q.push(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"relaxed_paid\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"paid-1\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">now</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">q.push(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fast\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fast-1\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">now</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">545</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">q.push(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fast\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fast-2\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">now</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">545</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(q.pop(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">now</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">545</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), q.pop(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">now</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">545</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), q.pop(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">now</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">546</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), q.pop(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">now</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">547</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># paid-1 fast-1 fast-2 free-1</span></span></code></pre>\n<ul>\n<li><strong>No preemption.</strong> A running batch is not interrupted for a fast job. Its state is small (a 128×128×4 latent in 16-bit is 131 KB per image, plus the step and random state), but rebuilding the batch would save only seconds. On a shared worker a fast job waits at most one batch, 4.6 seconds, or 8.9 if two requests were paired. Dedicated fast workers never run relaxed batches.</li>\n<li><strong>Credits.</strong> Fast jobs reserve credits from a price table (4 for text to image, 3 for an edit, 2 for an upscale) set from measured GPU-seconds, so spend is predictable. Failed jobs are refunded, and a job cancelled mid-run pays for its steps.</li>\n<li><strong>Shedding the free tier.</strong> At peak, paid relaxed work needs about 182 busy GPUs and free work about 436. With the relaxed fleet capped at 450 GPUs, free work gets 268, and its backlog grows by about 168 GPU-hours in the peak hour. When the estimated free wait (queue length over current throughput) passes 30 minutes, new free submissions get <code>429</code> with a suggested time. Refusing at the door costs nothing, while dropping an accepted job breaks a promise, as <a href=\"/learn/queues-and-async#back-pressure-when-the-queue-keeps-growing\">back pressure</a> explains.</li>\n</ul>","headings":[{"id":"tiered-queues-and-scheduling","title":"Tiered queues and scheduling","depth":3}]},{"id":"autoscaling-and-spot-gpus","title":"Autoscaling and spot GPUs","html":"<h3 id=\"autoscaling-and-spot-gpus\">Autoscaling and spot GPUs</h3>\n<p>Assume the quietest hour runs at 0.2 of the daily average. Fast mode then needs 140 × 0.2 ÷ 0.75 ≈ 37 GPUs at night and 469 at the evening peak. Scale on queue wait and waiting jobs per worker, since GPU utilization reads near 100% on any saturated worker whatever the queue length.</p>\n<p>The fleet chapter's <a href=\"/learn/inference-fleets#loading-models-quickly\">cold start</a> for a 141 GB model on a 10 Gbps link was about 209 seconds on a fresh node and 88 on a warm one. With the same 10 GB container image and link but 7 GB of weights, the download falls from 113 seconds to 5.6 and the disk load from 28 seconds to 1.4. Assume 45 seconds of engine start, including cached compiled kernels and a warm-up image. A cold node is ready in 8 + 5.6 + 1.4 + 45 = 60 seconds and a warm one in about 46, so start-up work now dominates. A new cloud machine adds minutes, so keep a small warm pool and scale ahead of the evening rise.</p>\n<p>The relaxed tier runs on spot GPUs, spare capacity the provider can reclaim at short notice. At $1.20 an hour instead of $3 (assumed), its 6,586 GPU-hours save about $11,860 a day. Providers warn from about 30 seconds to 2 minutes ahead, longer than a batch, so the worker stops leasing and finishes. A machine that vanishes unwarned loses on average half a job (about 2.3 GPU-seconds), and the expired leases send its jobs elsewhere. Fast mode stays on on-demand GPUs, because a reclaim wave at peak would break its target.</p>","headings":[{"id":"autoscaling-and-spot-gpus","title":"Autoscaling and spot GPUs","depth":3}]},{"id":"latency-tricks","title":"Latency tricks","html":"<h3 id=\"latency-tricks\">Latency tricks</h3>\n<ul>\n<li><strong>Distilled drafts.</strong> A 4-step distilled model without guidance makes four images in about 0.55 GPU-seconds, about 8 times cheaper than the full model. A 512×512 draft mode returns in about a second, and the user refines a chosen draft with the full model as an image-to-image job.</li>\n<li><strong>Cache text embeddings</strong> keyed on the encoder version and prompt hash, since users reroll the same prompt. It saves only 10 ms of 4,570, and an SDXL-style embedding (77 tokens × 2,048 values × 2 bytes) is about 315 KB.</li>\n<li><strong>Compile the denoiser</strong> for each supported shape with <code>torch.compile</code> or TensorRT. Gains of 20% to 40% per step are plausible, an assumption to measure. Each shape needs its own compiled graph, a second reason for five sizes, and stored artifacts let new workers skip compiling.</li>\n<li><strong>Generate at 1024, then upscale.</strong> A 2048×2048 step costs at least 4 times as much (4 times the pixels, and attention cost grows with their square), and models trained near one megapixel compose poorly at larger sizes. The tiled upscale costs 2.4 GPU-seconds and looks better.</li>\n</ul>","headings":[{"id":"latency-tricks","title":"Latency tricks","depth":3}]},{"id":"safety-pipeline","title":"Safety pipeline","html":"<h3 id=\"safety-pipeline\">Safety pipeline</h3>\n<figure class=\"diagram\" data-text=\"Prompt or upload -> Text and upload checks (request); Text and upload checks -> Generate on GPU (allowed); Text and upload checks -> Refused, with policy code (blocked); Text and upload checks -> Known-image hash match (uploads); Generate on GPU -> Image classifier (decoded images); Image classifier -> Stored with C2PA, watermark (pass or blur); Image classifier -> Review and appeals queue (flagged)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 380\" width=\"800\" role=\"img\" aria-label=\"Checks run before GPU spend and before anyone sees an image.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"240\" y1=\"133\" x2=\"240\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"240\" y1=\"191\" x2=\"240\" y2=\"311\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"162\" x2=\"647\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"560\" y1=\"191\" x2=\"560\" y2=\"311\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"136\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"158.5\">Prompt or</tspan><tspan x=\"80\" y=\"174.5\">upload</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"158.5\">Text and upload</tspan><tspan x=\"240\" y=\"174.5\">checks</tspan></text><rect class=\"dg-node dg-note\" x=\"172\" y=\"316\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"240\" y=\"338.5\">Known-image</tspan><tspan x=\"240\" y=\"354.5\">hash match</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"158.5\">Generate</tspan><tspan x=\"400\" y=\"174.5\">on GPU</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">Image</tspan><tspan x=\"560\" y=\"174.5\">classifier</tspan></text><path class=\"dg-node dg-store\" d=\"M652,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"161.5\">Stored with</tspan><tspan x=\"720\" y=\"177.5\">C2PA, watermark</tspan></text><rect class=\"dg-node dg-queue\" x=\"492\" y=\"316\" width=\"136\" height=\"52\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"612\" y1=\"324\" x2=\"612\" y2=\"360\"></line><line class=\"dg-line\" x1=\"602\" y1=\"324\" x2=\"602\" y2=\"360\"></line><line class=\"dg-line\" x1=\"592\" y1=\"324\" x2=\"592\" y2=\"360\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"546\" y=\"338.5\">Review and</tspan><tspan x=\"546\" y=\"354.5\">appeals queue</tspan></text><rect class=\"dg-node dg-note\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Refused, with</tspan><tspan x=\"240\" y=\"54.5\">policy code</tspan></text><rect class=\"dg-edge-bg\" x=\"132.3\" y=\"114\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"127\" text-anchor=\"middle\">request</text><rect class=\"dg-edge-bg\" x=\"292.3\" y=\"114\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"127\" text-anchor=\"middle\">allowed</text><rect class=\"dg-edge-bg\" x=\"213.3\" y=\"94\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"107\" text-anchor=\"middle\">blocked</text><rect class=\"dg-edge-bg\" x=\"213.3\" y=\"242\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"255\" text-anchor=\"middle\">uploads</text><rect class=\"dg-edge-bg\" x=\"430.6\" y=\"114\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"127\" text-anchor=\"middle\">decoded images</text><rect class=\"dg-edge-bg\" x=\"596.8\" y=\"109\" width=\"84.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"639\" y=\"122\" text-anchor=\"middle\">pass or blur</text><rect class=\"dg-edge-bg\" x=\"533.3\" y=\"242\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"255\" text-anchor=\"middle\">flagged</text></svg></div><figcaption>Checks run before GPU spend and before anyone sees an image.</figcaption></figure>\n<ol>\n<li><strong>Before the GPU.</strong> A text classifier scores the prompt and negative prompt, alongside maintained term lists, in a few milliseconds on CPU. Uploads for edits get an image classifier, stricter rules when they show a real person, and a match against hash lists of known child sexual abuse material (CSAM) from child-safety organizations. Each refusal saves 4.6 GPU-seconds, and the image never exists.</li>\n<li><strong>After decoding, before storage.</strong> An image classifier checks every output (about 10 ms per image, assumed), because harmless-looking prompts can produce bad images. Each image passes, is blurred behind a click-through (for adult accounts that opted in), or is blocked.</li>\n<li><strong>Review.</strong> Blocked outputs are kept encrypted for a fixed period, readable only by the trust and safety team. Users can appeal, decisions become training labels, and the overturn rate measures over-blocking.</li>\n</ol>\n<p>CSAM follows a separate path. In many jurisdictions, including the United States, a provider that becomes aware of apparent CSAM must report it to a designated body (in the US, NCMEC's CyberTipline) and preserve it as the law specifies. Counsel defines the process. Engineering provides a restricted pipeline, holds the material from automatic deletion for the preservation period, and acts on the account.</p>\n<p>Rate limits apply per account, device and network, and an account with repeated blocks slows down, then loses fast mode, then is suspended (see <a href=\"/learn/ai-security#abuse-and-content-safety\">abuse and content safety</a>). Unlike the windowed checks on streamed text in the <a href=\"/learn/design-chat-assistant#safety-and-moderation-in-the-stream\">chat assistant</a>, each image is checked whole before release.</p>","headings":[{"id":"safety-pipeline","title":"Safety pipeline","depth":3}]},{"id":"storage-and-delivery","title":"Storage and delivery","html":"<h3 id=\"storage-and-delivery\">Storage and delivery</h3>\n<p>Each file is stored under a <strong>content-addressed key</strong>, the SHA-256 hash of its bytes, such as <code>img/3f/3fa9c2…e1.webp</code>. The bytes behind a key never change, so the CDN caches it with <code>Cache-Control: public, max-age=31536000, immutable</code> (see <a href=\"/learn/dns-and-cdn#http-caching-headers\">HTTP caching headers</a>). The metadata database maps each <code>image_id</code> to its keys, job, owner, seed and visibility.</p>\n<p>Private images use <strong>signed URLs</strong>, with an expiry and a signature the CDN checks at the edge. The signature stays out of the cache key, or every viewer would miss. Thumbnails and WebP copies are written with the PNG, and a <a href=\"/learn/dns-and-cdn#pull-cdns\">pull CDN</a> fits, since most images are seen only by their creator.</p>\n<p>Deletion marks the row, which hides the image at once. A background job then deletes the objects (unless another row points at the same hash, as when a seeded job is rerun) and purges the CDN paths. Signed URLs expire within an hour, which bounds how long a copied link works, and backups age out on their own schedule.</p>","headings":[{"id":"storage-and-delivery","title":"Storage and delivery","depth":3}]},{"id":"provenance","title":"Provenance","html":"<h3 id=\"provenance\">Provenance</h3>\n<ul>\n<li><strong>A C2PA manifest.</strong> The Coalition for Content Provenance and Authenticity (C2PA) defines signed metadata embedded in the file. Ours states that this service generated the image, with which model version, and anyone can verify the signature. Screenshots, many re-encoders and some upload pipelines strip it.</li>\n<li><strong>An invisible watermark.</strong> A small payload, such as 48 bits mapping to the job, is hidden in the pixels. It survives resizing and moderate compression, while heavy edits, crops or regeneration through another model can remove it. Detection reports a confidence score, and its false positive rate needs measuring, because wrongly labelling an artist's work as generated does real harm.</li>\n<li><strong>A perceptual hash</strong> of each output, a fingerprint that changes little under resizing or recompression, answers \"did we make this?\" after the metadata is gone.</li>\n</ul>\n<p>These signals can show that a marked image came from this service. None can prove that an unmarked image was made by a person.</p>","headings":[{"id":"provenance","title":"Provenance","depth":3}]},{"id":"seeds-and-reproducibility","title":"Seeds and reproducibility","html":"<h3 id=\"seeds-and-reproducibility\">Seeds and reproducibility</h3>\n<p>The seed fixes the starting noise. Each image stores its seed (the request seed plus the image index) and the full recipe: model version, scheduler, steps, guidance, size, style version and source image. Reusing a seed with a small prompt change keeps a similar composition. Reruns are not bit-exact, because another GPU type, batch size, library version or nondeterministic kernel can change pixels slightly. Promise reproduction only on the same model version, keep old versions available as long as that promise lasts, and serve the stored file when an exact copy matters.</p>","headings":[{"id":"seeds-and-reproducibility","title":"Seeds and reproducibility","depth":3}]},{"id":"observability-and-quality","title":"Observability and quality","html":"<h3 id=\"observability-and-quality\">Observability and quality</h3>\n<ul>\n<li><strong>Latency:</strong> queue wait per tier at p50 and p95, time to first preview, and submit-to-finish time. Queue wait moves first under load, since generation time barely changes.</li>\n<li><strong>Efficiency:</strong> GPU utilization, batch fill, useful GPU-seconds over paid GPU-seconds (the rest is idle time, retries and cancelled work), and cost per image per tier.</li>\n<li><strong>Safety:</strong> prompt block rate, output block and blur rates, appeal overturn rate and review backlog. Output blocks rising while prompt blocks stay flat suggests a new evasion technique or a model change.</li>\n<li><strong>Quality:</strong> an image-text similarity score for prompt adherence, and human preference sampling, where raters compare current and candidate models on the same prompt and seed (see <a href=\"/learn/ai-evals-and-observability#human-review-and-binary-scores\">human review</a>). Also track the share of grids with an upscale or download, and rerolls per prompt.</li>\n</ul>\n<p>Trace each job with a span per stage, as in <a href=\"/learn/ai-evals-and-observability#tracing-model-calls\">tracing model calls</a>, and log only prompt hashes.</p>","headings":[{"id":"observability-and-quality","title":"Observability and quality","depth":3}]},{"id":"trade-offs-to-discuss","title":"Trade-offs to discuss","html":"<h3 id=\"trade-offs-to-discuss\">Trade-offs to discuss</h3>\n<ul>\n<li><strong>Batch size against latency.</strong> At 1024×1024 a second request in the batch adds under 3% throughput and nearly doubles latency. At small sizes and with distilled models, batching pays.</li>\n<li><strong>Spot against predictability.</strong> Spot GPUs make relaxed mode 60% cheaper and suit short, restartable jobs.</li>\n<li><strong>Shedding against queuing free work.</strong> Refusing early protects paid users and is honest. A long free queue accepts jobs that may finish after the user leaves.</li>\n<li><strong>Retention against cost.</strong> Keeping every PNG costs about $413,000 a month after a year, against $155,000 with the policy. Regenerating from the recipe is no substitute, since it is not bit-exact.</li>\n<li><strong>Prompt checks against output checks.</strong> Prompt checks are cheap and easy to evade. Output checks catch more and cost GPU time on every image, so the design runs both.</li>\n</ul>","headings":[{"id":"trade-offs-to-discuss","title":"Trade-offs to discuss","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>A product manager wants to turn classifier-free guidance off to save money. What does it save, and what else changes?</summary>\n<p>A step falls from 144 ms to 8 + 17 × 4 = 76 ms, and a request from 4.57 to 2.53 GPU-seconds, about 45% less, but images follow the prompt less closely. A distilled model with guidance built in saves more without that loss, once it passes a quality eval.</p>\n</details>\n<details>\n<summary>Fast mode serves 2.9 million jobs a day with a peak factor of 2.5, planned at 75% utilization. How many H100s at peak, and what does a fast image cost?</summary>\n<p>2.9 million × 4.18 ÷ 86,400 ≈ 140 busy GPUs on average, 351 at peak, so 351 ÷ 0.75 ≈ 469. Tracking the daily curve takes about 4,494 GPU-hours, $13,480, over 10.7 million images: about $0.0013 each.</p>\n</details>\n<details>\n<summary>Why is the API asynchronous with job IDs instead of one long HTTP request?</summary>\n<p>Nothing usable exists until about 4.6 seconds of GPU work are done, and relaxed jobs may wait minutes first. A long request holds a connection through every proxy and loses the result if it drops. A job ID lets clients reconnect, poll or take a webhook, and lets the server retry a failed worker unseen.</p>\n</details>\n<details>\n<summary>A fast job arrives while every shared worker is in a relaxed batch. How long can it wait, and why not pause the batch?</summary>\n<p>At most one batch, about 4.6 seconds, or 8.9 if two requests were paired. Pausing is possible, since a latent is only 131 KB per image, but rebuilding the batch would save only seconds.</p>\n</details>\n<details>\n<summary>The provider reclaims a spot machine running relaxed jobs. What happens?</summary>\n<p>On the warning, the worker stops leasing and finishes its batch, which is shorter than the notice. Without one, the leases expire after 30 seconds and other workers rerun the jobs, overwriting the same objects because writes are keyed by job ID and image index. The expected loss is half a job.</p>\n</details>\n<details>\n<summary>A news site asks whether a viral image came from your service. It has no C2PA manifest. What can you check?</summary>\n<p>Run the watermark detector, which may still find the payload after resizing or recompression, and compare the image's perceptual hash with stored output hashes. A match points to a job. No match proves nothing, because heavy edits or regeneration can remove both signals.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2006.11239\" target=\"_blank\" rel=\"noopener noreferrer\">Ho, Jain and Abbeel, Denoising Diffusion Probabilistic Models (arXiv 2006.11239)</a>: the step-by-step denoising process this design schedules.</li>\n<li><a href=\"https://arxiv.org/abs/2112.10752\" target=\"_blank\" rel=\"noopener noreferrer\">Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (arXiv 2112.10752)</a>: denoising in a VAE latent space, the basis of Stable Diffusion.</li>\n<li><a href=\"https://arxiv.org/abs/2207.12598\" target=\"_blank\" rel=\"noopener noreferrer\">Ho and Salimans, Classifier-Free Diffusion Guidance (arXiv 2207.12598)</a>: the guidance scale and its two evaluations.</li>\n<li><a href=\"https://arxiv.org/abs/2307.01952\" target=\"_blank\" rel=\"noopener noreferrer\">Podell et al., SDXL (arXiv 2307.01952)</a>: the model class behind the estimates, with its two text encoders and multi-aspect training.</li>\n<li><a href=\"https://arxiv.org/abs/2311.17042\" target=\"_blank\" rel=\"noopener noreferrer\">Sauer et al., Adversarial Diffusion Distillation (arXiv 2311.17042)</a>: images in one to four steps from a distilled model.</li>\n<li><a href=\"https://arxiv.org/abs/2303.01469\" target=\"_blank\" rel=\"noopener noreferrer\">Song et al., Consistency Models (arXiv 2303.01469)</a>: another route to few-step generation.</li>\n<li><a href=\"https://c2pa.org/\" target=\"_blank\" rel=\"noopener noreferrer\">C2PA</a>: the content provenance specification and its signed manifests.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}