{"id":"inference-engineer-role","title":"The inference engineer","sections":[{"id":"overview","title":"Overview","html":"<p>Training produces a model once. Serving it runs every day, for every user, and that is where most of the GPU bill goes. An inference engineer makes the serving fast, cheap and reliable. This handbook teaches that job from the bottom up: the arithmetic of a forward pass, the GPU it runs on, the kernels and engines that schedule it, the compression that shrinks it, and the fleet that keeps it up.</p>\n<p>The <a href=\"/learn/how-to-use-this-handbook\">system design handbook</a> on this site explains what an inference platform contains and why. This one goes a level down and asks you to build and measure the pieces. Each chapter links to the system design chapter it builds on instead of repeating it, so read <a href=\"/learn/llm-inference\">how LLM inference works</a> first if prefill, decode and the KV cache are new.</p>\n<p>This chapter covers what the job is, how it differs from its neighbours, what employers ask for, how interviews test it, and how to set up the lab you will use for the rest of the course.</p>","headings":[]},{"id":"what-an-inference-engineer-does","title":"What an inference engineer does","html":"<h3 id=\"what-an-inference-engineer-does\">What an inference engineer does</h3>\n<p>A request for a completion passes through several layers before a single multiplication happens. An inference engineer owns most of them.</p>\n<figure class=\"diagram\" data-text=\"Client app -> API gateway auth, quotas (HTTPS, streamed); API gateway auth, quotas -> Router picks a replica; Router picks a replica -> Engine scheduler batches, KV blocks; Engine scheduler batches, KV blocks -> Model runner CUDA graphs (one batch per step); Model runner CUDA graphs -> Kernels GEMM, attention; Kernels GEMM, attention -> GPU memory weights, KV cache (read every step)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -10 640 215\" width=\"640\" role=\"img\" aria-label=\"The path of one request. The inference engineer owns everything from the router down.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"560\" y1=\"71\" x2=\"560\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"489\" y1=\"162\" x2=\"473\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"329\" y1=\"162\" x2=\"313\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">Client app</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">API gateway</tspan><tspan x=\"240\" y=\"54.5\">auth, quotas</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Router</tspan><tspan x=\"400\" y=\"54.5\">picks a replica</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Engine scheduler</tspan><tspan x=\"560\" y=\"54.5\">batches, KV blocks</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">Model runner</tspan><tspan x=\"560\" y=\"174.5\">CUDA graphs</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"158.5\">Kernels</tspan><tspan x=\"400\" y=\"174.5\">GEMM, attention</tspan></text><path class=\"dg-node dg-store\" d=\"M172,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M172,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"161.5\">GPU memory</tspan><tspan x=\"240\" y=\"177.5\">weights, KV cache</tspan></text><rect class=\"dg-edge-bg\" x=\"107.5\" y=\"-6\" width=\"103\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"7\" text-anchor=\"middle\">HTTPS, streamed</text><rect class=\"dg-edge-bg\" x=\"499.2\" y=\"92\" width=\"121.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"105\" text-anchor=\"middle\">one batch per step</text><rect class=\"dg-edge-bg\" x=\"269.5\" y=\"109\" width=\"103\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"321\" y=\"122\" text-anchor=\"middle\">read every step</text></svg></div><figcaption>The path of one request. The inference engineer owns everything from the router down.</figcaption></figure>\n<ul>\n<li>The <strong>router</strong> chooses a replica, often by load or by which replica already holds the prompt's prefix in its cache.</li>\n<li>The <strong>engine scheduler</strong> decides, every step, which requests run, how many prompt tokens to process, and where each request's KV cache lives in GPU memory.</li>\n<li>The <strong>model runner</strong> turns that batch into GPU work: PyTorch modules, compiled graphs and captured CUDA graphs.</li>\n<li>The <strong>kernels</strong> do the arithmetic. Matrix multiplies (GEMMs) and attention take nearly all of the time.</li>\n<li><strong>GPU memory</strong> holds the weights and the KV cache. Reading it is what limits most decode steps, as Part 1 shows.</li>\n</ul>\n<p>The work is measured by a handful of numbers. Time to first token (TTFT) is how long a user waits before anything appears. Inter-token latency (ITL) is the gap between streamed tokens. Throughput is tokens per second per GPU, and cost is dollars per million tokens. <a href=\"/learn/llm-inference#latency-and-throughput-metrics\">The latency metrics section</a> defines them. In a typical week, the work might look like this:</p>\n<ul>\n<li>You bring up a new model in vLLM or SGLang and find a configuration that meets its latency target.</li>\n<li>You quantize the model to FP8 and show that the quality loss is acceptable.</li>\n<li>You trace a p99 latency regression to a scheduler change or a slow kernel.</li>\n<li>You write a fused kernel for an operation the engine runs unfused.</li>\n<li>You plan how many GPUs next quarter's traffic needs, and what they will cost.</li>\n</ul>","headings":[{"id":"what-an-inference-engineer-does","title":"What an inference engineer does","depth":3}]},{"id":"where-the-role-sits","title":"Where the role sits","html":"<h3 id=\"where-the-role-sits\">Where the role sits</h3>\n<p>Job titles in this area overlap, and the same title means different things at different companies. The table separates the jobs by what they own.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Role</th><th>Owns</th><th>A typical task</th><th>Hired for</th></tr></thead><tbody><tr><td>Inference engineer</td><td>Serving engine, model runtime, latency and cost per token</td><td>Cut p99 ITL by 30% on a new model</td><td>Engines, GPUs, quantization, profiling</td></tr><tr><td>Kernel or performance engineer</td><td>Individual GPU kernels</td><td>Write an FP8 attention kernel for a new GPU</td><td>CUDA, Triton, CUTLASS, hardware detail</td></tr><tr><td>ML platform or infrastructure engineer</td><td>Clusters, scheduling, deployment, observability</td><td>Autoscale model replicas on queue depth</td><td>Kubernetes, distributed systems, cloud</td></tr><tr><td>MLOps engineer</td><td>Pipelines from training to production</td><td>Automate evaluation and rollout of a new checkpoint</td><td>CI/CD, model registries, monitoring</td></tr><tr><td>Research engineer</td><td>Training runs and experiments</td><td>Scale a training job to 1,000 GPUs</td><td>Training frameworks, parallelism, numerics</td></tr><tr><td>Applied AI engineer</td><td>Products built on models</td><td>Build a retrieval assistant on an API</td><td>Prompting, RAG, evals, product sense</td></tr></tbody></table></div>\n<p>The inference engineer sits between the kernel engineer and the platform engineer. You need enough kernel knowledge to know when a kernel is the problem and to write a simple one, and enough platform knowledge to run the engine as a service. Small companies often merge all three jobs into one.</p>","headings":[{"id":"where-the-role-sits","title":"Where the role sits","depth":3}]},{"id":"what-employers-ask-for","title":"What employers ask for","html":"<h3 id=\"what-employers-ask-for\">What employers ask for</h3>\n<p>In a sample of inference postings from October 2026 (Together AI, Nebius, Intel, Hedra and Uniphore, listed under Further reading), the same requirements came up again and again:</p>\n<ul>\n<li><strong>A serving engine in production.</strong> Postings name vLLM, SGLang or TensorRT-LLM. Together AI's posting asks for familiarity with at least one of TensorRT-LLM, vLLM, SGLang or TGI.</li>\n<li><strong>GPU kernels.</strong> Postings expect you to read and debug CUDA, and treat writing kernels in CUDA, Triton or CUTLASS as a plus. Hedra's posting asks for building or optimizing GPU kernels when off-the-shelf implementations fall short.</li>\n<li><strong>Inference techniques.</strong> Postings list KV cache management, continuous batching, quantization and speculative decoding.</li>\n<li><strong>Profiling and measured results.</strong> Employers want experience with Nsight or similar profilers, the ability to find a workload's bottleneck, and speedups you can point to on real workloads.</li>\n<li><strong>Open-source work.</strong> Upstream contributions to vLLM or SGLang count in your favour, and Intel's posting makes upstreaming part of the job.</li>\n</ul>\n<p>Senior postings add distributed serving (tensor and expert parallelism, disaggregated prefill and decode) and multi-node operations. The handbook's parts follow this list:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>What postings ask for</th><th>Where this handbook teaches it</th></tr></thead><tbody><tr><td>Why a model is slow or fast on given hardware</td><td>Part 1, the model as a workload</td></tr><tr><td>GPU architecture, CUDA, Triton, profiling</td><td>Part 2, GPUs and kernels</td></tr><tr><td>PyTorch compilation, CUDA graphs, C++ extensions</td><td>Part 3, the runtime layer</td></tr><tr><td>Quantization (FP8, INT4, FP4) and quality evaluation</td><td>Part 4, compression</td></tr><tr><td>vLLM, SGLang, TensorRT-LLM internals, speculative decoding</td><td>Part 5, serving engines</td></tr><tr><td>Tensor, pipeline and expert parallelism, disaggregation</td><td>Part 6, distributed inference</td></tr><tr><td>LoRA and QLoRA, serving many adapters</td><td>Part 7, fine-tuning for serving</td></tr><tr><td>Containers, Kubernetes, autoscaling, observability</td><td>Part 8, production</td></tr><tr><td>On-device runtimes (optional)</td><td>Part 9, edge inference</td></tr><tr><td>The interview loop and your portfolio</td><td>Part 10, interviews</td></tr></tbody></table></div>","headings":[{"id":"what-employers-ask-for","title":"What employers ask for","depth":3}]},{"id":"how-interviews-test-it","title":"How interviews test it","html":"<h3 id=\"how-interviews-test-it\">How interviews test it</h3>\n<p>Loops differ by company, but most inference interviews draw on five kinds of round:</p>\n<ol>\n<li><strong>Coding.</strong> The language is usually Python and sometimes C++. Expect general problems, plus inference-flavoured ones: implement attention with a KV cache, a top-p sampler, a block allocator, or one step of a batching scheduler. Some companies ask for a small CUDA or Triton kernel.</li>\n<li><strong>ML systems deep dive.</strong> You get questions such as \"why is decode memory-bound?\", \"how much KV cache does this model need at 32K context?\", or \"when does speculative decoding make things slower?\" You are expected to work out numbers live.</li>\n<li><strong>System design.</strong> You design a service that serves a model to a given load within a latency target. The <a href=\"/learn/design-inference-platform\">AI system design questions</a> on this site are the practice set for this round.</li>\n<li><strong>Past-project deep dive.</strong> You explain something you built or sped up. Interviewers probe the baseline, how you measured, what you changed and what it cost. Projects with numbers beat projects without.</li>\n<li><strong>Take-home.</strong> Some companies ask you to make a kernel or a serving configuration faster and to write up what you did.</li>\n</ol>\n<p>The labs in this handbook produce material for rounds 1, 2 and 4: working code you wrote, numbers you measured, and write-ups you can talk through.</p>","headings":[{"id":"how-interviews-test-it","title":"How interviews test it","depth":3}]},{"id":"how-this-handbook-works","title":"How this handbook works","html":"<h3 id=\"how-this-handbook-works\">How this handbook works</h3>\n<p>Every chapter has the same parts:</p>\n<ul>\n<li><strong>Prerequisites</strong> at the top name the system design chapters to read first.</li>\n<li><strong>Concept sections</strong> work through the ideas with numbers. The numbers come from code you can run, and every printed output in a chapter is real output from that code.</li>\n<li><strong>A lab</strong> is tagged by the hardware it needs and has a task, a definition of done, and a reference solution folded under \"Reference solution\". Try the lab before opening the solution.</li>\n<li><strong>An interview angle</strong> lists questions the chapter prepares you for and what a strong answer contains.</li>\n<li><strong>Check yourself</strong> and <strong>Further reading</strong> close the chapter, as in the rest of the site.</li>\n</ul>\n<p>Labs use three hardware tiers:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Tag</th><th>What it needs</th><th>Where to get it</th></tr></thead><tbody><tr><td><strong>[Mac]</strong> or <strong>[any]</strong></td><td>A laptop. Apple silicon, any NVIDIA GPU, or a plain CPU</td><td>What you have</td></tr><tr><td><strong>[1 GPU]</strong></td><td>One NVIDIA GPU. An L4 or A10G covers most labs; an H100 for FP8</td><td>Hourly rental: RunPod, Lambda, Modal, Vast.ai, Colab</td></tr><tr><td><strong>[multi-GPU]</strong></td><td>2 to 8 GPUs in one machine</td><td>Hourly rental, a few hours per lab</td></tr></tbody></table></div>\n<p>Batch the GPU labs so one rental covers several, and log what each session cost. A line like \"measured on one rented L4 for $3\" is a good detail in an interview.</p>","headings":[{"id":"how-this-handbook-works","title":"How this handbook works","depth":3}]},{"id":"lab-set-up-your-lab","title":"Lab: set up your lab","html":"<h3 id=\"lab-set-up-your-lab\">Lab: set up your lab</h3>\n<p><strong>Tier: [any].</strong> This lab runs on any laptop and builds the workspace every later lab uses.</p>\n<p>Complete these tasks:</p>\n<ol>\n<li>Create a git repository, <code>inference-lab</code>, with folders <code>bench/</code>, <code>labs/</code>, <code>notes/</code> and <code>results/</code>.</li>\n<li>Create a Python 3.12 environment with <a href=\"https://docs.astral.sh/uv/\" target=\"_blank\" rel=\"noopener noreferrer\">uv</a> and install <code>torch</code>, <code>transformers</code> and <code>numpy</code>.</li>\n<li>Check that PyTorch sees an accelerator: <code>cuda</code> on NVIDIA, <code>mps</code> on Apple silicon.</li>\n<li>Write <code>bench/record.py</code> with a function <code>record(name, result)</code> that appends one JSON line to <code>results/results.jsonl</code>, holding the benchmark name, a UTC timestamp, the result, and the environment it ran in (Python, PyTorch, device name, CUDA and driver versions when present, and the git commit).</li>\n<li>Open an account with one GPU rental provider and note its hourly price for an L4 and an H100.</li>\n</ol>\n<p>The lab is done when <code>python bench/record.py</code> appends a line naming your device and <code>results.jsonl</code> is committed.</p>\n<p>The recorder matters because benchmark numbers without the versions and hardware they came from cannot be compared later. A software upgrade alone can move results a long way: <a href=\"https://vllm.ai/blog/2024-09-05-perf-update\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM reported</a> 2.7 times the throughput for version 0.6.0 over 0.5.3, on Llama 3 8B on one H100.</p>\n<p>The reference recorder is also available as a file, <a href=\"/labs/infra/record.py\">record.py</a>, to save into your lab repository as <code>bench/record.py</code>. Every Python file the handbook's labs use is in the site's repository under <code>public/labs/infra/</code>, and the scripts that need an NVIDIA GPU are kept apart in <code>public/labs/infra/gpu/</code>.</p>\n<details>\n<summary>Reference solution</summary>\n<p>Set up the repository and the environment:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">mkdir</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> inference-lab</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> &#x26;&#x26; </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">cd</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> inference-lab</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> &#x26;&#x26; </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">git</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> init</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">mkdir</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> bench</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> labs</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> notes</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> results</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">uv</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> venv</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --python</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3.12</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> &#x26;&#x26; </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">source</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> .venv/bin/activate</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">uv</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> pip</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> install</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> torch</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> transformers</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> numpy</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">python</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -c</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"import torch; print(torch.cuda.is_available(), torch.backends.mps.is_available())\"</span></span></code></pre>\n<p>Then write <code>bench/record.py</code>:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># bench/record.py: append one benchmark result, with the machine and software it ran on, to a log.</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> datetime</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> platform</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> subprocess</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sys</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> environment</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    env </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"python\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: platform.python_version(), </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"os\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: platform.platform(), </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"machine\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: platform.machine()}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> torch</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        env[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"torch\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> torch.</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">__version__</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> torch.cuda.is_available():</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            env[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"device\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> torch.cuda.get_device_name(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            env[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cuda\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> torch.version.cuda</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            env[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"gpu_count\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> torch.cuda.device_count()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        elif</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> torch.backends.mps.is_available():</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            env[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"device\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"mps\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    except</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> ImportError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        pass</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> platform.system() </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"Darwin\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        env[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cpu\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> subprocess.run([</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"sysctl\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"-n\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"machdep.cpu.brand_string\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">],</span></span>\n<span class=\"line\"><span style=\"color:#E36209;--shiki-dark:#FFAB70\">                                    capture_output</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">text</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">).stdout.strip()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># NVIDIA driver version, if nvidia-smi exists</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        env[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"driver\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> subprocess.run(</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"nvidia-smi\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"--query-gpu=driver_version\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"--format=csv,noheader\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">],</span></span>\n<span class=\"line\"><span style=\"color:#E36209;--shiki-dark:#FFAB70\">            capture_output</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">text</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">check</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">).stdout.split(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    except</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">FileNotFoundError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, subprocess.CalledProcessError):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        pass</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        env[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"git_commit\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> subprocess.run([</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"git\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"rev-parse\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"--short\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"HEAD\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">],</span></span>\n<span class=\"line\"><span style=\"color:#E36209;--shiki-dark:#FFAB70\">                                           capture_output</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">text</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">check</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">).stdout.strip()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    except</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">FileNotFoundError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, subprocess.CalledProcessError):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        pass</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> env</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> record</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(name: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, result: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, path: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"results/results.jsonl\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    row </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"name\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: name, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"time\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: datetime.datetime.now(datetime.timezone.utc).isoformat(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">timespec</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"seconds\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">           \"env\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: environment(), </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"result\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: result}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    with</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> open</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(path, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> f:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        f.write(json.dumps(row) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> row</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    row </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> record(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"smoke-test\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"note\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: sys.argv[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(sys.argv) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> else</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"setup works\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">})</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(json.dumps(row, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">indent</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span></code></pre>\n<p>This is the output on an Apple M2 laptop before the first commit, so there is no <code>git_commit</code> yet:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>{</span></span>\n<span class=\"line\"><span>  \"name\": \"smoke-test\",</span></span>\n<span class=\"line\"><span>  \"time\": \"2026-10-31T13:39:03+00:00\",</span></span>\n<span class=\"line\"><span>  \"env\": {</span></span>\n<span class=\"line\"><span>    \"python\": \"3.12.10\",</span></span>\n<span class=\"line\"><span>    \"os\": \"macOS-26.6.2-arm64-arm-64bit\",</span></span>\n<span class=\"line\"><span>    \"machine\": \"arm64\",</span></span>\n<span class=\"line\"><span>    \"torch\": \"2.15.0\",</span></span>\n<span class=\"line\"><span>    \"device\": \"mps\",</span></span>\n<span class=\"line\"><span>    \"cpu\": \"Apple M2\"</span></span>\n<span class=\"line\"><span>  },</span></span>\n<span class=\"line\"><span>  \"result\": {</span></span>\n<span class=\"line\"><span>    \"note\": \"setup works\"</span></span>\n<span class=\"line\"><span>  }</span></span>\n<span class=\"line\"><span>}</span></span></code></pre>\n<p>On a rented NVIDIA machine the same script adds <code>device</code> (for example \"NVIDIA L4\"), <code>cuda</code>, <code>gpu_count</code> and <code>driver</code>. Later labs import <code>record</code> and pass their measurements as <code>result</code>.</p>\n</details>","headings":[{"id":"lab-set-up-your-lab","title":"Lab: set up your lab","depth":3}]},{"id":"interview-angle","title":"Interview angle","html":"<h3 id=\"interview-angle\">Interview angle</h3>\n<ul>\n<li><strong>\"Tell me about a performance improvement you made.\"</strong> A strong answer has five parts: the baseline and how it was measured, the bottleneck and how you found it, the change, the result with the same measurement, and the cost or risk of the change. The labs are built to give you stories in this shape.</li>\n<li><strong>\"What does an inference engineer optimize for?\"</strong> The job optimizes latency (TTFT and ITL at a percentile), throughput per GPU and cost per token, while holding quality constant. A strong answer adds that these trade against each other through batch size, and that the right point depends on the product's latency target.</li>\n<li><strong>\"How is serving different from training?\"</strong> Training is one long throughput job with all data known in advance. Serving is many short, unpredictable requests with latency targets, where memory for the KV cache, not compute, usually limits how many users fit. <a href=\"/learn/training-infrastructure#training-compared-with-inference\">Training and fine-tuning infrastructure</a> compares them in detail.</li>\n</ul>","headings":[{"id":"interview-angle","title":"Interview angle","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Name the layers a request passes through between the API gateway and GPU memory, and one decision each layer makes.</summary>\n<p>The router picks a replica, for example the one holding the prompt's prefix in cache. The engine scheduler picks which requests join this step's batch and where their KV cache blocks go. The model runner turns the batch into GPU work, often by replaying a captured CUDA graph. The kernels do the matrix multiplies and attention, and they read weights and KV cache from GPU memory every step.</p>\n</details>\n<details>\n<summary>How does an inference engineer's job differ from a kernel engineer's and from an ML platform engineer's?</summary>\n<p>A kernel engineer makes individual GPU operations fast. A platform engineer runs clusters, deployments and autoscaling. The inference engineer owns the serving engine and the model's runtime between them: batching, memory, quantization, and the latency and cost of each token. The job needs enough kernel knowledge to find and fix a slow kernel, and enough platform knowledge to run the engine as a service.</p>\n</details>\n<details>\n<summary>Why record the software versions and hardware with every benchmark result?</summary>\n<p>Serving performance depends on the engine version, PyTorch, CUDA, the driver and the exact GPU. Without them, two results cannot be compared, and a later regression cannot be traced to its cause. An engine upgrade alone can move throughput by a large margin.</p>\n</details>\n<details>\n<summary>An interviewer asks how you would show that an optimization helped. What do you describe?</summary>\n<p>Describe a fixed workload (prompt and output lengths, request rate), a baseline measured with warmup and several runs, and the metric that matters, for example p99 ITL at a given throughput. Then describe the change made on its own, the same measurement repeated afterwards, and a quality check if the change could affect outputs. Report the spread, not only the best run.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li>The job postings sampled for this chapter in October 2026 were <a href=\"https://jobs.generalcatalyst.com/companies/together-ai-2/jobs/72517179-llm-inference-frameworks-and-optimization-engineer\" target=\"_blank\" rel=\"noopener noreferrer\">Together AI, LLM inference frameworks and optimization engineer</a>, <a href=\"https://www.dice.com/job-detail/736b5e4b-8610-411b-88bc-5b79f3d5607c\" target=\"_blank\" rel=\"noopener noreferrer\">Uniphore, senior inference engineer</a>, <a href=\"https://builtin.com/job/inference-optimization-engineer/11572766\" target=\"_blank\" rel=\"noopener noreferrer\">Hedra, inference optimization engineer</a>, <a href=\"https://jobpilot.suxrobgm.net/jobs/senior-machine-learning-engineer-llm-inference-optimization-at-nebius-group-05fe1a\" target=\"_blank\" rel=\"noopener noreferrer\">Nebius, LLM inference optimization</a> and <a href=\"https://intel.wd1.myworkdayjobs.com/en-US/External/job/AI-Infrastructure-Engineer_JR0286233\" target=\"_blank\" rel=\"noopener noreferrer\">Intel, AI infrastructure engineer</a>. Postings change or close, so some of these links may stop working.</li>\n<li><a href=\"https://www.techinterview.org/post/3233475331/ai-inference-infrastructure-interview-prep-2026-vllm-triton/\" target=\"_blank\" rel=\"noopener noreferrer\">AI inference infrastructure interview prep</a> and <a href=\"https://www.interviewcoder.co/blog/llm-engineer-interview-questions\" target=\"_blank\" rel=\"noopener noreferrer\">InterviewCoder's LLM engineer questions</a> are community question banks. They are useful for the shape of the questions, not as authorities.</li>\n<li>The <a href=\"https://docs.vllm.ai/\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM documentation</a> and the <a href=\"https://docs.sglang.io/\" target=\"_blank\" rel=\"noopener noreferrer\">SGLang documentation</a> cover the two open engines that most postings name.</li>\n<li><a href=\"https://vllm.ai/blog/2024-09-05-perf-update\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM v0.6.0: 2.7x throughput improvement and 5x latency reduction</a> is the vLLM team's report of how much one release changed performance on the same hardware.</li>\n<li><a href=\"https://docs.astral.sh/uv/\" target=\"_blank\" rel=\"noopener noreferrer\">uv</a> is the Python package and environment manager the labs use.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}