{"id":"graph-runtimes","title":"Compilers and graph runtimes","sections":[{"id":"overview","title":"Overview","html":"<p>Chapter 3.1 compiled a model just in time: <code>torch.compile</code> traced the decode step inside the serving process, on its first call, and the process kept the result in memory. The other way to deploy a model is to compile it ahead of time. A build step turns the model into an artifact, such as an ONNX file, an AOTInductor package or a TensorRT engine, and a runtime loads that artifact in production, sometimes without Python at all. This chapter covers the main ahead-of-time paths, what each one fixes at build time, and what that rigidity costs. It ends with the trend that matters most for large language models: NVIDIA's own TensorRT-LLM has moved from prebuilt TensorRT engines to a PyTorch program that captures CUDA graphs, the approach of vLLM and SGLang. The lab runs one small transformer four ways on a CPU and compares build time, speed and flexibility.</p>\n<p>Before this chapter, read <a href=\"/learn/pytorch-execution\">PyTorch execution for inference</a>, which explains graph capture, compilation and CUDA graphs.</p>","headings":[]},{"id":"ahead-of-time-or-just-in-time","title":"Ahead of time or just in time","html":"<h3 id=\"ahead-of-time-or-just-in-time\">Ahead of time or just in time</h3>\n<p>The two approaches differ in when the compiler runs and what it may assume. A just-in-time compiler such as <code>torch.compile</code> runs in the serving process, sees the real inputs, and can recompile when they change, falling back to Python for anything it cannot handle. An ahead-of-time compiler runs before deployment, must capture the whole model without falling back, and fixes in advance what the artifact will accept.</p>\n<figure class=\"diagram\" data-text=\"PyTorch model -> torch.compile (first call); torch.compile -> Serving process; PyTorch model -> torch.export; torch.export -> AOTInductor; AOTInductor -> .pt2 package; torch.export -> ONNX file; ONNX file -> ONNX Runtime; ONNX file -> TensorRT builder; TensorRT builder -> TensorRT engine\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 640 443\" width=\"640\" role=\"img\" aria-label=\"Two ways from a PyTorch model to running code\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"116\" y1=\"135\" x2=\"201.3\" y2=\"71\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"116\" y1=\"189\" x2=\"201.3\" y2=\"253\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"276\" y1=\"255\" x2=\"361.3\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"282\" x2=\"327\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"282\" x2=\"487\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"400\" y1=\"314\" x2=\"400\" y2=\"373\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"402\" x2=\"487\" y2=\"402\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">PyTorch model</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"46.5\">torch.compile</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"46.5\">Serving process</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"258\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"286.5\">torch.export</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"166.5\">AOTInductor</tspan></text><path class=\"dg-node dg-store\" d=\"M492,140 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,140 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"169.5\">.pt2 package</tspan></text><path class=\"dg-node dg-store\" d=\"M332,260 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,260 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"289.5\">ONNX file</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"258\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"286.5\">ONNX Runtime</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"378\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"406.5\">TensorRT builder</tspan></text><path class=\"dg-node dg-store\" d=\"M492,380 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,380 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"409.5\">TensorRT engine</tspan></text><rect class=\"dg-edge-bg\" x=\"122.7\" y=\"94\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"158.7\" y=\"107\" text-anchor=\"middle\">first call</text></svg></div><figcaption>Two ways from a PyTorch model to running code</figcaption></figure>\n<p>The top path is chapter 3.1's. The others start from <code>torch.export</code>, which captures the whole model as one graph, and then compile it with AOTInductor, convert it to ONNX for ONNX Runtime, or build a TensorRT engine from the ONNX file.</p>","headings":[{"id":"ahead-of-time-or-just-in-time","title":"Ahead of time or just in time","depth":3}]},{"id":"torchexport-one-graph-or-an-error","title":"torch.export: one graph or an error","html":"<h3 id=\"torchexport-one-graph-or-an-error\">torch.export: one graph or an error</h3>\n<p><code>torch.export.export</code> traces a model with example inputs and returns an <code>ExportedProgram</code>, a single graph that contains only PyTorch's ATen operators, plus any custom operators, together with the model's weights. Its documentation states the key difference from <code>torch.compile</code>: export aims for a full graph, so it raises an error when it reaches something it cannot trace, where <code>torch.compile</code> would insert a graph break and continue. Since PyTorch 2.8, export traces in non-strict mode by default, running the model's Python code normally and recording the tensor operations; <code>strict=True</code> traces through TorchDynamo instead.</p>\n<p>Shapes are static unless declared otherwise. A dimension that should vary is marked with <code>torch.export.Dim</code>, with an optional range, and the exported program checks every input against those ranges at run time. Python control flow that depends on a tensor's values cannot be exported; it has to be rewritten with <code>torch.cond</code>, which keeps both branches in the graph and chooses between them at run time. <code>torch.cond</code> is still marked as a prototype feature. The lab's <a href=\"/labs/infra/export_runtimes.py\">export_runtimes.py</a> exported a 4-layer, 50-million-parameter decoder with a sequence length between 2 and 1,024, then tried the cases that fail:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>2. torch.export: 0.9 s, one graph of 84 operations, sequence length allowed from 2 to 1024</span></span>\n<span class=\"line\"><span>   The first operations: sym_size.int, embedding.default, rms_norm.default, linear.default, view.default, transpose.int</span></span>\n<span class=\"line\"><span>   A 2,000-token input is rejected: AssertionError: Guard failed: ids.size()[1] &#x3C;= 1024</span></span>\n<span class=\"line\"><span>   Exported without dynamic_shapes, a 300-token input is rejected: Guard failed: ids.size()[1] == 128</span></span>\n<span class=\"line\"><span>   A branch on a tensor's value cannot be exported: GuardOnDataDependentSymNode</span></span>\n<span class=\"line\"><span>   Rewritten with torch.cond it exports, keeping both branches: ['cond']</span></span></code></pre>\n<p>Every limit is explicit: the exported program states the shapes it accepts and refuses others, instead of recompiling. An exported program can be saved with <code>torch.export.save</code> as a <code>.pt2</code> file, a zip archive in PyTorch's PT2 archive format.</p>","headings":[{"id":"torchexport-one-graph-or-an-error","title":"torch.export: one graph or an error","depth":3}]},{"id":"aotinductor","title":"AOTInductor","html":"<h3 id=\"aotinductor\">AOTInductor</h3>\n<p>AOTInductor is TorchInductor run ahead of time. <code>torch._inductor.aoti_compile_and_package</code> takes an exported program and produces a <code>.pt2</code> package of compiled shared libraries, which its documentation describes as crafted for deployment in non-Python environments. The package can be loaded back in Python with <code>torch._inductor.aoti_load_package</code>, or in a C++ program with <code>torch::inductor::AOTIModelPackageLoader</code>, so a C++ service can run the model with no Python interpreter, although it still links libtorch, PyTorch's C++ library. Like a TensorRT engine, the package holds native code compiled for the kind of processor it was built on. At run time, inputs must have the same sizes, types and strides as at export, except for dimensions exported as dynamic.</p>\n<p>The kernels are the same kind <code>torch.compile</code> generates, so the speed is similar; what changes is when the compile happens. In the lab, AOTInductor compiled the exported decoder in 4.3 seconds into a 200 MB package that contains the weights, and the package ran a 128-token forward pass in 18.0 ms, against 17.8 ms for <code>torch.compile</code> and 19.2 ms in eager mode. Loading the package took 2.3 seconds, which is part of every start-up.</p>","headings":[{"id":"aotinductor","title":"AOTInductor","depth":3}]},{"id":"onnx-and-onnx-runtime","title":"ONNX and ONNX Runtime","html":"<h3 id=\"onnx-and-onnx-runtime\">ONNX and ONNX Runtime</h3>\n<p>ONNX is an open format for machine learning models: a graph of operators from a common, versioned set. Every ONNX graph records an opset version, which fixes the definition of every operator in it, so a runtime knows exactly what the file means. PyTorch's exporter, <code>torch.onnx.export</code>, has used <code>torch.export</code> underneath by default since PyTorch 2.9 (the <code>dynamo=True</code> option), and takes the same kind of <code>dynamic_shapes</code> argument.</p>\n<p>ONNX Runtime runs ONNX files through execution providers, the back ends for particular hardware: the default CPU provider, CUDA, TensorRT, OpenVINO, Core ML, DirectML and others. A session is given a list of providers in priority order, and each node of the graph runs on the first provider that supports it. When a session is created, ONNX Runtime applies graph optimizations, such as fusing operators and removing redundant ones, at basic, extended and layout levels, all enabled by default; setting <code>optimized_model_filepath</code> saves the optimized graph to disk, and later sessions can load that file with optimizations disabled, so that the work is not repeated at every start-up. The saved file is valid only for the execution providers and hardware it was optimized for. For language models, the separate onnxruntime-genai library adds the generation loop around ONNX Runtime: tokenization, the KV cache, sampling and search.</p>\n<p>In the lab, exporting the decoder to ONNX took 2.7 seconds and creating an optimized CPU session 0.1 seconds, but ONNX Runtime ran the forward pass in 25.7 ms, about a third slower than eager PyTorch's 19.2 ms on the same 4 threads. A different runtime is not automatically a faster one. For this model on an Apple M2, ONNX Runtime's default CPU provider was simply slower than PyTorch's own CPU kernels; on other hardware, or with another execution provider, the result can reverse. That is why every change of runtime needs its own measurement.</p>","headings":[{"id":"onnx-and-onnx-runtime","title":"ONNX and ONNX Runtime","depth":3}]},{"id":"tensorrt","title":"TensorRT","html":"<h3 id=\"tensorrt\">TensorRT</h3>\n<p>TensorRT is NVIDIA's ahead-of-time compiler and runtime for its GPUs. Its builder takes a network, usually from an ONNX file, and produces an engine. During the build it fuses layers and selects the fastest available kernel for each layer on the target GPU by timing the candidates, which NVIDIA calls tactics. Because the choice is made by measurement, its documentation notes that when two implementations take similar time, system noise can decide which one is chosen. The build is meant to be an offline step. NVIDIA's documentation says it can take from seconds for small networks to tens of minutes or more for large models, and offers a timing cache, reusable across builds on the same device and versions, to shorten it.</p>\n<p>An engine is tied to where it was built. In the words of TensorRT's API documentation, serialized engines are not cross-platform portable and are specific to the exact GPU model on which they were built. By default an engine also runs only with the TensorRT version that built it. Builds can opt into hardware compatibility, for example across all GPUs from the Ampere generation on, and into version compatibility, at some cost in speed.</p>\n<p>Dynamic shapes must be planned too. A network with dimensions that vary at run time needs at least one optimization profile, which gives each dynamic dimension a minimum, an optimal and a maximum size; TensorRT tunes the engine for the optimal size, and some operations have fast kernels only when all three are equal. Weights can be replaced without a full rebuild only if the engine was built as refittable. Since TensorRT 11.0, every network must also be strongly typed, with the precision of every tensor stated explicitly rather than chosen by the builder.</p>\n<p>Torch-TensorRT connects TensorRT to PyTorch. It compiles a model just in time as a <code>torch.compile</code> backend, <code>backend=\"tensorrt\"</code>, or ahead of time from a <code>torch.export</code> program. Its options choose operations to keep running in PyTorch, <code>torch_executed_ops</code>, and whether TensorRT must compile the whole graph, <code>require_full_compilation</code>.</p>","headings":[{"id":"tensorrt","title":"TensorRT","depth":3}]},{"id":"tensorrt-llm-and-the-return-to-pytorch","title":"TensorRT-LLM and the return to PyTorch","html":"<h3 id=\"tensorrt-llm-and-the-return-to-pytorch\">TensorRT-LLM and the return to PyTorch</h3>\n<p>TensorRT-LLM applied the TensorRT approach to language models. Its classic workflow had three steps: convert a Hugging Face checkpoint into TensorRT-LLM's format, build an engine with <code>trtllm-build</code>, and run it. The build fixed the limits the engine would serve: <code>max_batch_size</code>, the most requests it can schedule at once; <code>max_input_len</code>, the longest prompt; <code>max_seq_len</code>, the longest prompt plus output; and <code>max_num_tokens</code>, the most tokens in one batch after padding is removed. Its documentation notes that once an engine is built, the size of its activation memory cannot be changed. A new model, a new GPU type, a longer context limit or a new TensorRT-LLM version meant a new build.</p>\n<p>That workflow is now history. TensorRT-LLM 1.0, released in September 2025, made a PyTorch backend the default for its Python LLM API and for <code>trtllm-serve</code>. In July 2026, starting with the pre-release 1.3.0rc21, the TensorRT engine backend was removed entirely, along with <code>trtllm-build</code>, and PyTorch became the only execution backend; the latest stable release, 1.2.1, still includes the old path. The PyTorch backend loads Hugging Face checkpoints directly, runs NVIDIA's custom kernels from a PyTorch program, and by default captures CUDA graphs for batches in which every request is decoding. Using <code>torch.compile</code> for lightweight fusion, and piecewise CUDA graphs for batches that include prefill, are optional settings that its documentation still marks as prototypes; vLLM turns both on by default. Both serve requests with in-flight batching, also called continuous batching, over a paged KV cache.</p>\n<p>The reasons are visible in this part of the handbook. A language model's shapes change at every step, with the batch size, the prompt lengths and the KV cache length, so an engine built for fixed shapes needs wide profiles or many builds. Models change quickly, and a team that must rebuild an engine for every model, GPU and library version spends its time building. Meanwhile, chapter 3.1 showed that the largest win for decoding, removing launch overhead, comes from CUDA graphs, which a PyTorch-based engine gets without an ahead-of-time build. NVIDIA's custom kernels still matter, but they now plug into a PyTorch program instead of a prebuilt engine.</p>","headings":[{"id":"tensorrt-llm-and-the-return-to-pytorch","title":"TensorRT-LLM and the return to PyTorch","depth":3}]},{"id":"other-compilers-and-runtimes","title":"Other compilers and runtimes","html":"<h3 id=\"other-compilers-and-runtimes\">Other compilers and runtimes</h3>\n<p>A few more names come up in interviews and in deployment work:</p>\n<ul>\n<li><strong>OpenVINO</strong> is Intel's toolkit for converting, optimizing and running models on Intel hardware.</li>\n<li><strong>Apache TVM</strong> is a compiler that turns trained models into deployable modules for many kinds of hardware, and <strong>MLC LLM</strong>, built on it, is a compiler and deployment engine for language models.</li>\n<li><strong>XLA</strong> is the compiler behind JAX, and all code that runs on Google's TPUs must be compiled by it.</li>\n<li><strong>Core ML</strong> is Apple's framework for running models on the CPU, GPU and Neural Engine of Apple devices, and <strong>MLX</strong> is Apple's NumPy-like array framework for machine learning on Apple silicon, which evaluates lazily.</li>\n<li><strong>llama.cpp</strong> runs language models in C and C++ on top of the GGML tensor library, which builds a computation graph from each model's operations and runs it on CPUs and GPUs.</li>\n</ul>","headings":[{"id":"other-compilers-and-runtimes","title":"Other compilers and runtimes","depth":3}]},{"id":"when-an-ahead-of-time-engine-wins","title":"When an ahead-of-time engine wins","html":"<h3 id=\"when-an-ahead-of-time-engine-wins\">When an ahead-of-time engine wins</h3>\n<p>An ahead-of-time artifact is the better choice when its rigidity costs little or is itself a requirement:</p>\n<ul>\n<li><strong>No Python in production</strong>, such as a C++ service, a mobile app or an embedded device. AOTInductor packages, ONNX Runtime, TensorRT and Core ML all run without a Python interpreter.</li>\n<li><strong>Fixed or narrow shapes</strong>, as in image classifiers, embedding models with bounded inputs, rerankers and speech models with fixed windows. One profile, or a few, covers all traffic.</li>\n<li><strong>Start-up time matters</strong>, for example when containers scale from zero, and a just-in-time compile of 20 or 30 seconds per process is unacceptable. A compile cache shortens it, as chapter 3.1 described, but a prebuilt artifact removes it.</li>\n<li><strong>A vendor's runtime has faster kernels for the target hardware</strong>, which only a measurement can show.</li>\n</ul>\n<p>An ahead-of-time artifact costs build time, an artifact per GPU type and library version, a fixed envelope of shapes and limits, debugging at a distance from the source code, and a lag before new model architectures are supported. For large language models on data-centre GPUs, the major engines have chosen PyTorch programs with CUDA graphs captured at start-up, compiled just in time in vLLM's case, as the better trade.</p>","headings":[{"id":"when-an-ahead-of-time-engine-wins","title":"When an ahead-of-time engine wins","depth":3}]},{"id":"lab-one-model-four-runtimes","title":"Lab: one model, four runtimes","html":"<h3 id=\"lab-one-model-four-runtimes\">Lab: one model, four runtimes</h3>\n<p><strong>Tier: [Mac].</strong> The lab runs on a CPU in a few minutes and needs torch, onnx, onnxscript and onnxruntime, plus a C++ compiler for <code>torch.compile</code> and AOTInductor.</p>\n<p>The script is ready to download: <a href=\"/labs/infra/export_runtimes.py\">export_runtimes.py</a>. Run it with <code>python export_runtimes.py</code>. It prints the partial graph of the failing export to standard error, which is expected.</p>\n<p><strong>Part A, export.</strong> Export the decoder with a dynamic sequence length. Show what the exported program does with an input outside its range, what happens without <code>dynamic_shapes</code>, and how to export a branch on a tensor's value.</p>\n<p><strong>Part B, four runtimes.</strong> Run the model in eager mode, with <code>torch.compile</code>, in ONNX Runtime and as an AOTInductor package. For each, report the build time, the time per forward pass and the largest difference from eager outputs, at two sequence lengths.</p>\n<p><strong>Part C, a recommendation.</strong> In a few sentences, recommend a runtime for a CPU embedding service that must start in under two seconds and run without Python, and explain what you would measure on the production hardware before deciding.</p>\n<details>\n<summary>Reference solution</summary>\n<p>The script's output on an Apple M2 MacBook with 8 GB of memory, on its first run with an empty compile cache:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>arm64, Darwin, PyTorch 2.15.0, 4 CPU threads; a 4-layer decoder with 50M parameters</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>2. torch.export: 0.9 s, one graph of 84 operations, sequence length allowed from 2 to 1024</span></span>\n<span class=\"line\"><span>   The first operations: sym_size.int, embedding.default, rms_norm.default, linear.default, view.default, transpose.int</span></span>\n<span class=\"line\"><span>   A 2,000-token input is rejected: AssertionError: Guard failed: ids.size()[1] &#x3C;= 1024</span></span>\n<span class=\"line\"><span>   Exported without dynamic_shapes, a 300-token input is rejected: Guard failed: ids.size()[1] == 128</span></span>\n<span class=\"line\"><span>   A branch on a tensor's value cannot be exported: GuardOnDataDependentSymNode</span></span>\n<span class=\"line\"><span>   Rewritten with torch.cond it exports, keeping both branches: ['cond']</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>3. A forward pass over 128 tokens (outputs also checked at 300 tokens)</span></span>\n<span class=\"line\"><span>                             build   per pass   max diff</span></span>\n<span class=\"line\"><span>   Eager PyTorch              0.0s     19.2 ms    0.0e+00</span></span>\n<span class=\"line\"><span>   torch.compile             19.6s     17.8 ms    1.9e-06   compiled on first call, in the serving process</span></span>\n<span class=\"line\"><span>   ONNX Runtime (CPU)         2.8s     25.7 ms    3.6e-06   export 2.7 s + session 0.1 s; onnxruntime 1.31.1</span></span>\n<span class=\"line\"><span>   AOTInductor package        5.2s     18.0 ms    1.9e-06   export 0.9 s + compile 4.3 s; 200 MB file, loads in 2.3 s</span></span></code></pre>\n<p><em>Part A.</em> The exported program enforces its declared range: a 2,000-token input fails a guard, and a program exported without <code>dynamic_shapes</code> accepts only its example's 128 tokens. A Python <code>if</code> on a tensor's sum cannot be exported, because the branch depends on data that does not exist at export time; <code>torch.cond</code> exports, with both branches in the graph.</p>\n<p><em>Part B.</em> All three alternatives match eager outputs to within about 4 × 10⁻⁶ at both lengths, which is float32 rounding. <code>torch.compile</code> and AOTInductor generate similar kernels and take 6 to 7% less time than eager, a difference between them of 0.2 ms that is within run-to-run noise; a second run of the script gave 18.2 ms and 17.4 ms. Their real difference is when compilation happens. <code>torch.compile</code>'s 19.6 seconds are paid in the serving process, on the first call and again when a second sequence length makes it recompile with a dynamic dimension; the second run compiled in 4.7 seconds, thanks to PyTorch's on-disk compile caches, kept in a <code>torchinductor_&#x3C;user></code> folder in the system's temporary directory. AOTInductor's 5.2 seconds are paid once, before deployment. ONNX Runtime built fastest but ran slowest here.</p>\n<p><em>Part C.</em> Both AOTInductor and ONNX Runtime can run without Python and do no compilation at start-up, and the lab's numbers pull in opposite directions. AOTInductor matched <code>torch.compile</code>'s speed, but loading its 200 MB package took 2.3 seconds, already over the two-second budget. ONNX Runtime created its session in 0.1 seconds but ran each pass about a third slower than eager PyTorch. ONNX Runtime is the safer recommendation for the start-up requirement, with AOTInductor worth trying if a C++ service can load its package faster than this Python measurement did. Before deciding, measure the start-up time of the real service, from process start to the first answer, the latency and throughput at the real input lengths, and the outputs against eager PyTorch, on the production CPU, because the ranking of runtimes changes from one CPU to another.</p>\n</details>","headings":[{"id":"lab-one-model-four-runtimes","title":"Lab: one model, four runtimes","depth":3}]},{"id":"interview-angle","title":"Interview angle","html":"<h3 id=\"interview-angle\">Interview angle</h3>\n<ul>\n<li><strong>\"When would you use TensorRT or another ahead-of-time engine instead of vLLM?\"</strong> Use one for models with fixed or narrow shapes, such as vision, embedding or speech models, for deployments without Python, or when a measurement shows the vendor's engine is faster on the target hardware. For large language models on data-centre GPUs, engines built on PyTorch with CUDA graphs are the norm, and TensorRT-LLM itself moved to that design.</li>\n<li><strong>\"What is fixed when you build a TensorRT engine?\"</strong> A build fixes the GPU model, the TensorRT version, the weights, the range of every dynamic dimension through optimization profiles, the kernels chosen for each layer, and, for TensorRT-LLM's classic engines, limits such as the maximum batch size, input length, sequence length and tokens per batch. Changing any of them means a rebuild, unless the engine was built with the matching compatibility or refit options.</li>\n<li><strong>\"How is torch.export different from torch.compile?\"</strong> Export captures the whole model ahead of time as one graph of ATen operators and fails on anything it cannot capture, with shapes and their ranges declared up front. <code>torch.compile</code> compiles just in time inside the process, tolerates graph breaks by falling back to Python, and recompiles when shapes change.</li>\n<li><strong>\"A colleague switched a model to ONNX Runtime and it got slower. Is that a bug?\"</strong> Not necessarily. Each runtime's kernels, fusions and execution providers suit some hardware and shapes better than others, and the lab found ONNX Runtime's default CPU provider slower than eager PyTorch on an M2. Check the execution provider in use, the optimization level, the thread settings and the shapes, then measure again.</li>\n</ul>","headings":[{"id":"interview-angle","title":"Interview angle","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Why does torch.export raise an error on a Python if over a tensor's value, when torch.compile would only insert a graph break?</summary>\n<p><code>torch.compile</code> runs in the live process, so it can run the <code>if</code> in Python with the real value and continue in a new graph. Export must produce a single graph that will later run without the original Python code, and the branch depends on data that does not exist at export time. Rewriting it with <code>torch.cond</code> puts both branches and the choice between them into the graph.</p>\n</details>\n<details>\n<summary>A TensorRT engine was built on an H100 and is copied to a server with an L40S and the same TensorRT version. What happens, and what could have been done differently?</summary>\n<p>By default the engine will not run: serialized engines are specific to the GPU model they were built on. The options are to build a separate engine on the L40S, or to build with hardware compatibility enabled, which makes an engine usable across GPUs from the Ampere generation on, at some cost in speed.</p>\n</details>\n<details>\n<summary>In the lab, torch.compile and AOTInductor ran at almost the same speed. What actually differs between them?</summary>\n<p>They differ in when and where compilation happens, and in what the result depends on. <code>torch.compile</code> compiles inside the serving process on the first call, recompiles if shapes or other guarded assumptions change, and needs Python. AOTInductor compiles once before deployment into a package whose accepted shapes are fixed at export, and the package can be loaded by a C++ program without Python.</p>\n</details>\n<details>\n<summary>Why did TensorRT-LLM make PyTorch its default backend and later remove the TensorRT engine backend?</summary>\n<p>Language model serving has shapes that change at every step and models that change every few weeks, so engines built for fixed limits needed frequent rebuilds per model, GPU and version. A PyTorch backend loads checkpoints directly and gets most of its speed from NVIDIA's custom kernels and from CUDA graphs captured at start-up, with <code>torch.compile</code> fusion and piecewise CUDA graphs available as options, all without a separate build step. It is the design vLLM and SGLang already used.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://docs.pytorch.org/docs/2.15/user_guide/torch_compiler/export.html\" target=\"_blank\" rel=\"noopener noreferrer\">torch.export</a> explains exported programs, dynamic shapes and the IR, and <a href=\"https://docs.pytorch.org/docs/2.15/higher_order_ops/cond.html\" target=\"_blank\" rel=\"noopener noreferrer\">torch.cond</a> documents data-dependent control flow.</li>\n<li><a href=\"https://docs.pytorch.org/docs/2.15/user_guide/torch_compiler/torch.compiler_aot_inductor.html\" target=\"_blank\" rel=\"noopener noreferrer\">AOTInductor</a> shows how to compile and package an exported program and load it from Python or C++.</li>\n<li><a href=\"https://docs.pytorch.org/docs/2.15/onnx_export.html\" target=\"_blank\" rel=\"noopener noreferrer\">torch.onnx</a> documents the exporter, and the <a href=\"https://onnx.ai/onnx/intro/concepts.html\" target=\"_blank\" rel=\"noopener noreferrer\">ONNX concepts page</a> explains graphs, operators and opsets.</li>\n<li><a href=\"https://onnxruntime.ai/docs/execution-providers/\" target=\"_blank\" rel=\"noopener noreferrer\">ONNX Runtime's execution providers</a> and <a href=\"https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html\" target=\"_blank\" rel=\"noopener noreferrer\">graph optimizations</a> explain how a session chooses kernels and transforms the graph.</li>\n<li><a href=\"https://docs.nvidia.com/deeplearning/tensorrt/latest/architecture/how-trt-works.html\" target=\"_blank\" rel=\"noopener noreferrer\">How TensorRT works</a>, <a href=\"https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/engine-compatibility.html\" target=\"_blank\" rel=\"noopener noreferrer\">engine compatibility</a> and <a href=\"https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/dynamic-shapes-basics.html\" target=\"_blank\" rel=\"noopener noreferrer\">dynamic shapes</a>, from NVIDIA's TensorRT documentation, cover the builder, portability and optimization profiles.</li>\n<li><a href=\"https://nvidia.github.io/TensorRT-LLM/legacy/tensorrt-backend-removal.html\" target=\"_blank\" rel=\"noopener noreferrer\">TensorRT-LLM's note on the removal of the TensorRT backend</a> and its page on <a href=\"https://nvidia.github.io/TensorRT-LLM/features/torch_compile_and_piecewise_cuda_graph.html\" target=\"_blank\" rel=\"noopener noreferrer\">torch.compile and piecewise CUDA graphs</a> describe the current design.</li>\n<li><a href=\"https://docs.pytorch.org/TensorRT/\" target=\"_blank\" rel=\"noopener noreferrer\">Torch-TensorRT</a> explains both its just-in-time and ahead-of-time paths.</li>\n<li><a href=\"https://docs.vllm.ai/en/stable/design/torch_compile.html\" target=\"_blank\" rel=\"noopener noreferrer\">vLLM's torch.compile integration</a> explains how a just-in-time engine compiles before serving and caches the result.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}