{"id":"cpp-syntax","title":"C++ syntax for Python programmers","sections":[{"id":"overview","title":"Overview","html":"<p>You already know what a loop is, what a function is and how an <code>if</code> works, so this chapter does not teach programming again. It teaches the differences, and there are two kinds. The first kind is cosmetic: braces instead of indentation, semicolons, types written out. You will absorb those in an afternoon. The second kind is dangerous: places where C++ compiles your code without complaint and then computes a different answer from the one Python would give. Integer division, overflow and comparisons between signed and unsigned numbers are the classic examples, and they are exactly the operations that quantization code, tensor indexing and tokenizers do all day.</p>\n<p>So the chapter moves fast through the phrasebook (variables, control flow, functions, enums, namespaces) and then slows down for the silent wrong answers, with a quiz where you predict what each snippet prints. The lab writes a small dtype table of the kind every tensor library has, quantizes a few weights to int8, and packs two 4-bit numbers into one byte, which is how 4-bit models are stored.</p>\n<p>Before this chapter, read <a href=\"/learn/cpp-toolchain\">the toolchain chapter</a>, which explains how a C++ file becomes a program. The lab builds with the same CMake commands.</p>","headings":[]},{"id":"a-first-program-line-by-line","title":"A first program, line by line","html":"<h3 id=\"a-first-program-line-by-line\">A first program, line by line</h3>\n<p>Here is a complete C++ program. It is longer than its Python equivalent, and every extra line is there for a reason.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">#include</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> &#x3C;iostream></span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">#include</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> &#x3C;vector></span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> main</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() {</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">> logits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.5</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> best </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> logits[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">];</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x : logits) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> best) best </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">  }</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::cout </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;&#x3C;</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"max logit: \"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> &#x3C;&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> best </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;&#x3C;</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> '</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<ul>\n<li><code>#include &#x3C;iostream></code> pastes in the declarations of the standard input and output streams. Python's <code>import</code> loads a module at run time; <code>#include</code> copies text before compiling, as the previous chapter showed.</li>\n<li><code>int main()</code> is where the program starts. It returns an <code>int</code> to the operating system, where 0 means success. <code>main</code> is the one function allowed to leave out its <code>return</code>, in which case it returns 0.</li>\n<li><code>std::vector&#x3C;float></code> is a growable array of 32-bit floats, the closest thing to a Python list of numbers. The <code>std::</code> prefix says it comes from the standard library's namespace.</li>\n<li><code>float best = logits[0];</code> declares a variable with its type. The type never changes: <code>best</code> is a <code>float</code> for the rest of its life.</li>\n<li>Every statement ends with a semicolon, and braces mark blocks. Indentation is for humans only.</li>\n<li><code>std::cout &#x3C;&#x3C; ... &#x3C;&#x3C; '\\n'</code> prints. The <code>&#x3C;&#x3C;</code> operator sends each value to the stream in turn. Single quotes make a <code>char</code>, double quotes make a string literal.</li>\n</ul>\n<p>You will also see <code>std::printf</code> from C, which takes a format string like Python's old <code>%</code> formatting. C++23 added <code>std::print</code>, whose format strings look like those of Python's <code>str.format</code>, and newer standard libraries provide it. This course uses <code>std::cout</code> and <code>std::printf</code> because every compiler in use supports them.</p>","headings":[{"id":"a-first-program-line-by-line","title":"A first program, line by line","depth":3}]},{"id":"variables-types-and-auto","title":"Variables, types and auto","html":"<h3 id=\"variables-types-and-auto\">Variables, types and auto</h3>\n<p>In Python, a name points at an object, and the object carries its type. In C++, a variable <em>is</em> a piece of memory of a fixed size, and its type is decided when the program is compiled. That is the source of C++'s speed: the compiler knows that <code>best</code> is four bytes holding a float, so <code>x > best</code> becomes a single machine instruction with no type check at run time.</p>\n<p>There are several ways to initialize a variable, and the brace form is the safest:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> a </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">        // copy initialization, the familiar form</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> b{</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">         // brace initialization</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> c{};</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">          // braces with nothing inside: zero</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3.9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">      // compiles, silently stores 3</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> e{</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3.9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">       // error: narrowing conversion from double to int</span></span></code></pre>\n<p>Braces refuse <strong>narrowing conversions</strong>, ones that can lose information, such as a <code>double</code> to an <code>int</code>. The <code>=</code> form performs them silently. That difference is one reason modern C++ style prefers braces.</p>\n<p><code>auto</code> asks the compiler to use the type of the initializer. It is still static typing: the type is fixed at compile time, you just did not write it.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">auto</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 10</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">               // int</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">auto</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 10.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">             // double: a literal with a decimal point is a double</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">auto</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> y </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 10.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">            // float: the f suffix makes it a float</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">auto</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> big </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3'000'000'000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // too big for int, so it becomes long on Linux and macOS</span></span></code></pre>\n<p>The apostrophes are digit separators, which C++14 added for readability, like Python's underscores in <code>3_000_000_000</code>. Use <code>auto</code> when the type is obvious from the right-hand side or long to write, and write the type when it carries meaning, such as <code>std::int8_t</code> for a quantized weight.</p>\n<h4 id=\"const-and-constexpr\">const and constexpr</h4>\n<p><code>const</code> means the variable cannot be changed after it is initialized. Its value may come from anywhere, including user input. <code>constexpr</code> means more: the value must be computable while compiling, so the compiler can use it as a constant, for example as the size of a fixed array.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> batch </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> read_batch_size</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">();</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // fine: fixed once set, known only at run time</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">constexpr</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> head_dim </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 128</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">         // fine: known while compiling</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">constexpr</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> vocab </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> read_vocab</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">();</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // error unless read_vocab is constexpr and can run at compile time</span></span></code></pre>\n<p>Make every variable that does not change <code>const</code>. It costs nothing, it documents intent, and the compiler then catches the accidental assignment. A function can also be <code>constexpr</code>, which lets the compiler run it at compile time when its arguments are constants. The lab uses that to test its int4 packing before the program even exists.</p>\n<h4 id=\"a-variable-you-forgot-to-initialize\">A variable you forgot to initialize</h4>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> count;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">      // no initializer</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">count </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     // reads a value nobody set</span></span></code></pre>\n<p>In Python, using a name before assigning it raises <code>NameError</code>. In C++, a local variable of a built-in type without an initializer holds whatever bits were already in that memory, and reading it is undefined behavior: the program may print a random number, or the optimizer may do something stranger. Chapter 5 covers undefined behavior and the tools that catch it. The habit that prevents this case is simple: always initialize, and <code>int count{};</code> costs nothing.</p>","headings":[{"id":"variables-types-and-auto","title":"Variables, types and auto","depth":3},{"id":"const-and-constexpr","title":"const and constexpr","depth":4},{"id":"a-variable-you-forgot-to-initialize","title":"A variable you forgot to initialize","depth":4}]},{"id":"control-flow","title":"Control flow","html":"<h3 id=\"control-flow\">Control flow</h3>\n<p>The control flow is familiar, with a few additions worth knowing.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n; </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">++</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i) {</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"> /* classic counted loop */</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> }</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x : logits) {</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"> /* read each element (a copy of it) */</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> }</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float&#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x : logits) { x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.5</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; }</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">        // &#x26; means x refers to the element itself, so this halves them</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> auto&#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> row : matrix) {</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"> /* read without copying */</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> }</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">while</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">!</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">queue.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">empty</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">()) {</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"> /* ... */</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> }</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sign </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">?</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> :</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> -</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // the conditional operator, Python's \"1 if x >= 0 else -1\"</span></span></code></pre>\n<p>The range-based <code>for</code> is the counterpart of Python's <code>for x in xs</code>. Without the <code>&#x26;</code>, each element is copied into <code>x</code>. That is free for a <code>float</code> but expensive for a large object, which is why <code>const auto&#x26;</code> is the usual way to read elements. Chapter 4 explains references properly.</p>\n<p>Since C++17, an <code>if</code> can declare a variable that lives only for that <code>if</code> and its <code>else</code>:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">auto</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> it </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> vocab.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">find</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(word); it </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">!=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> vocab.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">end</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">()) {</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">  token_id </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> it->second;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>This is close to Python's <code>if (it := vocab.get(word)) is not None:</code>, and it keeps <code>it</code> from leaking into the rest of the function.</p>\n<h4 id=\"switch-and-fall-through\">switch and fall-through</h4>\n<p><code>switch</code> jumps to the <code>case</code> that matches an integer or enum value. It does not work on strings. Its one surprise is that execution <strong>falls through</strong> into the next case unless a <code>break</code> or <code>return</code> stops it:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">switch</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (dtype) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  case</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> DType</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::F16:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  case</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> DType</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::BF16:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    bytes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">      // both cases share this code: intended</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    break</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  case</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> DType</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::I8:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    bytes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">      // forgot the break: execution continues into the next case</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  case</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> DType</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::F32:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    bytes </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">      // so an I8 tensor ends up with 4 bytes per element</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    break</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>Empty cases stacked together, like <code>F16</code> and <code>BF16</code>, are a normal idiom. A case with code that falls through is usually a bug, and GCC warns about it under <code>-Wextra</code>. When you mean it, write <code>[[fallthrough]];</code> at the end of the case, which tells both the compiler and the next reader that it is deliberate. A second useful warning: when you <code>switch</code> over an enum without a <code>default</code>, <code>-Wall</code> warns about any enumerator you did not handle. Add <code>DType::FP8</code> to the enum a year later, and the compiler lists every switch that needs a new case.</p>","headings":[{"id":"control-flow","title":"Control flow","depth":3},{"id":"switch-and-fall-through","title":"switch and fall-through","depth":4}]},{"id":"functions","title":"Functions","html":"<h3 id=\"functions\">Functions</h3>\n<p>A function declares the type of each parameter and of its result:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> mean</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">></span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> xs</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sum </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x : xs) sum </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sum </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> static_cast&#x3C;float></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(xs.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">size</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">());</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<h4 id=\"arguments-are-copied-unless-you-say-otherwise\">Arguments are copied unless you say otherwise</h4>\n<p>This is the difference that costs the most performance. In Python, passing a list to a function passes a reference to the same list. In C++, a parameter declared as <code>std::vector&#x3C;float> xs</code> receives a <strong>copy</strong>, element by element. Call a function like that with a vector of a million floats inside a loop, and every call allocates and copies 4 MB before doing any work. Writing the parameter as <code>const std::vector&#x3C;float>&#x26; xs</code> passes a reference instead: no copy, and <code>const</code> promises the function will not modify the caller's data. For small types such as <code>int</code>, <code>float</code> or <code>DType</code>, pass by value; for anything that owns memory, pass by <code>const&#x26;</code> unless you need your own copy. Chapters 4 and 7 return to this in depth.</p>\n<h4 id=\"overloading\">Overloading</h4>\n<p>Two functions can share a name if their parameter types differ, and the compiler picks one from the argument types:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> absmax</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">></span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> xs</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">   absmax</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int8_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">></span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> xs</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span></code></pre>\n<p>Overloading is convenient, and occasionally confusing. Given <code>void f(long)</code> and <code>void f(double)</code>, the call <code>f(1)</code> does not compile: <code>1</code> is an <code>int</code>, converting it to <code>long</code> and converting it to <code>double</code> count as equally good conversions, and the compiler reports the call as ambiguous instead of guessing. <code>f(1L)</code> or <code>f(1.0)</code> resolves it.</p>\n<h4 id=\"default-arguments-and-no-keyword-arguments\">Default arguments, and no keyword arguments</h4>\n<p>A parameter can have a default value, written once, in the declaration that callers see (normally in the header), and only for trailing parameters:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">// sampler.h</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sample</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">></span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> logits</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> temperature</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> top_k</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span></code></pre>\n<p>C++ has no keyword arguments, so there is no way to pass <code>top_k</code> while leaving <code>temperature</code> at its default. The usual workaround is a small options struct. C++20's designated initializers make it read almost like Python keywords, as long as the fields are named in the order they are declared:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">struct</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> SamplingParams</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> temperature </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> top_k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> top_p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sample</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">></span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> logits</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> SamplingParams</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> p</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> token </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sample</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(logits, {.top_k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 40</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, .top_p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">});</span></span></code></pre>","headings":[{"id":"functions","title":"Functions","depth":3},{"id":"arguments-are-copied-unless-you-say-otherwise","title":"Arguments are copied unless you say otherwise","depth":4},{"id":"overloading","title":"Overloading","depth":4},{"id":"default-arguments-and-no-keyword-arguments","title":"Default arguments, and no keyword arguments","depth":4}]},{"id":"enums-namespaces-and-structured-bindings","title":"Enums, namespaces and structured bindings","html":"<h3 id=\"enums-namespaces-and-structured-bindings\">Enums, namespaces and structured bindings</h3>\n<h4 id=\"enum-class\">enum class</h4>\n<p>Every tensor library has an enum of element types. PyTorch's C++ core calls it <code>c10::ScalarType</code>, and llama.cpp's ggml library calls it <code>ggml_type</code>. Here is a small one:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">enum</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> DType</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> : </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">uint8_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> { </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">F32</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">F16</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BF16</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">I8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">I4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> };</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">DType t </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> DType</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::BF16;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> raw </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> static_cast&#x3C;int></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(t);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // 2: converting to int must be explicit</span></span></code></pre>\n<p><code>enum class</code>, added in C++11, is a <strong>scoped</strong> enum: its names live inside it, so you write <code>DType::F16</code>, and it never converts to an integer by itself. The older plain <code>enum</code> puts its names in the surrounding scope, where <code>F16</code> can collide with something else, and converts to <code>int</code> silently, so <code>if (dtype == 3)</code> compiles. Prefer <code>enum class</code>. The <code>: std::uint8_t</code> after the name sets the underlying type, so each value fits in one byte when a file header or tensor descriptor stores it.</p>\n<h4 id=\"namespaces\">Namespaces</h4>\n<p>A namespace groups names to avoid collisions, much as a Python module does. Everything in the standard library is in <code>std</code>, which is why you write <code>std::vector</code>. You declare your own with <code>namespace tinymath { ... }</code>, and C++17 lets you nest them in one line, <code>namespace engine::kernels { ... }</code>. You can bring names into scope with <code>using std::vector;</code>, or everything with <code>using namespace std;</code>. The second form is common in tutorials and frowned on in real code, and it should never appear in a header, because it would leak into every file that includes it.</p>\n<h4 id=\"structured-bindings\">Structured bindings</h4>\n<p>C++17's structured bindings unpack a pair, a tuple or a simple struct into named variables, like Python's tuple unpacking:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">auto</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [q, scale] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> quantize_i8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(weights);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // the function returns a std::pair</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> auto&#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [token, id] : vocab) {</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    // vocab is a std::unordered_map&#x3C;std::string, int></span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // ...</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>The second form, iterating over a map's key-value pairs, is the counterpart of <code>for token, id in vocab.items():</code>.</p>","headings":[{"id":"enums-namespaces-and-structured-bindings","title":"Enums, namespaces and structured bindings","depth":3},{"id":"enum-class","title":"enum class","depth":4},{"id":"namespaces","title":"Namespaces","depth":4},{"id":"structured-bindings","title":"Structured bindings","depth":4}]},{"id":"fixed-width-integers-and-casts","title":"Fixed-width integers and casts","html":"<h3 id=\"fixed-width-integers-and-casts\">Fixed-width integers and casts</h3>\n<h4 id=\"int-is-not-a-fixed-size\">int is not a fixed size</h4>\n<p>Python has one integer type, and it grows as large as it needs to. C++ has several, and the standard only sets minimum sizes: <code>int</code> must hold at least 16 bits, <code>long</code> at least 32, <code>long long</code> at least 64. On every mainstream 64-bit system today <code>int</code> is 32 bits, but <code>long</code> differs between platforms: it is 64 bits on Linux and macOS, a model called LP64, and 32 bits on 64-bit Windows, called LLP64. Code that assumes <code>long</code> holds a 64-bit offset into a large file works on one and breaks on the other.</p>\n<p>When the size matters, and in AI infrastructure it usually does, use the fixed-width types from <code>&#x3C;cstdint></code>:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Type</th><th>Bits</th><th>Range</th></tr></thead><tbody><tr><td><code>std::int8_t</code></td><td>8</td><td>-128 to 127</td></tr><tr><td><code>std::uint8_t</code></td><td>8</td><td>0 to 255</td></tr><tr><td><code>std::int16_t</code></td><td>16</td><td>-32,768 to 32,767</td></tr><tr><td><code>std::uint16_t</code></td><td>16</td><td>0 to 65,535</td></tr><tr><td><code>std::int32_t</code></td><td>32</td><td>about -2.1 billion to 2.1 billion</td></tr><tr><td><code>std::uint32_t</code></td><td>32</td><td>0 to about 4.3 billion</td></tr><tr><td><code>std::int64_t</code></td><td>64</td><td>about -9.2 × 10^18 to 9.2 × 10^18</td></tr><tr><td><code>std::uint64_t</code></td><td>64</td><td>0 to about 1.8 × 10^19</td></tr></tbody></table></div>\n<p>These types are the vocabulary of quantization. An int8 checkpoint stores each weight as an <code>int8_t</code>. A 4-bit checkpoint packs two weights into each <code>uint8_t</code>, as the lab does. A file format reads its header fields as <code>uint32_t</code> and <code>uint64_t</code>, because a field's size on disk must not depend on the machine reading it. Since C++20, signed integers are required to use two's complement, the representation every real processor already used, so the bit patterns are portable as well as the sizes.</p>\n<p>Two more names appear everywhere. <code>std::size_t</code> is the unsigned type of sizes and indexes: <code>sizeof</code> returns it and <code>v.size()</code> returns it, and on 64-bit platforms it is 64 bits. <code>std::ptrdiff_t</code> is its signed partner, the type of the difference between two pointers. That <code>size()</code> is unsigned is the root of several traps below, and C++20 added <code>std::ssize(v)</code>, which returns the size as a signed number.</p>\n<p>Floating point needs one note too. Python's <code>float</code> is a C <code>double</code>, 64 bits. In C++, <code>float</code> is 32 bits and <code>double</code> is 64. NumPy defaults to 64-bit floats and PyTorch to 32-bit ones, and model weights live in 32 bits or fewer, so in this course <code>float</code> is the default and <code>double</code> is reserved for accumulators that need the precision. The 16-bit formats fp16 and bf16 have no standard built-in type you can count on: C++23 added the optional types <code>std::float16_t</code> and <code>std::bfloat16_t</code>, but compiler support is uneven, so libraries define their own, such as PyTorch's <code>c10::Half</code> and <code>c10::BFloat16</code>. The <a href=\"/learn/number-formats\">number formats chapter</a> of the AI infrastructure handbook explains what their bits mean.</p>\n<h4 id=\"casts\">Casts</h4>\n<p>Converting between types explicitly is called casting, and C++ has named casts so that each conversion says what kind it is:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> avg </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> static_cast&#x3C;float></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(sum) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> static_cast&#x3C;float></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(count);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // numeric conversion</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">auto</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> id </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> static_cast&#x3C;int></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">DType</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::I8);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                               // enum to integer</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">uint32_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> bits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">bit_cast</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">uint32_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">>(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">             // C++20: the raw bits of a float</span></span></code></pre>\n<p><code>static_cast</code> covers ordinary conversions between numbers and enums. <code>std::bit_cast</code>, in <code>&#x3C;bit></code>, reinterprets the bits of one type as another of the same size, which is how you inspect a float's sign, exponent and mantissa. Keeping the top 16 bits of a float's pattern gives a crude bf16 value, although real converters round rather than cut off the lower bits. C also has a cast written <code>(int)x</code>, which C++ accepts, but it will perform any of several conversions, including dangerous ones, without telling you which. Named casts are easy to search for and refuse conversions they were not designed for, so prefer them.</p>","headings":[{"id":"fixed-width-integers-and-casts","title":"Fixed-width integers and casts","depth":3},{"id":"int-is-not-a-fixed-size","title":"int is not a fixed size","depth":4},{"id":"casts","title":"Casts","depth":4}]},{"id":"silent-wrong-answers","title":"Silent wrong answers","html":"<h3 id=\"silent-wrong-answers\">Silent wrong answers</h3>\n<p>Here is the part of the chapter that matters most. Each of these compiles, runs, and gives an answer a Python programmer does not expect.</p>\n<p><strong>Integer division truncates.</strong> When both operands are integers, <code>/</code> divides as integers, and the conversion to a floating type, if any, happens afterwards. <code>double avg = total / count;</code> with two <code>int</code>s throws away the fraction before it reaches <code>avg</code>. Cast one operand first: <code>static_cast&#x3C;double>(total) / count</code>.</p>\n<p><strong>Negative division rounds toward zero.</strong> In C++, <code>-7 / 2</code> is -3 and <code>-7 % 2</code> is -1: the quotient truncates toward zero, and the remainder takes the sign of the left operand. Python's <code>//</code> rounds down, giving -4, and its <code>%</code> gives 1. Code ported from Python that computes a ring-buffer index or a hash bucket with <code>%</code> gives negative indexes for negative inputs.</p>\n<p><strong>Unsigned arithmetic wraps; signed overflow is undefined.</strong> Unsigned arithmetic wraps around modulo 2 to the power of the type's width, so a <code>uint32_t</code> holding 0 minus 1 is 4,294,967,295, and storing a value that is too big into an unsigned type wraps it the same way, so a <code>uint8_t</code> holding 250 becomes 4 after <code>x += 10</code>. Both are well defined, if surprising. Overflowing a <em>signed</em> integer in arithmetic, such as adding 1 to the largest <code>int</code>, is undefined behavior. The language makes no promise at all, and optimizers use that freedom: because overflow \"cannot happen\", a compiler may treat <code>x + 1 > x</code> as always true for a signed <code>x</code>. Chapter 5 shows this in action. Python never overflows, because its integers grow, so this whole class of bug is new.</p>\n<p><strong>Small types are promoted, and conversions back wrap.</strong> Arithmetic on types smaller than <code>int</code> first converts them to <code>int</code>. So with <code>std::int8_t q = 127;</code>, the expression <code>q + 1</code> is the <code>int</code> 128, with no overflow. Storing 128 back into <code>q</code> is a narrowing conversion, which C++20 defines to wrap, so <code>q</code> becomes -128. Quantization code clamps to the valid range <em>before</em> converting for this reason.</p>\n<p><strong>Signed and unsigned comparisons convert the signed side.</strong> When a signed and an unsigned integer of the same size meet in an expression, the signed one is converted to unsigned. The same happens when an <code>int</code> meets a larger unsigned type such as <code>size_t</code>. So <code>-1 &#x3C; v.size()</code> is false for every vector, because -1 becomes the largest <code>size_t</code>, 18,446,744,073,709,551,615 on a 64-bit platform. Compiling with <code>-Wall -Wextra</code> warns about signed and unsigned comparisons; keep those warnings on and read them.</p>\n<p><strong>size() minus one, and the loop that never ends.</strong> For an empty vector, <code>v.size() - 1</code> is not -1 but 18,446,744,073,709,551,615. A related classic is the backwards loop:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">size_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">size</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">--</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i) {</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"> /* ... */</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> }</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // never ends</span></span></code></pre>\n<p>An unsigned <code>i</code> is always <code>>= 0</code>, so after <code>i</code> reaches 0, <code>--i</code> wraps it to the largest value and the loop carries on, reading far outside the vector. GCC warns that the comparison is always true under <code>-Wextra</code>. Safe versions count with a signed index, <code>for (std::ptrdiff_t i = std::ssize(v) - 1; i >= 0; --i)</code>, or test before decrementing, <code>for (std::size_t i = v.size(); i-- > 0;)</code>.</p>\n<p><strong>int8_t prints as a character.</strong> On every common platform, <code>std::int8_t</code> is another name for <code>signed char</code> and <code>std::uint8_t</code> for <code>unsigned char</code>. Streams print character types as characters, so <code>std::cout &#x3C;&#x3C; q</code> with <code>q == 65</code> prints <code>A</code>, and a vector of int8 weights prints as a line of odd symbols. Write <code>+q</code> or <code>static_cast&#x3C;int>(q)</code> to print the number.</p>\n<p><strong>A float runs out of integers at 2^24.</strong> A 32-bit <code>float</code> keeps 24 bits of precision, so above 16,777,216 it can no longer represent every integer: 16,777,217 rounds to 16,777,216, and adding 1 to a float counter stuck at that value changes nothing. Python's <code>float</code> has 53 bits of precision, so the same surprise starts only above 9 × 10^15 there. It bites in C++ when you accumulate a sum of millions of values in <code>float</code>, which is why the dot product in chapter 1's lab adds its products into a <code>double</code>.</p>\n<p>Now test yourself. Each snippet below is one of these traps; decide what it prints before revealing the answer.</p>\n<div class=\"widget-slot\" data-widget=\"integer-traps\"></div>","headings":[{"id":"silent-wrong-answers","title":"Silent wrong answers","depth":3}]},{"id":"lab-a-dtype-table-int8-quantization-and-int4-packing","title":"Lab: a dtype table, int8 quantization and int4 packing","html":"<h3 id=\"lab-a-dtype-table-int8-quantization-and-int4-packing\">Lab: a dtype table, int8 quantization and int4 packing</h3>\n<p>The lab is two small programs, <a href=\"/labs/cpp/cpp-syntax/traps.cpp\"><code>traps.cpp</code></a> and <a href=\"/labs/cpp/cpp-syntax/dtypes.cpp\"><code>dtypes.cpp</code></a>, with a <a href=\"/labs/cpp/cpp-syntax/CMakeLists.txt\"><code>CMakeLists.txt</code></a> that builds both. Download the three files into one folder. The lab needs only a C++20 compiler and CMake, so it runs on a laptop. Build both with the usual two commands:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -S</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> .</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -B</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -DCMAKE_BUILD_TYPE=Debug</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --build</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build</span></span></code></pre>\n<h4 id=\"part-1-run-the-traps\">Part 1: run the traps</h4>\n<p><code>traps.cpp</code> contains the quiz snippets as one program, numbered the same way, plus one more: <code>v.size() - 1</code> for an empty vector. Before running <code>./build/traps</code>, write down what you expect each line to print. Then run it and compare. Read the compiler warnings from the build as well, because <code>-Wall -Wextra</code> flags several of the traps before the program ever runs.</p>\n<p>Then rebuild with the sanitizers on and run it again:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -S</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> .</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -B</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build-san</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -DCMAKE_BUILD_TYPE=Debug</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -DLAB_SANITIZE=ON</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --build</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build-san</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build-san/traps</span></span></code></pre>\n<p>UndefinedBehaviorSanitizer checks every signed arithmetic operation while the program runs, so line 7 should now be reported with a message of the form <code>runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'</code>. The overflow is no less undefined without the sanitizer; the tool just makes it visible.</p>\n<h4 id=\"part-2-the-dtype-table\">Part 2: the dtype table</h4>\n<p><code>dtypes.cpp</code> starts with the <code>DType</code> enum from this chapter and two <code>constexpr</code> functions that <code>switch</code> over it: <code>bits_of</code>, the bits per element, and <code>name_of</code>. From those, <code>bytes_for(t, n)</code> computes the storage for <code>n</code> elements, adding 7 bits before dividing by 8 so that a half-filled byte of 4-bit values rounds up. The program prints the weight memory of a 7-billion-parameter model in each dtype, which works out to 28.0 GB in f32, 14.0 GB in f16 and bf16, 7.0 GB in i8, and 3.5 GB in i4. Those numbers are the first thing an inference engineer computes about any model, because they decide which GPU it fits on.</p>\n<p>Try this: delete the <code>case DType::I4:</code> lines from <code>bits_of</code> and rebuild. The compiler warns that the enumeration value <code>I4</code> is not handled in the switch, and then the build stops at the <code>static_assert</code> that checks <code>bytes_for(DType::I4, 3)</code>, because <code>bits_of</code> now falls through to its final <code>return 0</code> for <code>I4</code>. The warning pointed at the real cause before the failing test did, and that is why the functions have no <code>default</code> case: a <code>default</code> would silence the warning and leave only the test to catch the mistake, or nothing at all if no test covered it.</p>\n<h4 id=\"part-3-symmetric-int8-quantization\">Part 3: symmetric int8 quantization</h4>\n<p><code>quantize_i8</code> maps a vector of floats to <code>int8_t</code> values with one shared scale. It finds the largest absolute value, sets <code>scale = max_abs / 127</code>, divides each weight by the scale, rounds to the nearest integer with <code>std::lround</code>, and clamps the result to [-127, 127] with <code>std::clamp</code> before the narrowing cast. <code>dequantize_i8</code> multiplies back. The <a href=\"/learn/quantization-math\">quantization math chapter</a> of the AI infrastructure handbook explains why this works and when it does not; here the point is the C++.</p>\n<p>For the six weights in the program, {0.12, -0.50, 0.33, 0.01, -0.27, 0.49}, the largest magnitude is 0.5, so the scale is 0.5 / 127, about 0.003937. Dividing by it gives 30.48, -127, 83.82, 2.54, -68.58 and 124.46, which round to the int8 values 30, -127, 84, 3, -69 and 124. Restoring 30 gives 30 × 0.003937, about 0.1181, an error of about 0.0019 against the original 0.12. No error can exceed half a step, 0.003937 / 2, about 0.00197, as long as nothing was clamped. Notice that the function returns two things, the values and the scale, as a <code>std::pair</code>, and that <code>main</code> unpacks them with a structured binding.</p>\n<h4 id=\"part-4-two-weights-in-one-byte\">Part 4: two weights in one byte</h4>\n<p>4-bit models store two weights per byte. <code>pack_i4(lo, hi)</code> keeps the low four bits of each value with <code>&#x26; 0x0F</code>, shifts <code>hi</code> left by four, and combines them with <code>|</code>. Because signed integers are two's complement, -1 has the low four bits <code>1111</code>, so <code>pack_i4(-1, 1)</code> is the byte <code>0x1F</code>. <code>unpack_i4</code> does the reverse: <code>byte &#x26; 0x0F</code> and <code>byte >> 4</code> recover the two 4-bit patterns, and <code>sign_extend4</code> maps the patterns 8 to 15 back to -8 to -1. The program checks the round trip for all 256 pairs of values and should report 0 failures.</p>\n<p>The file also has four <code>static_assert</code> lines. Each is a test that runs while compiling, because the functions it calls are <code>constexpr</code>. Change <code>0x1F</code> to <code>0x1E</code> in one of them and the program stops building, with the failing assertion as the error.</p>\n<p>Real formats make different choices for the same idea. llama.cpp's Q4_0 format, defined in <code>ggml/src/ggml-common.h</code>, groups weights into blocks of 32. Each block stores one scale as a 16-bit float and 16 bytes of 4-bit values, 18 bytes in all, or 4.5 bits per weight. Instead of two's complement, each stored 4-bit value is unsigned, from 0 to 15, and the dequantizer subtracts 8, which also gives the range -8 to 7. Byte <code>j</code> of a block holds weight <code>j</code> in its low half and weight <code>j + 16</code> in its high half, rather than two neighbouring weights.</p>\n<h4 id=\"stretch-goals\">Stretch goals</h4>\n<ul>\n<li>Change <code>quantize_i8</code> to use one scale per block of 32 weights, the way Q4_0 does, and compare the largest error with the single-scale version on a vector that has one large outlier.</li>\n<li>Write <code>quantize_i4</code> and <code>dequantize_i4</code> on top of <code>pack_i4</code> and <code>unpack_i4</code>, with a scale of <code>max_abs / 7</code>, and print how the error compares with int8.</li>\n</ul>","headings":[{"id":"lab-a-dtype-table-int8-quantization-and-int4-packing","title":"Lab: a dtype table, int8 quantization and int4 packing","depth":3},{"id":"part-1-run-the-traps","title":"Part 1: run the traps","depth":4},{"id":"part-2-the-dtype-table","title":"Part 2: the dtype table","depth":4},{"id":"part-3-symmetric-int8-quantization","title":"Part 3: symmetric int8 quantization","depth":4},{"id":"part-4-two-weights-in-one-byte","title":"Part 4: two weights in one byte","depth":4},{"id":"stretch-goals","title":"Stretch goals","depth":4}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>A function computes double avg = total / count; where total and count are int. For total = 7 and count = 2, avg is 3. Why, and what is the fix?</summary>\n<p>Both operands of <code>/</code> are <code>int</code>, so the division is integer division and produces 3. Only then is 3 converted to <code>double</code> for the assignment. Converting one operand before dividing makes it floating-point division: <code>static_cast&#x3C;double>(total) / count</code> gives 3.5.</p>\n</details>\n<details>\n<summary>What is wrong with for (std::size_t i = v.size() - 1; i >= 0; --i), and how do you write it safely?</summary>\n<p><code>std::size_t</code> is unsigned, so <code>i >= 0</code> is always true, and when <code>i</code> is 0, <code>--i</code> wraps around to the largest <code>size_t</code>. The loop never ends and indexes far outside the vector. For an empty vector, <code>v.size() - 1</code> is already the largest <code>size_t</code>, so it goes wrong from the first iteration. Use a signed index, <code>for (std::ptrdiff_t i = std::ssize(v) - 1; i >= 0; --i)</code>, or test before decrementing, <code>for (std::size_t i = v.size(); i-- > 0;)</code>.</p>\n</details>\n<details>\n<summary>Why should a dtype be an enum class rather than a plain enum or an int?</summary>\n<p>An <code>enum class</code> keeps its names inside its own scope, so <code>DType::I8</code> cannot collide with another <code>I8</code>, and it never converts to an integer implicitly, so comparing a dtype with a bare number or passing one where an <code>int</code> is expected does not compile. A <code>switch</code> over it without a <code>default</code> also gets a warning from <code>-Wall</code> for each value it does not handle, so adding a new dtype points you to every place that must learn about it.</p>\n</details>\n<details>\n<summary>A function float mean(std::vector&#x3C;float> xs) is called once per request on a vector of a million floats. It gives the right answer. What is the problem?</summary>\n<p>The parameter is passed by value, so every call copies the whole vector, allocating and copying 4 MB before computing anything. Declaring it <code>const std::vector&#x3C;float>&#x26; xs</code> passes a reference to the caller's vector instead, with no copy, and <code>const</code> guarantees <code>mean</code> cannot modify it.</p>\n</details>\n<details>\n<summary>You print a row of int8 weights with std::cout and get a line of odd symbols. Why, and how do you print the numbers?</summary>\n<p><code>std::int8_t</code> is a name for <code>signed char</code> on every common platform, and streams print character types as characters, so each weight is printed as the character with that code. Converting each value to <code>int</code> first, with <code>+q</code> or <code>static_cast&#x3C;int>(q)</code>, prints the number.</p>\n</details>\n<details>\n<summary>How many bytes does one block of 32 weights take in llama.cpp's Q4_0 format, and how many bits per weight is that?</summary>\n<p>A block holds one 16-bit scale, 2 bytes, and 32 four-bit values packed two per byte, 16 bytes, for 18 bytes in total. 18 bytes is 144 bits for 32 weights, which is 4.5 bits per weight: the 4 bits of each value plus the scale's 16 bits shared among 32 weights.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/operator_arithmetic\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: arithmetic operators</a> gives the exact rules for integer division, remainder, promotion and overflow.</li>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/implicit_conversion\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: implicit conversions</a> covers integral promotion and the conversions behind the signed and unsigned traps.</li>\n<li><a href=\"https://en.cppreference.com/w/cpp/types/integer\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: fixed width integer types</a> lists everything in <code>&#x3C;cstdint></code>, with the limits of each type.</li>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/enum\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: enumeration declaration</a> explains scoped and unscoped enums and underlying types.</li>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/structured_binding\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: structured binding declaration</a> shows every kind of object that can be unpacked.</li>\n<li><a href=\"https://www.learncpp.com/\" target=\"_blank\" rel=\"noopener noreferrer\">learncpp.com</a> is a free, thorough tutorial that covers this chapter's material at a slower pace, with exercises.</li>\n<li><a href=\"https://www.stroustrup.com/tour3.html\" target=\"_blank\" rel=\"noopener noreferrer\">A Tour of C++, third edition</a>, by Bjarne Stroustrup, the language's creator, is a short book that surveys modern C++ for programmers who already know another language.</li>\n<li><a href=\"https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-common.h\" target=\"_blank\" rel=\"noopener noreferrer\">llama.cpp's ggml-common.h</a> defines <code>block_q4_0</code> and the other quantized block formats, as real-world examples of fixed-width types and packed data.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}