{"id":"cpp-pointers","title":"Pointers, references and const","sections":[{"id":"overview","title":"Overview","html":"<p>Open any CUDA kernel, any BLAS routine or any PyTorch custom operator, and the first line looks something like this:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">void</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> rms_norm</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> x</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> weight</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> out</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> n</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> eps</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span></code></pre>\n<p>There is no tensor type in that signature, no shape object and no Python list. There are addresses: <code>x</code> says where the input starts, <code>weight</code> where the weights start, and <code>out</code> where the result should go. The <code>const</code> on the first two promises the function only reads them, and the missing <code>const</code> on <code>out</code> says it writes there. The <code>n</code> says how many floats to process, because an address alone does not say where the data ends. That is the language every low-level API speaks, and this chapter teaches you to read and write it.</p>\n<p>Python hides addresses completely. A Python name refers to an object somewhere in memory, and you never see where. C++ lets you hold the address itself, do arithmetic on it and pass it around, which is exactly what you need to describe \"the 4,096 floats starting here\" without copying them. It is also how a single block of memory can be read as a matrix, its transpose, a single row or a single column, all without moving a byte. By the end of this chapter a PyTorch tensor will look like what it is: a pointer, a shape, strides and an offset.</p>\n<p>This chapter builds on <a href=\"/learn/cpp-memory\">the memory chapter</a>, which explains where objects live and how long they live. Pointers are only safe when you know both.</p>","headings":[]},{"id":"addresses-and-pointers","title":"Addresses and pointers","html":"<h3 id=\"addresses-and-pointers\">Addresses and pointers</h3>\n<p>Every object in a running program occupies some bytes of memory, and the address of its first byte identifies it. The <code>&#x26;</code> operator, placed before a variable, gives that address, and a <strong>pointer</strong> is a variable that stores one:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 42</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> &#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">x;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    // p holds the address of x; its type is \"pointer to int\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 7</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">         // * follows the address: this writes 7 into x</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> y </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">p;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     // and this reads x through p, so y is 7</span></span></code></pre>\n<figure class=\"diagram\" data-text=\"p -> x = 7 (holds the address of x)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 480 78\" width=\"480\" role=\"img\" aria-label=\"A pointer holds the address of another object\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">p</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"46.5\">x = 7</tspan></text><rect class=\"dg-edge-bg\" x=\"165.8\" y=\"33\" width=\"146.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"239\" y=\"46\" text-anchor=\"middle\">holds the address of x</text></svg></div><figcaption>A pointer holds the address of another object</figcaption></figure>\n<p>The same two symbols appear in two roles, which is the main source of confusion at first. In a declaration, <code>int* p</code> means \"p is a pointer to int\". In an expression, <code>*p</code> means \"the object p points at\", which is called <strong>dereferencing</strong>, and <code>&#x26;x</code> means \"the address of x\". The <code>*</code> in a declaration binds to the name, not the type, so <code>int* a, b;</code> declares one pointer <code>a</code> and one plain <code>int</code> <code>b</code>. Declare one pointer per line and the trap never matters.</p>\n<p>A few facts about pointers are worth knowing from the start:</p>\n<ul>\n<li>On a 64-bit system every object pointer is 8 bytes, whether it points at a <code>char</code> or a 10-gigabyte buffer. The type of the pointer, <code>float*</code> or <code>int*</code>, tells the compiler what is at the address and how big each element is. It does not change the pointer's size.</li>\n<li>Printing a pointer prints its address in hexadecimal. The value changes from one run to the next, because operating systems place the stack and the heap at random addresses each time a program starts, a defence called address space layout randomization.</li>\n<li>To reach a member through a pointer to a struct, write <code>p->rows</code>, which is shorthand for <code>(*p).rows</code>.</li>\n</ul>\n<h4 id=\"nullptr\">nullptr</h4>\n<p>A pointer that points at nothing holds the <strong>null pointer</strong>, written <code>nullptr</code>. It is how C++ says \"no object here\": a function that searches for something can return <code>nullptr</code> when it finds nothing, the way a Python function returns <code>None</code>.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> Tensor</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> t </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> find_tensor</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(name);</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (t </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> nullptr</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) {</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // not found: handle it</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>Dereferencing a null pointer is undefined behavior. On Linux and macOS it usually crashes the program at once with a segmentation fault, because the operating system leaves the lowest addresses unmapped on purpose, but the language promises nothing, and Chapter 5 shows how an optimizer can make it worse. The rule is simple: if a pointer can be null, test it before you dereference it.</p>\n<p><code>nullptr</code> arrived in C++11. Older code writes <code>NULL</code> or <code>0</code>, which are integers rather than pointers. That matters with overloading: given <code>f(int)</code> and <code>f(char*)</code>, the call <code>f(NULL)</code> picks the <code>int</code> version or fails as ambiguous, depending on how the library defines <code>NULL</code>, while <code>f(nullptr)</code> always picks the pointer version. Use <code>nullptr</code> in new code.</p>","headings":[{"id":"addresses-and-pointers","title":"Addresses and pointers","depth":3},{"id":"nullptr","title":"nullptr","depth":4}]},{"id":"references","title":"References","html":"<h3 id=\"references\">References</h3>\n<p>A <strong>reference</strong> is another name for an existing object. You declare one with <code>&#x26;</code> after the type:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 42</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">int&#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> r </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     // r is now another name for x</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">r </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 7</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">          // this changes x</span></span></code></pre>\n<p>That <code>&#x26;</code> is a third meaning of the symbol, in a declaration this time, and it is unrelated to taking an address. Read <code>int&#x26; r</code> as \"r is a reference to int\".</p>\n<p>A reference behaves like a pointer that the language manages for you, with three rules that make it safer:</p>\n<ol>\n<li>It must be bound to an object when it is created. There is no such thing as an uninitialised reference.</li>\n<li>It cannot be null. A valid program never has a reference that refers to nothing.</li>\n<li>It cannot be re-pointed. Assigning to a reference assigns to the object it refers to.</li>\n</ol>\n<p>The third rule surprises Python programmers, because a Python name <em>can</em> be re-pointed. In Python, <code>b = a</code> makes <code>b</code> refer to the same list as <code>a</code>, and a later <code>b = other</code> makes <code>b</code> refer to something else while <code>a</code> is untouched. In C++, after <code>int&#x26; r = x;</code>, the statement <code>r = y;</code> copies the <em>value</em> of <code>y</code> into <code>x</code>. The lab checks <code>&#x26;r == &#x26;x</code> afterwards to show that <code>r</code> still refers to <code>x</code>. A Python name is closer to a C++ pointer that can be re-pointed, and a C++ reference is closer to a permanent alias.</p>\n<p>References exist mainly for function parameters, which the next section covers, and for range-based <code>for</code> loops such as <code>for (float&#x26; x : v) x *= 0.5f;</code>, which the previous chapter used. They carry one danger: a reference does not keep its object alive. A function that returns a reference to one of its own local variables returns a name for an object that is destroyed as the function returns, which is the dangling reference that <a href=\"/learn/cpp-memory\">the memory chapter</a> demonstrates. The same is true of a pointer to a local.</p>","headings":[{"id":"references","title":"References","depth":3}]},{"id":"four-ways-to-pass-an-argument","title":"Four ways to pass an argument","html":"<h3 id=\"four-ways-to-pass-an-argument\">Four ways to pass an argument</h3>\n<p>Chapter 2 showed that a parameter declared as <code>std::vector&#x3C;float> xs</code> receives a copy. Pointers and references give three more choices, and together they let a function's signature say exactly what it does with each argument:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Parameter</th><th>What the function gets</th><th>Use it when</th></tr></thead><tbody><tr><td><code>T x</code></td><td>Its own copy</td><td><code>T</code> is small and cheap to copy, such as <code>int</code>, <code>float</code>, a pointer or an enum, or the function needs its own copy anyway</td></tr><tr><td><code>const T&#x26; x</code></td><td>Read-only access to the caller's object, with no copy</td><td>The function only reads a large object, such as a vector, a string or a tensor</td></tr><tr><td><code>T&#x26; x</code></td><td>Read and write access to the caller's object</td><td>The function must modify the caller's object</td></tr><tr><td><code>T* x</code> or <code>const T* x</code></td><td>An address, which may be <code>nullptr</code></td><td>\"No object\" is a valid argument, or the argument is the start of an array of <code>T</code></td></tr></tbody></table></div>\n<p>The C++ Core Guidelines, the style guide maintained by Bjarne Stroustrup and Herb Sutter, state the same rules: pass cheap types by value and others by <code>const&#x26;</code> for input (rule F.16), use a non-const reference for a parameter the function modifies (F.17), and prefer a pointer over a reference when \"no argument\" is a valid option (F.60).</p>\n<p>The last row is the one that low-level code uses most, and for a different reason. A function that processes an array receives a pointer to its first element and a count, because a pointer alone does not know how many elements follow it:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> data</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">size_t</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> n</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">size_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n; </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">++</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i) total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> data[i];</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> total;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::vector</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;float></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(v.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">data</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(), v.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">size</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">());</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // data() returns a pointer to the vector's first element</span></span></code></pre>\n<p><code>std::vector</code> guarantees that its elements sit next to each other in memory, so <code>v.data()</code> and <code>v.size()</code> are all a C-style function needs. This is how a C++ program hands a vector to a BLAS library or copies it to a GPU.</p>\n<h4 id=\"stdspan-the-pointer-and-the-length-together\">std::span: the pointer and the length together</h4>\n<p>Passing a pointer and a length separately invites mistakes: the wrong length, or a length that belongs to a different buffer. C++20 added <code>std::span</code>, in the <code>&#x3C;span></code> header, which holds both in one small object that you pass by value:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">span</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">> </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">xs</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x : xs) total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> total;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(v);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                                     // a vector converts to a span automatically</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">span</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">>(v).</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">first</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">));</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    // a span over the first two elements, no copy</span></span></code></pre>\n<p>A span does not own its elements. It is a <strong>view</strong>, a pointer and a size describing memory that something else owns, so it is cheap to create and pass, and it dangles if the owner goes away. <code>std::span&#x3C;const float></code> is read-only, and <code>std::span&#x3C;float></code> can write. Its <code>operator[]</code> does not check bounds in C++20, so it is a convenience, not a safety net. New C++ code that does not have to match a C interface should prefer spans to separate pointers and lengths.</p>","headings":[{"id":"four-ways-to-pass-an-argument","title":"Four ways to pass an argument","depth":3},{"id":"stdspan-the-pointer-and-the-length-together","title":"std::span: the pointer and the length together","depth":4}]},{"id":"const-in-every-position","title":"const in every position","html":"<h3 id=\"const-in-every-position\">const in every position</h3>\n<p><code>const</code> can appear in several places in a declaration, and each place means something different. The rule for reading them is to read the declaration from right to left, starting at the name:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> a;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">         // a is a pointer to a float that is const</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> const</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> &#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">v;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    // b is a const pointer to a float</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> const</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> c </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> &#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">v;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // c is a const pointer to a const float</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float&#x26;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     // d is a reference to a float that is const</span></span></code></pre>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Declaration</th><th>Can you change the float through it?</th><th>Can you point it somewhere else?</th></tr></thead><tbody><tr><td><code>const float* a</code></td><td>No</td><td>Yes</td></tr><tr><td><code>float* const b</code></td><td>Yes</td><td>No</td></tr><tr><td><code>const float* const c</code></td><td>No</td><td>No</td></tr><tr><td><code>const float&#x26; d</code></td><td>No</td><td>References can never be re-pointed</td></tr></tbody></table></div>\n<p>The first row is the one you will write constantly. <code>const float* x</code> is a promise that the function will not modify the floats at <code>x</code>, and the compiler enforces it: writing <code>x[0] = 1.0f;</code> in that function is a compile error. You will also see <code>float const* x</code>, which means exactly the same thing, because <code>const</code> applies to whatever is on its left, or to the type on its right when nothing is on its left.</p>\n<p>The second row, a <code>const</code> pointer, is less common. It fixes where the pointer points, and leaves the data writable.</p>\n<h4 id=\"why-const-correctness-is-worth-the-typing\">Why const-correctness is worth the typing</h4>\n<p>Look at the real signature of cuBLAS's single-precision matrix multiply, one of the NVIDIA library routines that PyTorch calls for float matmuls on a GPU:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">cublasStatus_t</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> cublasSgemm</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">cublasHandle_t</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> handle</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">                           cublasOperation_t</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> transa</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">cublasOperation_t</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> transb</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                           int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> m</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> n</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> k</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                           const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> alpha</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                           const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> A</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> lda</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                           const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> B</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> ldb</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                           const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> beta</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                           float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> C</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> ldc</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span></code></pre>\n<p>Without reading a line of documentation you know that <code>A</code> and <code>B</code> are inputs and <code>C</code> is the output, because only <code>C</code> lacks <code>const</code>. That is const-correctness: when every function marks what it only reads, a signature tells you the data flow, and the compiler stops anyone from breaking the promise by accident. It also spreads: a function that receives a <code>const float*</code> cannot pass it on to a function that takes a plain <code>float*</code>, so one missing <code>const</code> deep in a library forces awkward workarounds on everyone who calls it. Mark every pointer and reference parameter <code>const</code> unless the function writes through it.</p>\n<p>Two details complete the picture. A member function can be marked <code>const</code> after its parameter list, as in <code>std::size_t size() const</code>, which promises that calling it does not modify the object; Chapter 6 covers this with classes. And <code>const</code> restricts one path to the data, not the data itself: if another non-const pointer refers to the same floats, they can still change through that one. <code>const_cast</code> can remove <code>const</code>, but modifying an object that was <em>defined</em> <code>const</code> is undefined behavior, so treat <code>const_cast</code> as a sign that something is wrong.</p>","headings":[{"id":"const-in-every-position","title":"const in every position","depth":3},{"id":"why-const-correctness-is-worth-the-typing","title":"Why const-correctness is worth the typing","depth":4}]},{"id":"raw-arrays-and-how-they-decay","title":"Raw arrays and how they decay","html":"<h3 id=\"raw-arrays-and-how-they-decay\">Raw arrays and how they decay</h3>\n<p>C++ inherited fixed-size arrays from C:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> a[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // four floats, side by side</span></span></code></pre>\n<p>The array's size is part of its type, <code>float[4]</code>, and <code>sizeof(a)</code> is 16 bytes, four floats of 4 bytes each. But in almost every expression, the name <code>a</code> turns into a pointer to its first element. This is called <strong>array-to-pointer decay</strong>, and it is why <code>float* p = a;</code> compiles.</p>\n<p>Decay has one nasty consequence. A function parameter written as an array is really a pointer:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">size_t</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> size_inside</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> a</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[]) {</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // the compiler reads this as const float* a</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  return</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> sizeof</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(a);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                          // 8 on a 64-bit system: the size of a pointer</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>Inside the function, the array's length is gone. The same mistake in C has caused countless bugs, and compilers now warn about <code>sizeof</code> on an array parameter. The lab triggers that warning on purpose.</p>\n<p>The modern replacements avoid decay. <code>std::array&#x3C;float, 4></code> is a fixed-size array that behaves like a normal object: it does not decay, it can be copied and returned, and <code>.size()</code> works. <code>std::vector&#x3C;float></code> is the growable version. When a function must accept any of them, take a <code>std::span&#x3C;const float></code>. Raw arrays still appear inside low-level code and especially in CUDA kernels, where fixed-size local arrays such as <code>float acc[4];</code> are common, so you need to recognise them.</p>","headings":[{"id":"raw-arrays-and-how-they-decay","title":"Raw arrays and how they decay","depth":3}]},{"id":"pointer-arithmetic","title":"Pointer arithmetic","html":"<h3 id=\"pointer-arithmetic\">Pointer arithmetic</h3>\n<p>Adding an integer to a pointer moves it by that many <em>elements</em>, not bytes. If <code>p</code> is a <code>float*</code>, then <code>p + 2</code> points two floats further on, which is 8 bytes further, and if <code>p</code> were a <code>double*</code>, <code>p + 2</code> would be 16 bytes further. The compiler multiplies by the element size for you, which is why a pointer has a type at all.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> a[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> first </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> a;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> third </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> first </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">       // points at a[2]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">third;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">               // 3.0f</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">ptrdiff_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> third </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> first;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // 2: subtracting pointers counts elements</span></span></code></pre>\n<p>Indexing is pointer arithmetic in disguise. The standard defines <code>p[i]</code> as <code>*(p + i)</code>, so <code>a[2]</code>, <code>*(a + 2)</code> and <code>first[2]</code> are the same expression. Once you see this, CUDA code stops looking mysterious: <code>x[blockIdx.x * n + threadIdx.x]</code> is simply \"start at <code>x</code>, move this many floats, read\".</p>\n<p>Pointer arithmetic is only valid inside one array. You may move a pointer to any element of the array or to the position one past the last element, which is what <code>end()</code> iterators represent, but not further, and you must never dereference the one-past-the-end position. Computing an address outside the array is undefined behavior even if you never read it. Chapter 5 shows what out-of-bounds access does in practice and how AddressSanitizer catches it.</p>\n<p>Remember too the way a pointer goes bad that <a href=\"/learn/cpp-memory\">the memory chapter</a> showed: when a <code>std::vector</code> grows beyond its capacity, it allocates a bigger block, moves its elements there and frees the old block, so any pointer you took with <code>data()</code> or <code>&#x26;v[0]</code> before that points at freed memory. Take pointers into a vector after it has finished growing, or reserve the space first.</p>","headings":[{"id":"pointer-arithmetic","title":"Pointer arithmetic","depth":3}]},{"id":"tensors-are-pointers-with-shapes-and-strides","title":"Tensors are pointers with shapes and strides","html":"<h3 id=\"tensors-are-pointers-with-shapes-and-strides\">Tensors are pointers with shapes and strides</h3>\n<p>Memory is one-dimensional: a long row of bytes with consecutive addresses. A matrix is two-dimensional. Every tensor library has to map the second onto the first, and almost all of them use <strong>row-major</strong> order, also called C order: the first row, then the second row, and so on. A 2 × 3 matrix holding the numbers 0 to 5 is stored as six consecutive floats:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th></th><th>column 0</th><th>column 1</th><th>column 2</th></tr></thead><tbody><tr><td><strong>row 0</strong></td><td>0 (index 0)</td><td>1 (index 1)</td><td>2 (index 2)</td></tr><tr><td><strong>row 1</strong></td><td>3 (index 3)</td><td>4 (index 4)</td><td>5 (index 5)</td></tr></tbody></table></div>\n<p>So element <code>(i, j)</code> of a matrix with <code>cols</code> columns lives at flat index <code>i * cols + j</code>. In code:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">> </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">A</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">rows</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> cols</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">A[i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cols </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">          // element (i, j)</span></span></code></pre>\n<p>You will write <code>i * cols + j</code> thousands of times in this course. It is worth seeing it in a more general form. Moving one column to the right moves 1 element in memory, and moving one row down moves <code>cols</code> elements. Those two numbers are the matrix's <strong>strides</strong>, and with them the index formula becomes:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>flat index = offset + i * stride0 + j * stride1</span></span></code></pre>\n<p>For a plain row-major matrix, <code>offset</code> is 0, <code>stride0</code> is <code>cols</code> and <code>stride1</code> is 1. PyTorch stores exactly this for every tensor: a pointer to its storage, the shape, the strides and the storage offset. You can see it from Python, where <code>t = torch.arange(6).reshape(2, 3)</code> gives <code>t.stride() == (3, 1)</code>.</p>\n<p>The power of strides is that many operations only change those numbers and never touch the data:</p>\n<ul>\n<li><strong>Transpose.</strong> <code>t.T</code> has shape <code>(3, 2)</code> and strides <code>(1, 3)</code>: the shape and the strides are swapped, and the storage is the same six floats. Transposing a 10,000 × 10,000 matrix this way costs nothing.</li>\n<li><strong>Selecting a column.</strong> <code>t[:, 1]</code> has shape <code>(2,)</code>, stride <code>(3,)</code> and storage offset 1: start at the second float and jump a whole row each time.</li>\n<li><strong>Selecting rows.</strong> <code>t[1:]</code> has shape <code>(1, 3)</code> and storage offset 3. It is the second row, still row-major.</li>\n<li><strong>Step slicing.</strong> <code>t[:, ::2]</code> keeps the strides of the rows and doubles the column stride to 2, so it reads every other column.</li>\n</ul>\n<figure class=\"diagram\" data-text=\"t: shape (2, 3) stride (3, 1) -> storage: 6 floats; t.T: shape (3, 2) stride (1, 3) -> storage: 6 floats\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 480 200\" width=\"480\" role=\"img\" aria-label=\"Two views share one storage\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"55.3\" x2=\"327\" y2=\"88.3\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"148.7\" x2=\"327\" y2=\"115.7\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">t: shape (2, 3)</tspan><tspan x=\"80\" y=\"54.5\">stride (3, 1)</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"158.5\">t.T: shape (3, 2)</tspan><tspan x=\"80\" y=\"174.5\">stride (1, 3)</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"78\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"106.5\">storage: 6 floats</tspan></text></svg></div><figcaption>Two views share one storage</figcaption></figure>\n<p>A tensor is <strong>contiguous</strong> when its elements, read in row-major order of its own shape, sit back to back in memory, which means every dimension's stride equals the product of the sizes after it (PyTorch ignores dimensions of size 1, whose stride never matters). <code>t</code> is contiguous. <code>t.T</code> is not, because walking along one of its rows jumps 3 elements at a time. PyTorch reports this with <code>t.is_contiguous()</code>, and <code>t.contiguous()</code> returns the same tensor if it is already contiguous, or otherwise allocates new storage and copies the elements into row-major order. Two numbering conventions differ between libraries, so check which one you are reading: PyTorch counts strides in elements, NumPy counts them in bytes, and llama.cpp's ggml keeps byte strides in an array called <code>nb</code> and lists dimensions from the innermost outwards.</p>\n<p>Why should a C++ programmer care? Because code that receives a raw <code>const float*</code> assumes a layout, almost always contiguous row-major. Pass it the data pointer of a transposed view and it silently reads the wrong elements. That is why custom PyTorch operators so often begin with a check such as <code>TORCH_CHECK(x.is_contiguous())</code>, or call <code>.contiguous()</code> before taking the pointer, and why <code>t.T.view(-1)</code> raises an error that suggests <code>.reshape</code> instead: a non-contiguous tensor cannot be flattened without a copy. Try it here:</p>\n<div class=\"widget-slot\" data-widget=\"tensor-strides\"></div>\n<h4 id=\"row-major-and-column-major\">Row-major and column-major</h4>\n<p>Not every library uses row-major order. Fortran stored matrices column by column, <strong>column-major</strong>, and the original BLAS interface followed it, so cuBLAS expects column-major matrices too. That is what the <code>lda</code>, <code>ldb</code> and <code>ldc</code> arguments in the <code>cublasSgemm</code> signature above are about: each is a <strong>leading dimension</strong>, the stride between consecutive columns. Code that holds row-major data usually does not transpose anything to call cuBLAS. It uses a piece of algebra instead: a row-major matrix read as column-major is its transpose, and (A × B)ᵀ = Bᵀ × Aᵀ, so swapping the two inputs (and their dimensions) makes cuBLAS compute the row-major product directly. Strides are the idea that makes that trick, and many others like it, possible.</p>","headings":[{"id":"tensors-are-pointers-with-shapes-and-strides","title":"Tensors are pointers with shapes and strides","depth":3},{"id":"row-major-and-column-major","title":"Row-major and column-major","depth":4}]},{"id":"a-naive-matmul-with-pointers","title":"A naive matmul with pointers","html":"<h3 id=\"a-naive-matmul-with-pointers\">A naive matmul with pointers</h3>\n<p>Everything in this chapter comes together in the most important function in AI: matrix multiplication. Here is the textbook version for row-major matrices, C = A × B, where A is M × K, B is K × N and C is M × N:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">void</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> matmul</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> A</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> B</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> C</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> M</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> N</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> K</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> M; </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">++</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> N; </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">++</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">j) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">      float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sum </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">      for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> K; </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">++</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">k) sum </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> A[i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> K </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> k] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> B[k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> N </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j];</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">      C[i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> N </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sum;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    }</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">  }</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>Read the signature first. <code>A</code> and <code>B</code> are <code>const float*</code>, inputs; <code>C</code> is a <code>float*</code>, the output; and the three sizes say how to interpret the flat buffers. Each element of C is the dot product of one row of A with one column of B. The row of A is contiguous, <code>A[i * K + k]</code> for consecutive <code>k</code>. The column of B is not: <code>B[k * N + j]</code> jumps N floats at every step. For N = 512, that is 2,048 bytes per step, so every multiply touches a different cache line. Chapter 13 explains why that pattern is slow and Project C fixes it step by step, from this version to one many times faster. For now, the point is that the order in which you walk memory is a property of the indexing code you write, and strides are how you control it.</p>\n<p>The function does 2 × M × N × K floating-point operations, one multiply and one add for each combination of i, j and k. For three 512 × 512 matrices that is 2 × 512³ = 268,435,456 operations, which is the number you divide by the run time to get FLOP/s.</p>\n<p>Two more lessons hide in those five lines.</p>\n<p><strong>Indexes overflow.</strong> <code>i * K + k</code> is computed as an <code>int</code>, and the largest <code>int</code> is 2,147,483,647. An 8 GiB buffer of 16-bit values holds 4,294,967,296 elements, twice that, and KV caches and embedding tables of that size are routine in serving. Indexing them with <code>int</code> overflows, which is undefined behavior. PyTorch stores sizes and strides as 64-bit integers for this reason, and its CUDA kernels check whether a tensor is small enough for faster 32-bit index arithmetic (the helper is called <code>canUse32BitIndexMath</code>) before using it. Use <code>std::int64_t</code> or <code>std::size_t</code> for indexes into large buffers.</p>\n<p><strong>Pointers can alias.</strong> Nothing in the signature stops a caller from passing the same buffer as <code>A</code> and <code>C</code>. The compiler must allow for that possibility, so after every write to <code>C</code> it has to assume <code>A</code> and <code>B</code> might have changed, which limits how much it can keep in registers and reorder. GCC, Clang and MSVC accept a non-standard <code>__restrict</code> qualifier that promises a pointer does not alias any other, and CUDA code spells it <code>__restrict__</code>. Chapter 15 comes back to it when vectorizing loops.</p>\n<h4 id=\"the-same-matmul-through-views\">The same matmul through views</h4>\n<p>Combine the matmul with strides and you get a function that multiplies any two views, transposed or not, without copying. The lab defines a small struct with the same information as a PyTorch tensor's metadata:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">struct</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> View</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> data;</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int64_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> rows, cols;</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int64_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> stride0, stride1;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // in elements</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> get</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> View</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> v</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int64_t</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> i</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int64_t</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> j</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) { </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v.data[i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v.stride0 </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> v.stride1]; }</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">View</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> transposed</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> View</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> v</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) { </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {v.data, v.cols, v.rows, v.stride1, v.stride0}; }</span></span></code></pre>\n<p><code>transposed</code> swaps two pairs of numbers and returns. No element moves. A matmul written with <code>get(A, i, k) * get(B, k, j)</code> then works on any layout, which is how the lab computes A × Aᵀ without ever building Aᵀ. Real libraries go further: C++23 added <code>std::mdspan</code>, a standard multi-dimensional view with pluggable layouts for row-major, column-major and arbitrary strides, although only recent standard libraries ship it.</p>","headings":[{"id":"a-naive-matmul-with-pointers","title":"A naive matmul with pointers","depth":3},{"id":"the-same-matmul-through-views","title":"The same matmul through views","depth":4}]},{"id":"lab-pointers-views-and-a-first-matmul","title":"Lab: pointers, views and a first matmul","html":"<h3 id=\"lab-pointers-views-and-a-first-matmul\">Lab: pointers, views and a first matmul</h3>\n<p>The lab has two programs, <a href=\"/labs/cpp/cpp-pointers/pointers.cpp\"><code>pointers.cpp</code></a> and <a href=\"/labs/cpp/cpp-pointers/matmul.cpp\"><code>matmul.cpp</code></a>, and a <a href=\"/labs/cpp/cpp-pointers/CMakeLists.txt\"><code>CMakeLists.txt</code></a> that builds both. Download the three files into one folder. It needs only a C++20 compiler and CMake.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -S</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> .</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -B</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -DCMAKE_BUILD_TYPE=Release</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --build</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build/pointers</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build/matmul</span></span></code></pre>\n<h4 id=\"part-1-pointerscpp\">Part 1: pointers.cpp</h4>\n<p><code>pointers.cpp</code> runs eleven numbered experiments that follow this chapter: taking and following an address, pointer sizes, pointer arithmetic in elements and in bytes, array decay, <code>std::array</code>, references and <code>scale</code>, a pointer-and-length <code>sum</code>, the same with <code>std::span</code>, <code>const</code> in each position, <code>nullptr</code>, and a vector whose <code>data()</code> moves when it reallocates. Before running it, predict each line. Three things to check:</p>\n<ul>\n<li>Experiment 3 should report the two pointers 2 elements and 8 bytes apart.</li>\n<li>Experiment 4 should show 16 bytes for the array in <code>main</code> and 8 inside the function, and the build should have warned about <code>sizeof</code> on an array parameter.</li>\n<li>In experiment 9, remove the <code>//</code> from one of the lines marked \"error\" and rebuild, then read how your compiler describes the problem. Put the <code>//</code> back before trying the other.</li>\n</ul>\n<h4 id=\"part-2-matmulcpp\">Part 2: matmul.cpp</h4>\n<p><code>matmul.cpp</code> builds two n × n matrices with a fixed pattern of values (n is 512 unless you pass another size, as in <code>./build/matmul 256</code>) and runs three experiments:</p>\n<ol>\n<li><code>matmul</code> with raw pointers, timed, with the largest difference from the same product computed in double precision. The difference should be small, because float rounding errors grow with the length of each dot product but stay far below the values themselves.</li>\n<li>The same product through views, where B is given as a transposed view of a matrix <code>Bt</code> that holds B's transpose. The values are identical, but now the inner loop reads B's columns as contiguous rows of <code>Bt</code>. Compare its time with experiment 1, then try <code>./build/matmul 1024</code>; Chapter 13 explains whatever difference you see.</li>\n<li>A × Aᵀ through views, without building Aᵀ, checked for the symmetry that such a product must have.</li>\n</ol>\n<p>Then build in Debug as in Chapter 1 and run it again, to see how much the optimizer contributes to a loop like this.</p>\n<h4 id=\"stretch-goals\">Stretch goals</h4>\n<ul>\n<li>Write <code>View row(const View&#x26; v, std::int64_t i)</code> and <code>View column(const View&#x26; v, std::int64_t j)</code> that return one row or column as a 1 × n or n × 1 view, by setting the pointer and the shape and leaving the strides alone.</li>\n<li>Add an <code>is_contiguous(const View&#x26;)</code> function that follows PyTorch's rule, and check it on a view, its transpose and a column.</li>\n<li>Change the indexes in <code>matmul</code> from <code>int</code> to <code>std::int64_t</code> and compare the times. Then explain why the result does not overflow for n = 1024 even with <code>int</code>.</li>\n</ul>","headings":[{"id":"lab-pointers-views-and-a-first-matmul","title":"Lab: pointers, views and a first matmul","depth":3},{"id":"part-1-pointerscpp","title":"Part 1: pointers.cpp","depth":4},{"id":"part-2-matmulcpp","title":"Part 2: matmul.cpp","depth":4},{"id":"stretch-goals","title":"Stretch goals","depth":4}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>What is the difference between const float* p and float* const p?</summary>\n<p>Read each from right to left. <code>const float* p</code> is a pointer to a <code>const float</code>: you cannot change the floats through <code>p</code>, but <code>p</code> itself can be moved to point elsewhere. <code>float* const p</code> is a <code>const</code> pointer to a <code>float</code>: it must be initialised, it can never point anywhere else, but the float it points at can be changed through it. Function inputs are almost always the first kind.</p>\n</details>\n<details>\n<summary>Inside void f(float a[]), sizeof(a) is 8, although the caller passed an array of 100 floats. Why?</summary>\n<p>An array parameter is really a pointer parameter: the compiler reads <code>float a[]</code> as <code>float* a</code>, and the array decays to a pointer to its first element when it is passed. <code>sizeof(a)</code> is therefore the size of a pointer, 8 bytes on a 64-bit system, and the length is lost. Pass the length separately, or take a <code>std::span&#x3C;const float></code>, which carries both.</p>\n</details>\n<details>\n<summary>After int&#x26; r = x; the code runs r = y;. What does this change?</summary>\n<p>It copies the value of <code>y</code> into <code>x</code>. A reference cannot be re-pointed: <code>r</code> was bound to <code>x</code> when it was created and refers to <code>x</code> for the rest of its life, so assigning to <code>r</code> assigns to <code>x</code>. This is unlike Python, where <code>b = other</code> makes the name <code>b</code> refer to a different object.</p>\n</details>\n<details>\n<summary>A row-major tensor t has shape (4, 5). What are its strides, where is element (2, 1), and what are the shape and strides of t.T?</summary>\n<p>The strides are (5, 1): one row down skips 5 elements and one column right skips 1. Element (2, 1) is at 2 × 5 + 1 = 11. <code>t.T</code> has shape (5, 4) and strides (1, 5), and it shares <code>t</code>'s storage, so it is not contiguous. Element (1, 2) of <code>t.T</code> is at 1 × 1 + 2 × 5 = 11, the same float.</p>\n</details>\n<details>\n<summary>A custom operator takes a const float* from x.data_ptr() and reads x as a row-major matrix. It gives wrong answers only when the caller passes a transposed tensor. Why, and what are two fixes?</summary>\n<p>A transposed tensor shares its storage with the original and differs only in its strides, so its data pointer points at the same floats in the original order. Code that assumes contiguous row-major layout ignores the strides and reads the elements in the wrong order. Either call <code>.contiguous()</code> before taking the pointer, which copies the data into row-major order when needed, or check <code>x.is_contiguous()</code> and reject other layouts with an error. A third option is to pass the strides to the kernel and index with them.</p>\n</details>\n<details>\n<summary>Why can indexing a large tensor with int be a bug even when every individual index is small?</summary>\n<p>The flat index is a product and a sum, such as <code>i * cols + j</code>, and the result can exceed the largest <code>int</code>, 2,147,483,647, even when <code>i</code>, <code>cols</code> and <code>j</code> are all modest. Signed overflow is undefined behavior, so the program may read the wrong address. Buffers of more than 2^31 elements are common in inference, such as an 8 GiB buffer of 16-bit values with 2^32 elements, so index arithmetic on large buffers should use <code>std::int64_t</code> or <code>std::size_t</code>, as PyTorch does for sizes and strides.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/pointer\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: pointer declaration</a> and <a href=\"https://en.cppreference.com/w/cpp/language/reference\" target=\"_blank\" rel=\"noopener noreferrer\">reference declaration</a> give the exact rules for both, including null pointers and what references may bind to.</li>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/cv\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: cv qualifiers</a> explains <code>const</code> and <code>volatile</code> in every position.</li>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/array\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: array declaration</a> covers array-to-pointer decay, and <a href=\"https://en.cppreference.com/w/cpp/container/span\" target=\"_blank\" rel=\"noopener noreferrer\">std::span</a> documents the C++20 view.</li>\n<li><a href=\"https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#S-functions\" target=\"_blank\" rel=\"noopener noreferrer\">The C++ Core Guidelines on parameter passing</a>, in the section on functions, give the rules behind the table of four ways to pass an argument.</li>\n<li><a href=\"http://blog.ezyang.com/2019/05/pytorch-internals/\" target=\"_blank\" rel=\"noopener noreferrer\">Edward Yang's \"PyTorch internals\"</a> explains tensors as storage plus sizes, strides and an offset, with pictures, from one of PyTorch's core developers.</li>\n<li><a href=\"https://pytorch.org/docs/stable/generated/torch.Tensor.stride.html\" target=\"_blank\" rel=\"noopener noreferrer\">PyTorch's documentation of Tensor.stride</a> and <a href=\"https://pytorch.org/docs/stable/generated/torch.Tensor.contiguous.html\" target=\"_blank\" rel=\"noopener noreferrer\">Tensor.contiguous</a> show the same ideas from the Python side.</li>\n<li><a href=\"https://docs.nvidia.com/cuda/cublas/\" target=\"_blank\" rel=\"noopener noreferrer\">The cuBLAS documentation</a> describes its column-major convention and the leading-dimension arguments.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}