{"id":"ai-evals-and-observability","title":"Evaluating and observing AI systems","sections":[{"id":"overview","title":"Overview","html":"<p>A web service gets two checks on every request. The status code says whether the call worked, and tests say whether the code does what its author meant. A language model call breaks both. A request can return <code>200 OK</code> with an answer that quotes the wrong refund policy, and the same request sent twice can produce two different sentences.</p>\n<p><strong>Evaluation</strong> measures whether answers are good, using a repeatable procedure and a number you can track. <strong>Observability</strong> shows what each model call did, how long it took and what it cost. For logs, metrics, traces and SLOs in general, read <a href=\"/learn/observability\">Monitoring, logging and tracing</a>. This chapter covers what changes when the thing you monitor writes text.</p>\n<p>The running example is a support assistant for an online store. It reads the store's policy pages and answers customers in a chat window. Before each release, the team asks whether the new version answers more questions correctly. After each release, it asks whether the answers are still correct, fast and affordable.</p>","headings":[]},{"id":"why-ai-systems-need-evals","title":"Why AI systems need evals","html":"<h3 id=\"why-ai-systems-need-evals\">Why AI systems need evals</h3>\n<h4 id=\"same-question-different-wording\">Same question, different wording</h4>\n<p>A test that asserts the answer equals \"Refunds take 5 business days.\" fails on a correct answer such as \"Your refund should arrive within five business days.\" Exact equality is the wrong check for free text, so the check has to judge meaning.</p>\n<p>Outputs also vary between runs. A model generates text one token at a time, where a token is a short chunk of text. Settings such as temperature add randomness on purpose (<a href=\"/learn/llm-inference#sampling\">How LLM inference works</a>). Temperature 0 is meant to remove that randomness, and in practice some remains. Thinking Machines sent one prompt to a large Qwen model 1,000 times at temperature 0 and got 80 distinct completions. The most common one appeared 78 times. The cause sits in the serving software. GPU kernels, the programs that run each math operation, add numbers in an order that depends on the batch size (how many requests share one pass through the model). The batch size depends on other users' traffic (<a href=\"https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/\" target=\"_blank\" rel=\"noopener noreferrer\">Thinking Machines, 2025</a>). A single run is therefore one sample.</p>\n<h4 id=\"a-200-can-hide-a-bad-answer\">A 200 can hide a bad answer</h4>\n<p>For a model, HTTP 200 means only that the call completed. Anthropic's API returns a refusal, where the model declines the request, as a normal HTTP 200 with <code>stop_reason</code> set to <code>refusal</code>. A reply cut short by the token limit ends with <code>max_tokens</code> and no error (<a href=\"https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic, stop reasons</a>). You see either one only by reading the response body.</p>\n<p>The most damaging failures look like success: a fluent answer with the wrong number, a citation to a page that doesn't cover the topic, or a promise the policy doesn't allow. Only a check of the content catches them.</p>\n<h4 id=\"what-an-eval-is\">What an eval is</h4>\n<p>An <strong>eval</strong> runs a system on a fixed set of inputs, grades each output and reports a score. The inputs and their expectations form the <strong>eval set</strong>. A <strong>grader</strong> decides whether one output passes. It can be code, a second model or a person. OpenAI's guide describes evals as performance tests that keep outputs reliable despite nondeterminism (<a href=\"https://developers.openai.com/api/docs/guides/evaluation-best-practices\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI, evaluation best practices</a>).</p>\n<p>A score needs its context: the cases, the graders and the model version that produced it. The target needn't be 100% either. Hamel Husain writes that \"Your pass rate is a product decision\" (<a href=\"https://hamel.dev/blog/posts/evals/\" target=\"_blank\" rel=\"noopener noreferrer\">Husain, 2024</a>). The assistant may need every refund fact right, and it can accept misses on small talk.</p>","headings":[{"id":"why-ai-systems-need-evals","title":"Why AI systems need evals","depth":3},{"id":"same-question-different-wording","title":"Same question, different wording","depth":4},{"id":"a-200-can-hide-a-bad-answer","title":"A 200 can hide a bad answer","depth":4},{"id":"what-an-eval-is","title":"What an eval is","depth":4}]},{"id":"building-an-eval-set","title":"Building an eval set","html":"<h3 id=\"building-an-eval-set\">Building an eval set</h3>\n<h4 id=\"start-from-what-success-means\">Start from what success means</h4>\n<p>Before writing cases, write down what a good answer does for this product. For the support assistant, a good answer quotes the refund policy correctly, links the page it used, promises nothing the policy doesn't promise, and stays short. A grader can check each of those. \"Helpful\" can't be checked until you split it into properties like these.</p>\n<h4 id=\"where-cases-come-from\">Where cases come from</h4>\n<p>A <strong>golden set</strong> is a fixed collection of cases with trusted expectations, run on every change so the scores stay comparable. Build it from four sources.</p>\n<ul>\n<li><strong>Production traffic.</strong> Sample real questions from the logs, and remove personal data before storing them.</li>\n<li><strong>Production failures.</strong> Every bug report and escalated conversation becomes a case. Husain updates tests as new failures appear.</li>\n<li><strong>Expert hard cases.</strong> Ask a support lead for the questions that confuse customers.</li>\n<li><strong>Synthetic cases.</strong> A model can draft questions from a description. Husain suggests starting this way. Check them against the logs, since they drift from real traffic.</li>\n</ul>\n<p>Anthropic advises: \"Design evals that mirror your real-world task distribution\" (<a href=\"https://platform.claude.com/docs/en/test-and-evaluate/develop-tests\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic, define success criteria and build evaluations</a>). Include edge cases: irrelevant or nonexistent input, very long input, harmful input, and ambiguous cases where people would disagree too.</p>\n<p>Tag each case with its category and source, so you can report scores by group. Keep a slice of cases you never look at while tuning prompts, or the prompt learns the test set.</p>\n<h4 id=\"how-many-cases\">How many cases</h4>\n<p>How many cases you need follows from sampling error. The 95% <strong>margin of error</strong> on a pass rate p over n cases is about 1.96 × √(p × (1 − p) ÷ n). At an 80% pass rate, 100 cases give 1.96 × √(0.8 × 0.2 ÷ 100) ≈ ±7.8 points, and 400 cases give ±3.9 points. A 100-case set catches a large break, such as a fall from 80% to 65%, but it can't show that a 3-point change is real. Detecting a few points takes hundreds of cases, compared on the same cases.</p>","headings":[{"id":"building-an-eval-set","title":"Building an eval set","depth":3},{"id":"start-from-what-success-means","title":"Start from what success means","depth":4},{"id":"where-cases-come-from","title":"Where cases come from","depth":4},{"id":"how-many-cases","title":"How many cases","depth":4}]},{"id":"ways-to-grade","title":"Ways to grade","html":"<h3 id=\"ways-to-grade\">Ways to grade</h3>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Grader</th><th>Good at</th><th>Cost and speed</th><th>Main risk</th></tr></thead><tbody><tr><td>Exact match</td><td>Labels, IDs, fixed values</td><td>Free, instant</td><td>Rejects correct rewording</td></tr><tr><td>Code-based check</td><td>Format, required facts, banned phrases, cited pages</td><td>Cheap and fast, runs on every change</td><td>Checks only what you thought to write</td></tr><tr><td>Model-graded (LLM-as-judge)</td><td>Meaning, tone, whether an answer is supported</td><td>Costs tokens, seconds per case</td><td>Biased; must be calibrated</td></tr><tr><td>Human review</td><td>Anything, including subjective quality</td><td>Slow and expensive</td><td>Reviewers disagree; hard to scale</td></tr></tbody></table></div>\n<p>Anthropic states the central trade-off: \"More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals\" (<a href=\"https://platform.claude.com/docs/en/test-and-evaluate/develop-tests\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic</a>). Use cheap graders for volume, and people to check them.</p>\n<h4 id=\"exact-match-and-code-based-checks\">Exact match and code-based checks</h4>\n<p>Exact match suits categorical answers. If the assistant tags each message <code>refund</code>, <code>shipping</code> or <code>other</code>, compare the tag. Code checks structure and many facts: valid JSON, required fields, no internal order IDs in a customer reply, a length limit, and cited pages that appeared in the retrieved context. Husain recommends building a base of code checks first, since they are fast enough to run on every change. They can't judge meaning, so an answer can pass every check and still be wrong.</p>\n<h4 id=\"model-graded-evaluation\">Model-graded evaluation</h4>\n<p>A <strong>judge</strong> is a second model that reads the question, the answer and a rubric, then returns a verdict. It handles criteria code can't express, such as whether the policy supports an answer. OpenAI's guide lists pairwise comparison, single-answer grading and reference-guided grading as formats.</p>\n<p>Judges have measurable biases. In \"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena\" (<a href=\"https://arxiv.org/abs/2306.05685\" target=\"_blank\" rel=\"noopener noreferrer\">arXiv 2306.05685</a>, Zheng and colleagues, 2023), GPT-4 agreed with human preference votes on MT-Bench 85% of the time when ties were excluded. Two humans agreed 81% of the time under the same rule (Table 5). The abstract describes GPT-4's agreement as \"over 80%,\" \"the same level of agreement between humans.\" The appendix warns that the human figure may be low, given how it was computed. The authors name four biases and limits.</p>\n<ul>\n<li><strong>Position bias.</strong> With the default prompt, GPT-4 gave the same verdict after the two answers were swapped in 65.0% of comparisons, and it favored the first position in 30.0% (Table 2).</li>\n<li><strong>Verbosity bias.</strong> Judges prefer longer answers. The authors padded 23 answers by rewording each one's numbered list and adding the reworded copy in front, which made the answers longer without adding anything. Claude-v1 and GPT-3.5 preferred the padded answers in 91.3% of cases, and GPT-4 in 8.7% (Table 3).</li>\n<li><strong>Self-enhancement bias.</strong> Judges may favor answers from their own model family. The authors say their data are too limited to settle it.</li>\n<li><strong>Limited reasoning.</strong> On math questions, GPT-4 with the default prompt accepted an incorrect answer in 14 of 20 judgments (70%). With a reference answer, that fell to 3 of 20 (15%) (Section 3.4, Table 4). The sample is small.</li>\n</ul>\n<p>The paper's fixes include judging both orders and counting a disagreement as a tie, which the authors adopted as their default. Few-shot examples raised GPT-4's position consistency from 65.0% to 77.5% (Table 13), at four times the API cost for the longer prompts. Also grade with a different model than the one that wrote the answer, as Anthropic's example code suggests.</p>\n<p>Before you trust a judge, <strong>calibrate</strong> it against human labels. Raw agreement misleads when most outputs are fine, so measure <strong>precision</strong> (the share of the judge's bad flags that were right) and <strong>recall</strong> (the share of truly bad answers it caught). Husain says to \"measure precision and recall separately.\"</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> judge_report</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(judge_bad: list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], human_bad: list[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Compare a judge's 'bad' flags with human labels on the same outputs.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    pairs </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> list</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">zip</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(judge_bad, human_bad))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    tp </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j, h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pairs)      </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># both say bad</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    fp </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j, h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pairs)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># judge says bad, human says fine</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    fn </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j, h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pairs)  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># human says bad, judge missed it</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    agree </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(j </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> j, h </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> pairs) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(pairs)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"raw agreement </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">agree</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.0%</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, precision </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">tp </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (tp </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> fp)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.0%</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">          f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"recall </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">tp </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (tp </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> fn)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.0%</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 100 outputs. Humans flagged 20 as bad. The judge flagged 15, and 12 of those were right.</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">judge </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 12</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 8</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 77</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">human </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 12</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 8</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 77</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">judge_report(judge, human)</span></span></code></pre>\n<p>Output:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>raw agreement 89%, precision 80%, recall 60%</span></span></code></pre>\n<p>The judge looks good at 89% agreement, yet it caught only 60% of the bad answers. A judge that always said \"good\" would score 80% on the same sample, with zero recall. Re-calibrate whenever the model, the rubric or the answer format changes.</p>\n<h4 id=\"human-review-and-binary-scores\">Human review and binary scores</h4>\n<p>People are the reference. They notice failure modes nobody wrote down. OpenAI's guide rates human evals as top quality but slow and costly, so use people to label the calibration sample, read sampled traces and write rubric examples.</p>\n<p>Default to pass or fail. Husain often starts \"by labeling examples as good or bad,\" and he finds scores and finer ratings \"more onerous to manage than binary ratings.\" OpenAI's guide gives the same advice for reliability: \"Use pairwise comparison or pass/fail for more reliability.\"</p>","headings":[{"id":"ways-to-grade","title":"Ways to grade","depth":3},{"id":"exact-match-and-code-based-checks","title":"Exact match and code-based checks","depth":4},{"id":"model-graded-evaluation","title":"Model-graded evaluation","depth":4},{"id":"human-review-and-binary-scores","title":"Human review and binary scores","depth":4}]},{"id":"running-evals-before-release","title":"Running evals before release","html":"<h3 id=\"running-evals-before-release\">Running evals before release</h3>\n<p>Run the eval set whenever something that affects output changes: the prompt, the model version, the retrieval settings, the tools or the grader. Compare the results with a <strong>baseline</strong>, the scores of the version in production. A release gate blocks the change when a rule fails.</p>\n<figure class=\"diagram\" data-text=\"Candidate prompt or model -> Run each case, several times; Run each case, several times -> Graders: code, judge, sample (outputs); Graders: code, judge, sample -> Gate: any regressions? (scores); Gate: any regressions? -> Release or block; Eval set (versioned) -> Run each case, several times (inputs); Baseline scores -> Gate: any regressions? (last release)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -18 800 223\" width=\"800\" role=\"img\" aria-label=\"A change runs the eval set, and the gate compares the results with the baseline\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"240\" y1=\"128\" x2=\"240\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"560\" y1=\"128\" x2=\"560\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">Candidate</tspan><tspan x=\"80\" y=\"54.5\">prompt or model</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Run each case,</tspan><tspan x=\"240\" y=\"54.5\">several times</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"8\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"30.5\">Graders:</tspan><tspan x=\"400\" y=\"46.5\">code, judge,</tspan><tspan x=\"400\" y=\"62.5\">sample</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Gate: any</tspan><tspan x=\"560\" y=\"54.5\">regressions?</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"16\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"38.5\">Release</tspan><tspan x=\"720\" y=\"54.5\">or block</tspan></text><path class=\"dg-node dg-store\" d=\"M172,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M172,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"161.5\">Eval set</tspan><tspan x=\"240\" y=\"177.5\">(versioned)</tspan></text><path class=\"dg-node dg-store\" d=\"M492,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"161.5\">Baseline</tspan><tspan x=\"560\" y=\"177.5\">scores</tspan></text><rect class=\"dg-edge-bg\" x=\"292.3\" y=\"-14\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"-1\" text-anchor=\"middle\">outputs</text><rect class=\"dg-edge-bg\" x=\"455.4\" y=\"-14\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"-1\" text-anchor=\"middle\">scores</text><rect class=\"dg-edge-bg\" x=\"216.4\" y=\"91.5\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"104.5\" text-anchor=\"middle\">inputs</text><rect class=\"dg-edge-bg\" x=\"517.8\" y=\"91.5\" width=\"84.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"104.5\" text-anchor=\"middle\">last release</text></svg></div><figcaption>A change runs the eval set, and the gate compares the results with the baseline</figcaption></figure>\n<p>Husain suggests code checks on every change, model-based checks on a schedule, and A/B tests after significant product changes. Pin the exact model version each run used, and store it with the result, so the score can be reproduced. Decide the gate rules before the run. One workable set: no safety case may regress, the overall pass rate may not fall by more than 2 points, and p95 latency may not rise by more than 10%.</p>\n<h4 id=\"comparing-two-prompt-versions\">Comparing two prompt versions</h4>\n<p>The sketch runs an eval on two fake prompt versions. Each fake answers correctly with a fixed probability, promises too much with another, and hedges otherwise. The grader requires the key fact and rejects the word \"guarantee.\" Each question runs three times per seed, and the run repeats for five seeds to show the spread.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> random</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Question -> the fact a passing answer must contain.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">KEY_FACTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"How long do refunds take?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"5 business days\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"Can I return a sale item?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"30 days\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"Do you ship to Canada?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"ship to canada\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"Can I change my delivery address?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"until the order leaves\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> grade</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(answer: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, question: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Code-based grader: the fact must appear, and no banned promise may.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> KEY_FACTS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[question] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> answer.lower() </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"guarantee\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> answer.lower()</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> pass_rate</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(model, runs: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, seed: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Outputs vary, so run each question several times and count the passes.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    rng </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> random.Random(seed)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    results </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [grade(model(q, rng), q) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> q </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> KEY_FACTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> _ </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(runs)]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(results) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(results)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> make_model</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(p_good: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, p_banned: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Fake model: right fact with p_good, a banned promise with p_banned, else a hedge.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> model</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(question: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, rng: random.Random) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        fact, roll </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> KEY_FACTS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[question], rng.random()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> roll </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p_good:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Per our policy: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">fact</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">.\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> roll </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p_good </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p_banned:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"We guarantee </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">fact</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">.\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"I'm not sure. Please contact support.\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> model</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">PROMPTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"v1 terse prompt\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: make_model(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">p_good</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.6</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">p_banned</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"v2 with policy excerpts\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: make_model(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">p_good</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">p_banned</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> name, model </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PROMPTS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.items():</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        rates </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [pass_rate(model, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">seed</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)]</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">name</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(rates)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.0%</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> to </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{max</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(rates)</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.0%</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> across 5 seeds\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span></code></pre>\n<p>Running it prints:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>v1 terse prompt: 42% to 75% across 5 seeds</span></span>\n<span class=\"line\"><span>v2 with policy excerpts: 75% to 83% across 5 seeds</span></span></code></pre>\n<p>The fakes have true pass rates of 60% for v1 and 80% for v2. Yet one seed of v1 scored 75%, the same as the lowest seed of v2. Twelve passes or fails per seed cannot separate the two versions. Before you ship v2, add cases and repeats, and list which cases flipped from pass to fail and back. The flips often tell you more than the headline number.</p>","headings":[{"id":"running-evals-before-release","title":"Running evals before release","depth":3},{"id":"comparing-two-prompt-versions","title":"Comparing two prompt versions","depth":4}]},{"id":"evaluating-in-production","title":"Evaluating in production","html":"<h3 id=\"evaluating-in-production\">Evaluating in production</h3>\n<p>Offline evals cover only the cases you thought of. OpenAI's guide lists \"biased design,\" meaning test sets that don't represent real traffic, as an anti-pattern. Production adds three signals: sampled judging, online checks and user outcomes.</p>\n<h4 id=\"sampled-judging\">Sampled judging</h4>\n<p>Score a sample of live answers with the same judge and rubric you use offline. Do it off the request path, so customers never wait for a judge.</p>\n<figure class=\"diagram\" data-text=\"Assistant app emits spans -> Collector (OTLP); Collector -> Trace store (redacted) (all traces); Collector -> Sample 1-5% of traces (sample); Sample 1-5% of traces -> Judge workers; Judge workers -> Scores (pass or fail); Scores -> Dashboard and alerts\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -15 800 218\" width=\"800\" role=\"img\" aria-label=\"Sampled judging runs off the request path\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"240\" y1=\"69\" x2=\"240\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"311\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"162\" x2=\"647\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">Assistant app</tspan><tspan x=\"80\" y=\"54.5\">emits spans</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"46.5\">Collector</tspan></text><path class=\"dg-node dg-store\" d=\"M332,18 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,18 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"41.5\">Trace store</tspan><tspan x=\"400\" y=\"57.5\">(redacted)</tspan></text><rect class=\"dg-node dg-queue\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"292\" y1=\"144\" x2=\"292\" y2=\"180\"></line><line class=\"dg-line\" x1=\"282\" y1=\"144\" x2=\"282\" y2=\"180\"></line><line class=\"dg-line\" x1=\"272\" y1=\"144\" x2=\"272\" y2=\"180\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"226\" y=\"158.5\">Sample 1-5%</tspan><tspan x=\"226\" y=\"174.5\">of traces</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"166.5\">Judge workers</tspan></text><path class=\"dg-node dg-store\" d=\"M492,140 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,140 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"169.5\">Scores</tspan></text><rect class=\"dg-node dg-box\" x=\"652\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"158.5\">Dashboard</tspan><tspan x=\"720\" y=\"174.5\">and alerts</tspan></text><rect class=\"dg-edge-bg\" x=\"141.6\" y=\"-6\" width=\"34.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"7\" text-anchor=\"middle\">OTLP</text><rect class=\"dg-edge-bg\" x=\"283\" y=\"-11\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"2\" text-anchor=\"middle\">all traces</text><rect class=\"dg-edge-bg\" x=\"216.4\" y=\"91\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"104\" text-anchor=\"middle\">sample</text><rect class=\"dg-edge-bg\" x=\"436.8\" y=\"111\" width=\"84.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"124\" text-anchor=\"middle\">pass or fail</text></svg></div><figcaption>Sampled judging runs off the request path</figcaption></figure>\n<p>A judge call is a model call, so judging costs money. Take 1 million requests a day, judge 1% of them, and assume each judge call averages 1,500 input and 150 output tokens. That is 10,000 calls: 15 million input tokens and 1.5 million output tokens. At assumed prices of $3 and $15 per million tokens, the judge costs 15 × $3 + 1.5 × $15 = $45 + $22.50 = $67.50 a day.</p>\n<p>Sampling adds noise. If 500 judged answers a day pass at 90%, the daily margin is 1.96 × √(0.9 × 0.1 ÷ 500) ≈ ±2.6 points. Comparing two such days gives a margin on the difference of 1.96 × √(2 × 0.9 × 0.1 ÷ 500) ≈ ±3.7 points. Alert on sustained drops across several days, and treat one bad day as a reason to look.</p>\n<h4 id=\"online-checks\">Online checks</h4>\n<p>Some checks must run before a reply leaves the server, because they can block it. Such a check is a <strong>guardrail</strong>, such as JSON validation, detection of personal data that shouldn't appear in replies, or handling of refusals and truncated answers. <a href=\"/learn/llm-gateways#logging-and-guardrails\">LLM gateways</a> covers where guardrails usually sit.</p>\n<h4 id=\"ab-tests-and-user-outcomes\">A/B tests and user outcomes</h4>\n<p>An A/B test splits users between two prompt versions and compares an outcome that matters, such as conversations that end without a handoff to a person. Size it first. Moving a thumbs-up rate from 50% to 52%, at 5% significance (the chance of a false win) and 80% power (the chance of detecting a real lift), needs about 9,800 users per arm. The formula is n = (1.96 × √(2 × pbar × (1 − pbar)) + 0.84 × √(p1 × (1 − p1) + p2 × (1 − p2)))² ÷ (p1 − p2)², where pbar is the average of p1 and p2. Fix the end date or sample size in advance, since stopping when the result looks good inflates false wins.</p>\n<p>User feedback is a useful signal that is also noisy. Thumbs up and down, regenerations, edits, copies and escalations all carry information, but explicit feedback is sparse and skews toward strong opinions. Use these signals to pick conversations for review, and add the reviewed failures to the eval set, so each production failure guards the next release. Redact personal data, restrict access and set a retention limit (<a href=\"/learn/ai-security#leaking-data\">Security for AI systems</a>).</p>","headings":[{"id":"evaluating-in-production","title":"Evaluating in production","depth":3},{"id":"sampled-judging","title":"Sampled judging","depth":4},{"id":"online-checks","title":"Online checks","depth":4},{"id":"ab-tests-and-user-outcomes","title":"A/B tests and user outcomes","depth":4}]},{"id":"tracing-model-calls","title":"Tracing model calls","html":"<h3 id=\"tracing-model-calls\">Tracing model calls</h3>\n<p>A <strong>span</strong> is one timed step, and a <strong>trace</strong> is the tree of spans for one request. <a href=\"/learn/observability#distributed-tracing\">Monitoring, logging and tracing</a> explains both.</p>\n<h4 id=\"one-request-many-spans\">One request, many spans</h4>\n<p>A support question usually embeds the question, searches the index, calls the model and sometimes calls a tool, such as an order lookup. Each step is a span. The model call gets its own span, because it is often the slowest and most expensive step.</p>\n<figure class=\"diagram\" data-text=\"Root span POST /ask 2,350 ms -> Embedding query, 25 ms; Root span POST /ask 2,350 ms -> Retrieval vector search 40 ms; Root span POST /ask 2,350 ms -> Model call chat, 2,100 ms; Model call chat, 2,100 ms -> Tool call execute_tool 120 ms (order lookup)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 480 328\" width=\"480\" role=\"img\" aria-label=\"One support question as a trace, with timings from an example request\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"129.3\" y1=\"125\" x2=\"198.7\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"129.3\" y1=\"199\" x2=\"198.7\" y2=\"251\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"282\" x2=\"327\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"128\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"150.5\">Root span</tspan><tspan x=\"80\" y=\"166.5\">POST /ask</tspan><tspan x=\"80\" y=\"182.5\">2,350 ms</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Embedding</tspan><tspan x=\"240\" y=\"54.5\">query, 25 ms</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"128\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"150.5\">Retrieval</tspan><tspan x=\"240\" y=\"166.5\">vector search</tspan><tspan x=\"240\" y=\"182.5\">40 ms</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"278.5\">Model call</tspan><tspan x=\"240\" y=\"294.5\">chat, 2,100 ms</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"248\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"270.5\">Tool call</tspan><tspan x=\"400\" y=\"286.5\">execute_tool</tspan><tspan x=\"400\" y=\"302.5\">120 ms</tspan></text><rect class=\"dg-edge-bg\" x=\"276.8\" y=\"226\" width=\"84.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"239\" text-anchor=\"middle\">order lookup</text></svg></div><figcaption>One support question as a trace, with timings from an example request</figcaption></figure>\n<p>The model span carries the time to first token and the token counts. The tool span shows whether the order lookup, rather than the model, was the slow step.</p>\n<h4 id=\"the-genai-attributes\">The GenAI attributes</h4>\n<p>OpenTelemetry's semantic conventions for generative AI name the attributes a model span should carry. The spec lives in its own repository and is still marked <strong>Development</strong>, a level short of stable. Its stability page doesn't say what Development allows for attribute names, so treat the names as provisional (<a href=\"https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md\" target=\"_blank\" rel=\"noopener noreferrer\">OpenTelemetry GenAI spans</a>). An inference span is named <code>{operation} {model}</code>, such as <code>chat</code> plus the requested model's name, and uses the client span kind (<a href=\"https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md\" target=\"_blank\" rel=\"noopener noreferrer\">OpenTelemetry GenAI client inference</a>).</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Attribute</th><th>What it records</th><th>Requirement</th></tr></thead><tbody><tr><td><code>gen_ai.operation.name</code></td><td><code>chat</code>, <code>text_completion</code>, <code>embeddings</code>, <code>retrieval</code>, <code>invoke_agent</code> or <code>execute_tool</code></td><td>Required on the span</td></tr><tr><td><code>gen_ai.response.model</code></td><td>The model that actually answered</td><td>Recommended</td></tr><tr><td><code>gen_ai.usage.input_tokens</code>, <code>gen_ai.usage.output_tokens</code></td><td>Tokens in and out</td><td>Recommended</td></tr><tr><td><code>gen_ai.usage.cache_read.input_tokens</code></td><td>Input tokens served from a cache, counted within input tokens</td><td>Recommended when non-zero</td></tr><tr><td><code>gen_ai.response.finish_reasons</code></td><td>Why generation stopped</td><td>Recommended</td></tr><tr><td><code>gen_ai.request.temperature</code>, <code>gen_ai.request.max_tokens</code></td><td>Sampling and length settings</td><td>Recommended</td></tr><tr><td><code>gen_ai.response.time_to_first_chunk</code></td><td>Seconds from the request to the first streamed chunk</td><td>Recommended for streaming</td></tr></tbody></table></div>\n<p>Add your own attributes too, such as the prompt, retrieval index and grader versions, so you can slice every dashboard by version. Keep them out of metric labels, because the <a href=\"/learn/observability#cardinality-and-cost\">cardinality rules</a> apply to prompt versions as much as to user IDs.</p>\n<h4 id=\"content-and-privacy\">Content and privacy</h4>\n<p>The fields <code>gen_ai.input.messages</code> and <code>gen_ai.output.messages</code> are opt-in. The spec warns that output messages are \"likely to contain sensitive information including user/PII data.\" Record metadata by default: models, token counts, timings and finish reasons. Store full content only for a small sample, in a restricted store, with redaction and a short retention period.</p>","headings":[{"id":"tracing-model-calls","title":"Tracing model calls","depth":3},{"id":"one-request-many-spans","title":"One request, many spans","depth":4},{"id":"the-genai-attributes","title":"The GenAI attributes","depth":4},{"id":"content-and-privacy","title":"Content and privacy","depth":4}]},{"id":"metrics-to-watch","title":"Metrics to watch","html":"<h3 id=\"metrics-to-watch\">Metrics to watch</h3>\n<p>Four kinds of panel answer what an operator asks: speed, cost, errors and quality.</p>\n<h4 id=\"latency\">Latency</h4>\n<p><strong>Time to first token (TTFT)</strong> is the wait before the first output token. <strong>Time per output token (TPOT)</strong> is the average gap between output tokens after that. <a href=\"/learn/llm-inference#latency-and-throughput-metrics\">How LLM inference works</a> defines both. OpenTelemetry defines server-side histograms for them, <code>gen_ai.server.time_to_first_token</code> and <code>gen_ai.server.time_per_output_token</code>, in seconds (<a href=\"https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md\" target=\"_blank\" rel=\"noopener noreferrer\">OpenTelemetry GenAI metrics</a>). Chart percentiles, because averages hide slow requests (<a href=\"/learn/performance-and-latency#averages-hide-the-slow-requests\">Performance and latency</a>).</p>\n<p>For the handbook's running deployment, Llama 3.1 70B on four H100 GPUs, memory bandwidth alone caps one user at about 96 tokens a second, and 64 users sharing the GPUs at about 74 tokens a second each (<a href=\"/learn/ai-estimation#how-fast-it-can-run\">Estimating AI workloads</a> works this out). Real deployments run slower than these limits. Alert from the TPOT you measure on a healthy system, since a TPOT well above that baseline usually means a bigger batch, a fuller cache or a slower code path.</p>\n<h4 id=\"tokens-and-cost\">Tokens and cost</h4>\n<p>Track input and output tokens per request, and chart their distribution, since a few very long prompts can dominate the bill. Cost per request is input tokens × input price + output tokens × output price. On Anthropic's pricing page, as of October 2026, output tokens cost five times the base input price for every model listed, and cache hits cost a tenth of the base input price or less (<a href=\"https://platform.claude.com/docs/en/about-claude/pricing\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic pricing</a>). So a verbose answer costs more than a prompt of the same length. A request with 3,000 input and 400 output tokens costs 3,000 × $3 ÷ 1,000,000 + 400 × $15 ÷ 1,000,000 = $0.009 + $0.006 = $0.015 at assumed prices, or $15,000 a month for a million such requests. Chart cost per feature, per team and per successful task (<a href=\"/learn/llm-gateways#tracking-cost\">LLM gateways</a>).</p>\n<h4 id=\"errors-and-refusals\">Errors and refusals</h4>\n<p>Track error rates per route and per prompt version. The <code>error.type</code> attribute, marked Stable in the spec, names the failure. Count stop reasons too. A refusal is a successful HTTP call, so it hides among your 200s until you chart it. Rising refusals or truncations are quality problems, even when no status code changes. Page on sudden jumps that hurt users, and send slow drifts to a ticket (<a href=\"/learn/observability#alerting-on-symptoms\">symptom-based alerting</a>).</p>\n<h4 id=\"quality\">Quality</h4>\n<p>Quality panels show the pass rate of the golden set, run nightly against production, the pass rate of sampled judging, and user outcomes such as escalations. Read the judge's daily moves against its margin of about ±2.6 points, and treat a change inside that margin as noise until it persists. Place quality beside latency and cost, so a quality gain that doubles cost is visible at once. <a href=\"/learn/rag#evaluating-retrieval-and-answers\">RAG</a> covers retrieval quality. Filter every panel by model and prompt version, and read the four questions in order: is it up, is it fast, is it affordable, is it good?</p>","headings":[{"id":"metrics-to-watch","title":"Metrics to watch","depth":3},{"id":"latency","title":"Latency","depth":4},{"id":"tokens-and-cost","title":"Tokens and cost","depth":4},{"id":"errors-and-refusals","title":"Errors and refusals","depth":4},{"id":"quality","title":"Quality","depth":4}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>A new prompt raises the judge's pass rate on a 100-case eval set from 81% to 84%. Should the team ship it?</summary>\n<p>Not on this evidence. The 95% margin on each rate is about 1.96 × √(0.81 × 0.19 ÷ 100) ≈ ±7.7 points, so a 3-point gap sits inside it. If only a few cases flipped, the change is probably noise. Add cases and repeats, and ship only if no critical case regressed.</p>\n</details>\n<details>\n<summary>A judge flags 15 of 100 answers as bad. Humans flagged 20, and 12 of the judge's flags were correct. A colleague says 89% raw agreement proves the judge is good. What do you say?</summary>\n<p>Recall is 12 ÷ 20 = 60%, so the judge misses 8 of the 20 bad answers. Precision is 12 ÷ 15 = 80%. A judge that always said \"good\" would score 80% raw agreement here. Keep it out of release decisions until its recall is acceptable.</p>\n</details>\n<details>\n<summary>After a model upgrade, the pass rate falls 6 points, while latency and cost stay flat. Where do you start?</summary>\n<p>Split the results by category, and read the traces of the cases that flipped. Check stop reasons and output token counts, since longer answers can change a grader's verdict. Re-run the same cases on the old model, and check whether the judge changed in the same release.</p>\n</details>\n<details>\n<summary>An A/B test should show whether a prompt lifts the thumbs-up rate from 40% to 42%. Roughly how many users does each arm need at 5% significance and 80% power? How long does the test take with 5,000 eligible users a day?</summary>\n<p>With p1 = 0.40, p2 = 0.42 and pbar = 0.41, the numerator is (1.96 × √(2 × 0.41 × 0.59) + 0.84 × √(0.40 × 0.60 + 0.42 × 0.58))², about (1.363 + 0.584)² = 3.79. Divide by 0.02² = 0.0004 to get about 9,500 users per arm, or 19,000 in total. At 5,000 eligible users a day, split evenly, that takes about four days.</p>\n</details>\n<details>\n<summary>A trace shows the model span stopped at the token limit, and its output token count equals the configured maximum. What happened, and what would you change?</summary>\n<p>The reply was cut off, so the customer got half an answer. Check whether it ends mid-sentence or mid-JSON. Then decide whether the limit is too low, or whether the new prompt made answers longer. Raise the limit or shorten the instructions, and chart the truncation rate per prompt version.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md\" target=\"_blank\" rel=\"noopener noreferrer\">OpenTelemetry GenAI spans</a>: attribute names and requirement levels. Still at Development.</li>\n<li><a href=\"https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md\" target=\"_blank\" rel=\"noopener noreferrer\">OpenTelemetry GenAI client inference</a>: span names and token and streaming attributes.</li>\n<li><a href=\"https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md\" target=\"_blank\" rel=\"noopener noreferrer\">OpenTelemetry GenAI metrics</a>: the latency histograms.</li>\n<li><a href=\"https://arxiv.org/abs/2306.05685\" target=\"_blank\" rel=\"noopener noreferrer\">Zheng et al., \"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena\"</a>: the agreement numbers and the four biases.</li>\n<li><a href=\"https://hamel.dev/blog/posts/evals/\" target=\"_blank\" rel=\"noopener noreferrer\">Hamel Husain, \"Your AI Product Needs Evals\"</a>: code checks first, binary labels, and judge precision and recall.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/evaluation-best-practices\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI, Evaluation best practices</a>: the five-step process, grader formats and anti-patterns.</li>\n<li><a href=\"https://platform.claude.com/docs/en/test-and-evaluate/develop-tests\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic, Define success criteria and build evaluations</a>: grader types and the edge-case checklist.</li>\n<li><a href=\"https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic, Stop reasons and fallback</a>: the refusal and token-limit stop reasons.</li>\n<li><a href=\"https://platform.claude.com/docs/en/about-claude/pricing\" target=\"_blank\" rel=\"noopener noreferrer\">Anthropic pricing</a>: output-to-input price ratios. Prices change, so check the page before you budget.</li>\n<li><a href=\"https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/\" target=\"_blank\" rel=\"noopener noreferrer\">Thinking Machines, Defeating Nondeterminism in LLM Inference</a>: why temperature 0 still varies.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}