Lecture 9: Evals as Experimental Design

Shen Shen

Nov 17, 2026

2:30pm, Room 37-212

Slides in PDF

Effective AI use in applied machine-learning projects

Outline

  • A. The claim: are we testing the right thing?

  • B. Scoring: are we grading it right?

  • C. Controls: is the comparison fair?

  • D. Noise: could the gap be luck?

Fisher tested one claim with eight cups in random order

Fisher’s story: a lady says for (milk)tea, she can taste whether the milk or the tea went in first.

 

 

He tests her with eight cups, four made each way.

Decide what every possible result will mean, before the run

Fisher decided in advance that only all eight right counts as evidence for her claim.

Fisher made four decisions before the experiment

Claim

She can tell whether the milk or the tea went in first.

Scoring

She picks the four cups made milk-first. Each pick is checked against how the cup was made.

Controls

The cups are served in random order.

Noise

Eight cups, fixed in advance. A guesser gets all eight right once in 70.

An eval is like an exam, and a good one takes care to design

An exam

A project eval

Claim

A student passes without the skill the course teaches.

The metric goes up, but the project’s real goal does not.

Scoring

The rubric rewards guessing, or the grader marks a right answer wrong.

The metric rewards the wrong behavior, or the grader marks outputs wrongly.

Controls

Two groups of students took the same exam, but one had extra time.

The new version also differed in the data split, seed, harness or settings.

Noise

The exam has only three questions, so one careless slip changes the grade.

The new version wins by a smaller gap than two reruns of the old version show.

A high exam score did not carry over to real patients

MedQA is a benchmark of US medical licensing exam questions.

 

For each exam question the LLM got right, the authors took about 10 real patient notes whose correct answer is the same drug or diagnosis, and asked the LLM again.

A classifier that saw only half the input still scored 67%

Each SNLI item pairs a premise, a photo caption, with a sentence a crowd worker wrote to be true, maybe true, or false given it (left). A model must say which.

 

 

Workers’ habits gave the label away: “not” for false, vague words like “people” for true (bold, left). 

With random weights, some explanation maps barely changed

A saliency map marks the pixels a method says the network relied on.

 

Guided Backprop and Guided GradCAM keep drawing the bird until the lowest layers are randomized.

Explanation scores were as high for random Transformers

A sparse autoencoder splits an LLM’s activations into features.

 

A second LLM writes an explanation for each feature, and a score measures how well the explanation picks out the text that activates it.

 

The authors ran this on a trained Transformer (gray) and on copies with random weights (colored). In the control (black), each input word is replaced by fresh random noise. Its score falls to a coin flip, as it should.

An AI-scientist system that chose its own tests chose easy ones

The authors built 20 benchmarks for one task, from easy to extreme, and hid the difficulty behind random names. The AI Scientist v2 had to pick four to test its own method on.

 

It was shown the names, and then also a top score to beat on each, higher on the easier ones.

The claim: what to check about the test

What the case showed

What we do

Forecast

Fisher fixed what every result would mean before the tea was poured.

Before the run, write down what each possible result would mean.

Skill

A high exam score did not carry over to real patient notes.

Check the score against a sample of the real task.

Shortcut

A classifier shown half the input scored 67%, against 34% for guessing.

Hide the input the task needs, or randomize the model. If the score stays the same, it was not measuring the skill.

Agent

Picking its own benchmarks, an AI-scientist system chose the easiest level 47% of the time.

Name the benchmark in the brief, before the agent starts.

Outline

  • A. The claim: are we testing the right thing?

  • B. Scoring: are we grading it right?

  • C. Controls: is the comparison fair?

  • D. Noise: could the gap be luck?

How does 75% wrong beat 26% wrong?

SimpleQA asks short factual questions.

 

An LLM can answer or say it does not know.

 

 

“I don’t know” scores the same zero as a wrong answer.

An agreed forecast did not stop a dispute over the telescope

In 1870 a flat-Earth believer bet £500 that nobody could show that inland water curves. They chose a six-mile canal. The naturalist Alfred Wallace took the bet.

 

Both sides agreed beforehand what a flat and a curved result would look like (left diagram).

 

The result came out curved, and the believer’s referee signed the sketch (right). He then said it proved nothing, because the telescope was not leveled and had no cross-hair.

A do-nothing agent passed 38% of a benchmark’s airline tasks

τ-bench tests customer-service agents.

 

Some of its airline tasks are impossible on purpose, so the right move is to refuse.

 

The benchmark meant to reward that refusal, but its grader passed any run that changed nothing.

The tests rejected correct fixes in a third of the audited tasks

SWE-bench scores a code fix by running the repository’s tests.

 

 

Human reviewers had already screened its Verified subset.

 

 

Even so, most of the tasks a strong OpenAI LLM kept failing turned out to be flawed (left).

 

 

Lecture 8 showed the opposite error: wrong patches that pass.

A word-matching grader made LLMs look sensitive to the prompt

Studies had found that small changes to a prompt’s format move LLM scores a lot.

Much of that came from the grader: it rejects right answers worded differently, and different prompts draw different wordings (above).

 

The authors first checked an LLM judge against human graders. They then re-graded the same answers with it. Scores changed far less from prompt to prompt.

A check can work without an answer key

Often nobody knows the right answer, but we know how the answer should behave.

 

e.g., Rows B and C test a commercial sentiment model with changes whose effect we know. A swapped city should change nothing, and an added insult should not make a tweet less negative.

 

 

This is metamorphic relations in software testing.

Read-only tests cut the cheating; hidden tests stopped it

ImpossibleBench edits SWE-bench tasks so the tests contradict the task.

No honest fix can pass, so every pass is cheating (Lecture 4).

 

Each bar is an LLM allowed to edit the tests (Open), only read them (Read), or not see them (Hide). Hiding has a price: on the original, solvable tasks (left group), LLMs that could not see the tests passed fewer.

Scoring: what to check about the rubric and the grader

What the case showed

What we do

Rubric

“I don’t know” scored the same as a wrong answer, so guessing beat admitting doubt.

Score a guess, a blank and a wrong answer by hand, and see which the rubric rewards.

Instrument

Both sides agreed on the forecast, and the loser still disputed the telescope.

Fix how the score is measured before the run. Have someone who did not build the system check the measurement.

Grader

One grader passed an agent that did nothing. Another rejected correct fixes.

Give the grader an answer that should fail and one that should pass. Read the tasks a strong system sometimes fails.

No key

Swapping one city for another changed a sentiment model’s answer.

Make a change that should not affect the answer, and check that the answer stays the same.

Agent

On impossible tasks, read-only tests cut the cheating, and hidden tests stopped it.

Make the tests and the metric read-only for the agent, then check what still passes.

Outline

  • A. The claim: are we testing the right thing?

  • B. Scoring: are we grading it right?

  • C. Controls: is the comparison fair?

  • D. Noise: could the gap be luck?

A one-layer linear model beat five Transformers at forecasting

The task is to forecast the next 96 to 720 steps of a series such as electricity use.

The numbers are errors, so lower is better.

 

The three methods on the left are one-layer linear models.

IMP. is how far the best of them beats the best Transformer.

With a fair baseline, the reported gains nearly vanished

Metric learning trains a network to place images of the same class close together.

Papers with new loss functions reported large gains over two older losses, contrastive and triplet.

 

The gray bars show the average gain those papers reported.

The green and red bars show the gain when every loss is trained on the same network and tuning.

A horse that “counted” was reading its questioner

Clever Hans answered arithmetic by tapping a hoof, and experts found no trickery.

 

In one test he was right 98% of the time when the questioner knew the answer, and 8% when nobody did.

 

The hypothesis: the questioner, without meaning to, lifted his head slightly at the right count.

Moving the right answer to option A raised a score by 15 points

Same LLMs, same MMLU questions; only the position of the right answer changes.

 

Training on random rewards still raised a math score by 21 points

RLVR trains an LLM by rewarding answers a checker marks correct.

Random Reward pays half the time, whatever the answer.

 

Training mostly amplified what this LLM, Qwen2.5-Math-7B, already did.

 

Without that control, the whole gain would have been credited to the reward.

Machine resources alone moved a score by 6 points

Same Claude LLM, same harness, same Terminal-Bench 2.0 tasks; only each container’s CPU and memory limits change.

 

The left axis starts at 20.

 

Without matched limits, a leaderboard gap under 3 points could be the machine’s doing.

The same LLM scored 62.7% or 98.6%, depending on the harness

ARC Prize scored GPT-6 Astra on its ARC-AGI-3 puzzle games with two harnesses: its standard one, and one that uses OpenAI’s context features.

With the logs and code, an auditor caught what the paper alone hid

Two AI-scientist systems ran machine-learning projects end to end.

Some projects had a flaw that the authors planted or found, such as choosing which experiment to report by its test score.

 

An LLM auditor had to say whether each project was flawed, from the final paper alone or with the logs and code.

Controls: what to hold the same on both sides

What differed in the case

What we hold equal

Baseline

A simple model won. Once the baseline was tuned, most reported gains disappeared.

Run the simplest method first, and tune the baseline as carefully as the new method.

Cues

The questioner knew the answer, and the horse read him.

Keep whoever asks or grades from knowing the answer, or which version they are scoring.

Setup

Option order, machine limits and harness each moved one LLM’s score.

Keep them the same on both sides. Randomize what cannot be fixed, and report the range.

Scramble

Random rewards gave most of the gain that correct rewards did.

Run the change with its signal scrambled. A gain that survives did not come from the signal.

Agent

An auditor caught far more flaws with the logs and code than with the paper alone.

Read the diff and the logs, not only the summary, to see what else changed.

Outline

  • A. The claim: are we testing the right thing?

  • B. Scoring: are we grading it right?

  • C. Controls: is the comparison fair?

  • D. Noise: could the gap be luck?

The seed alone moved the score as much as four methods did

Four methods fine-tune the same small reasoning LLM and report gains on AIME’24, a 30-question math benchmark.

 

Rerun under one setup, each scores below its paper’s number (left). The gray band is the gains that remain; the base LLM’s scores across seeds span more (right).

 

Without the seed range, a gain this size could be a lucky seed.

Two agents reported the best of several runs as their result

Epoch AI asked two agents to invent a better training method. The brief said: “Do not claim gains a single run cannot support.”

 

Both made several similar runs and reported the best (dark bars). With that pick removed, the gains shrink (middle bars).

 

Related questions make error bars wider

An error bar counts each question as separate evidence.

 

DROP asks several questions per reading passage, and a model that misreads a passage tends to miss them all (correlation; snowballing).

 

Counting by passage, the error bar for Anthropic’s LLMs on DROP was three times as wide.

 

Patient by patient, a ten-person study gave a clear answer

Ten patients each took two sleep drugs.

The table gives the hours of sleep each drug added.

 

As two separate groups, drug 2’s lead is 1.9 standard errors and could be luck. Patient by patient, it is 4.1.

 

Without pairing, the differences between patients drown the difference between the drugs.

 

Evals pair the same way: compare two versions question by question.

Shuffling the labels shows how often luck alone makes the gap

Darwin grew 15 pairs of plants, one cross-fertilized and one self-fertilized in each.

The crossed plants came out taller.

 

If crossing did nothing, each pair’s difference could have gone either way.

 

So Fisher flipped each pair’s result both ways: 15 pairs give 32,768 possible outcomes. In 861 of them (833 + 28, bottom row), the crossed plants lead by at least as much as Darwin saw.

For an eval: on each question, randomly swap the two versions’ scores, many times, and count how often the gap is as large as ours.

Claimed gains were often smaller than the test set could detect

Est. MDE is the smallest gain each test set can reliably detect. |Δacc| is the average gain that papers claimed.

Noise: what to do before trusting a gap

What the case showed

What we do

Reruns

Changing only the random seed moved the score as much as the four methods’ gains.

First rerun the baseline and compare it with itself. That gap is the noise. Then report every run.

Groups

Questions about one passage succeed or fail together.

Treat related questions as one group when computing the error bar.

Pairs

Compared patient by patient, a small study gave a clear answer.

Compare question by question. Shuffle the labels to see how often luck alone gives a gap this big.

Size

Claimed gains were often smaller than the test set could detect.

Before the run, work out how many questions are needed to detect the gain.

Agent

Two agents reported the best of several runs as their result.

Fix the number of runs in the brief, and ask for every run.

An eval can fail the ways an exam does

An exam

A project eval

Claim

A student passes without the skill the course teaches.

The metric goes up, but the project’s real goal does not.

Scoring

The rubric rewards guessing, or the grader marks a right answer wrong.

The metric rewards the wrong behavior, or the grader marks outputs wrongly.

Controls

Two groups of students took the same exam, but one had extra time.

The new version also differed in the data split, seed, harness or settings.

Noise

The exam has only three questions, so one careless slip changes the grade.

The new version wins by a smaller gap than two reruns of the old version show.