A. The claim: are we testing the right thing?
B. Scoring: are we grading it right?
C. Controls: is the comparison fair?
D. Noise: could the gap be luck?
Fisher’s story: a lady says for (milk)tea, she can taste whether the milk or the tea went in first.
He tests her with eight cups, four made each way.
Fisher decided in advance that only all eight right counts as evidence for her claim.
Claim
She can tell whether the milk or the tea went in first.
Scoring
She picks the four cups made milk-first. Each pick is checked against how the cup was made.
Controls
The cups are served in random order.
Noise
Eight cups, fixed in advance. A guesser gets all eight right once in 70.
An exam
A project eval
Claim
A student passes without the skill the course teaches.
The metric goes up, but the project’s real goal does not.
Scoring
The rubric rewards guessing, or the grader marks a right answer wrong.
The metric rewards the wrong behavior, or the grader marks outputs wrongly.
Controls
Two groups of students took the same exam, but one had extra time.
The new version also differed in the data split, seed, harness or settings.
Noise
The exam has only three questions, so one careless slip changes the grade.
The new version wins by a smaller gap than two reruns of the old version show.
MedQA is a benchmark of US medical licensing exam questions.
For each exam question the LLM got right, the authors took about 10 real patient notes whose correct answer is the same drug or diagnosis, and asked the LLM again.
Each SNLI item pairs a premise, a photo caption, with a sentence a crowd worker wrote to be true, maybe true, or false given it (left). A model must say which.
Workers’ habits gave the label away: “not” for false, vague words like “people” for true (bold, left).
A saliency map marks the pixels a method says the network relied on.
Guided Backprop and Guided GradCAM keep drawing the bird until the lowest layers are randomized.
A sparse autoencoder splits an LLM’s activations into features.
A second LLM writes an explanation for each feature, and a score measures how well the explanation picks out the text that activates it.
The authors ran this on a trained Transformer (gray) and on copies with random weights (colored). In the control (black), each input word is replaced by fresh random noise. Its score falls to a coin flip, as it should.
The authors built 20 benchmarks for one task, from easy to extreme, and hid the difficulty behind random names. The AI Scientist v2 had to pick four to test its own method on.
It was shown the names, and then also a top score to beat on each, higher on the easier ones.
What the case showed
What we do
Forecast
Fisher fixed what every result would mean before the tea was poured.
Before the run, write down what each possible result would mean.
Skill
A high exam score did not carry over to real patient notes.
Check the score against a sample of the real task.
Shortcut
A classifier shown half the input scored 67%, against 34% for guessing.
Hide the input the task needs, or randomize the model. If the score stays the same, it was not measuring the skill.
Agent
Picking its own benchmarks, an AI-scientist system chose the easiest level 47% of the time.
Name the benchmark in the brief, before the agent starts.
A. The claim: are we testing the right thing?
B. Scoring: are we grading it right?
C. Controls: is the comparison fair?
D. Noise: could the gap be luck?
SimpleQA asks short factual questions.
An LLM can answer or say it does not know.
“I don’t know” scores the same zero as a wrong answer.
In 1870 a flat-Earth believer bet £500 that nobody could show that inland water curves. They chose a six-mile canal. The naturalist Alfred Wallace took the bet.
Both sides agreed beforehand what a flat and a curved result would look like (left diagram).
The result came out curved, and the believer’s referee signed the sketch (right). He then said it proved nothing, because the telescope was not leveled and had no cross-hair.
τ-bench tests customer-service agents.
Some of its airline tasks are impossible on purpose, so the right move is to refuse.
The benchmark meant to reward that refusal, but its grader passed any run that changed nothing.
SWE-bench scores a code fix by running the repository’s tests.
Human reviewers had already screened its Verified subset.
Even so, most of the tasks a strong OpenAI LLM kept failing turned out to be flawed (left).
Lecture 8 showed the opposite error: wrong patches that pass.
Studies had found that small changes to a prompt’s format move LLM scores a lot.
Much of that came from the grader: it rejects right answers worded differently, and different prompts draw different wordings (above).
The authors first checked an LLM judge against human graders. They then re-graded the same answers with it. Scores changed far less from prompt to prompt.
Often nobody knows the right answer, but we know how the answer should behave.
e.g., Rows B and C test a commercial sentiment model with changes whose effect we know. A swapped city should change nothing, and an added insult should not make a tweet less negative.
This is metamorphic relations in software testing.
ImpossibleBench edits SWE-bench tasks so the tests contradict the task.
No honest fix can pass, so every pass is cheating (Lecture 4).
Each bar is an LLM allowed to edit the tests (Open), only read them (Read), or not see them (Hide). Hiding has a price: on the original, solvable tasks (left group), LLMs that could not see the tests passed fewer.
What the case showed
What we do
Rubric
“I don’t know” scored the same as a wrong answer, so guessing beat admitting doubt.
Score a guess, a blank and a wrong answer by hand, and see which the rubric rewards.
Instrument
Both sides agreed on the forecast, and the loser still disputed the telescope.
Fix how the score is measured before the run. Have someone who did not build the system check the measurement.
Grader
One grader passed an agent that did nothing. Another rejected correct fixes.
Give the grader an answer that should fail and one that should pass. Read the tasks a strong system sometimes fails.
No key
Swapping one city for another changed a sentiment model’s answer.
Make a change that should not affect the answer, and check that the answer stays the same.
Agent
On impossible tasks, read-only tests cut the cheating, and hidden tests stopped it.
Make the tests and the metric read-only for the agent, then check what still passes.
A. The claim: are we testing the right thing?
B. Scoring: are we grading it right?
C. Controls: is the comparison fair?
D. Noise: could the gap be luck?
The task is to forecast the next 96 to 720 steps of a series such as electricity use.
The numbers are errors, so lower is better.
The three methods on the left are one-layer linear models.
IMP. is how far the best of them beats the best Transformer.
Metric learning trains a network to place images of the same class close together.
Papers with new loss functions reported large gains over two older losses, contrastive and triplet.
The gray bars show the average gain those papers reported.
The green and red bars show the gain when every loss is trained on the same network and tuning.
Clever Hans answered arithmetic by tapping a hoof, and experts found no trickery.
In one test he was right 98% of the time when the questioner knew the answer, and 8% when nobody did.
The hypothesis: the questioner, without meaning to, lifted his head slightly at the right count.
Same LLMs, same MMLU questions; only the position of the right answer changes.
RLVR trains an LLM by rewarding answers a checker marks correct.
Random Reward pays half the time, whatever the answer.
Training mostly amplified what this LLM, Qwen2.5-Math-7B, already did.
Without that control, the whole gain would have been credited to the reward.
Same Claude LLM, same harness, same Terminal-Bench 2.0 tasks; only each container’s CPU and memory limits change.
The left axis starts at 20.
Without matched limits, a leaderboard gap under 3 points could be the machine’s doing.
ARC Prize scored GPT-6 Astra on its ARC-AGI-3 puzzle games with two harnesses: its standard one, and one that uses OpenAI’s context features.
Two AI-scientist systems ran machine-learning projects end to end.
Some projects had a flaw that the authors planted or found, such as choosing which experiment to report by its test score.
An LLM auditor had to say whether each project was flawed, from the final paper alone or with the logs and code.
What differed in the case
What we hold equal
Baseline
A simple model won. Once the baseline was tuned, most reported gains disappeared.
Run the simplest method first, and tune the baseline as carefully as the new method.
Cues
The questioner knew the answer, and the horse read him.
Keep whoever asks or grades from knowing the answer, or which version they are scoring.
Setup
Option order, machine limits and harness each moved one LLM’s score.
Keep them the same on both sides. Randomize what cannot be fixed, and report the range.
Scramble
Random rewards gave most of the gain that correct rewards did.
Run the change with its signal scrambled. A gain that survives did not come from the signal.
Agent
An auditor caught far more flaws with the logs and code than with the paper alone.
Read the diff and the logs, not only the summary, to see what else changed.
A. The claim: are we testing the right thing?
B. Scoring: are we grading it right?
C. Controls: is the comparison fair?
D. Noise: could the gap be luck?
Four methods fine-tune the same small reasoning LLM and report gains on AIME’24, a 30-question math benchmark.
Rerun under one setup, each scores below its paper’s number (left). The gray band is the gains that remain; the base LLM’s scores across seeds span more (right).
Without the seed range, a gain this size could be a lucky seed.
Epoch AI asked two agents to invent a better training method. The brief said: “Do not claim gains a single run cannot support.”
Both made several similar runs and reported the best (dark bars). With that pick removed, the gains shrink (middle bars).
An error bar counts each question as separate evidence.
DROP asks several questions per reading passage, and a model that misreads a passage tends to miss them all (correlation; snowballing).
Counting by passage, the error bar for Anthropic’s LLMs on DROP was three times as wide.
Ten patients each took two sleep drugs.
The table gives the hours of sleep each drug added.
As two separate groups, drug 2’s lead is 1.9 standard errors and could be luck. Patient by patient, it is 4.1.
Without pairing, the differences between patients drown the difference between the drugs.
Evals pair the same way: compare two versions question by question.
Darwin grew 15 pairs of plants, one cross-fertilized and one self-fertilized in each.
The crossed plants came out taller.
If crossing did nothing, each pair’s difference could have gone either way.
So Fisher flipped each pair’s result both ways: 15 pairs give 32,768 possible outcomes. In 861 of them (833 + 28, bottom row), the crossed plants lead by at least as much as Darwin saw.
For an eval: on each question, randomly swap the two versions’ scores, many times, and count how often the gap is as large as ours.
Est. MDE is the smallest gain each test set can reliably detect. |Δacc| is the average gain that papers claimed.
What the case showed
What we do
Reruns
Changing only the random seed moved the score as much as the four methods’ gains.
First rerun the baseline and compare it with itself. That gap is the noise. Then report every run.
Groups
Questions about one passage succeed or fail together.
Treat related questions as one group when computing the error bar.
Pairs
Compared patient by patient, a small study gave a clear answer.
Compare question by question. Shuffle the labels to see how often luck alone gives a gap this big.
Size
Claimed gains were often smaller than the test set could detect.
Before the run, work out how many questions are needed to detect the gain.
Agent
Two agents reported the best of several runs as their result.
Fix the number of runs in the brief, and ask for every run.
An exam
A project eval
Claim
A student passes without the skill the course teaches.
The metric goes up, but the project’s real goal does not.
Scoring
The rubric rewards guessing, or the grader marks a right answer wrong.
The metric rewards the wrong behavior, or the grader marks outputs wrongly.
Controls
Two groups of students took the same exam, but one had extra time.
The new version also differed in the data split, seed, harness or settings.
Noise
The exam has only three questions, so one careless slip changes the grade.
The new version wins by a smaller gap than two reruns of the old version show.