How long can your skills be
before your agent
forgets what you told it?
Who is this guy?
Laurie Voss, Head of DevRel at Arize
This is all Dexter Horthy's fault
200 instructions is not very many
You already know this feeling
Here's what you'll
walk out knowing
The number came from a benchmark called IFScale
IFScale — Jaroslawicz et al. (2025)
The test: write a report
using N exact words
- "Include the exact word: 'customer'."
- "Include the exact word: 'revenue'."
Two metrics:
density and accuracy
- density (N) = how many rules at once
- accuracy = % of rules actually followed
Why arbitrary words
tell us about real skills
First:
does the old finding still hold?
The 2025 models do
exactly what the paper said
Then we pointed it at
the 2026 models
Same prompt, same words, same test
The problem:
they wouldn't break
So we made it harder.
And harder. And harder.
- 500 words → perfect scores
- 1,000 → mostly perfect scores
- 2,000 → still mostly perfect scores
- Eventually: a 10,000-word vocabulary
An order of magnitude better
in one year
Log scale!
This is not a toy benchmark,
this is a test
of a thing that matters.
And it's still moving as we speak
Opus 4.8, Fable, GPT 5.6 released since this ran
How they fail
is the interesting part
They don't just "forget" anymore
DeepSeek V4 Pro:
the old failure
drops instructions at ~750,
half gone by 2,000
Claude Opus 4.7:
thinks the test is dangerous
Instead of forgetting, it refuses
Claude is scared of
giving dangerous advice
"anthrax" + "cyanide" = refusal
Gemini 3.1 Pro:
overthinks
spends all its tokens on thinking,
emits no visible tokens
GPT-5.5:
thinks the test is stupid
- "I'm sorry, but the requested report cannot be produced in full within the practical response limits of this interface because it requires incorporating 4,000 exact terms while also maintaining a coherent professional business-report structure."
GPT-5.5 has a point
Four models, four personalities
neutral · cautious · overthinking · sarcastic
So what does this mean
for your skills files?
One: skills files don't have a compression problem anymore
old playbook: keep it under 200 instructions, fan out to sub-skills
Two: your prompts can be extremely detailed
hundreds of constraints in one prompt is fine now
Two thousand instructions
is a lot
Three: the wall became a trade-off
"can it do this?" → "is it worth the cost?"
Some caveats
Evidence is not the same as proof
Remembering your instructions is not the same as following them
Chroma's "context rot," 2025
Silent failures are
the hardest to deal with
Claude refuses loudly. GPT refuses quietly.
What did all this cost?
2,345 calls · $209.19
Evals will help
You knew I was going to mention evals eventually
Newer findings
- "Revisiting the Reliability of Language Models in Instruction-Following," 2026
We were early;
the field is catching up
FireBench · CCR-Bench · GuideBench
Your job has changed from compression to verification
Writing skills files is different in 2026
Thank you!
The code and data:
github.com/Arize-ai/instruction-budget
Follow me on BlueSky!
@seldo.com 🦋
Come to our world cup watch party today at 5pm!