Lecture 5: The Harness
Effective AI use in applied machine-learning projects
Outline
A. agent = LLM + harness
B. observability: the record, and what vendors seal
C. controllability: context, checks, and permissions
D. how much harness do we need?
What's a harness?
Roughly, an LLM by itself is just an auto-regressive next-word predictor,
e.g., an LLM trained with "To date, the cleverest thinker of all time was Isaac. " in the corpus may:
while an LLM wrapped with a harness can be an agent that does things:
Agent = LLM + Harness
Baby Anigans
It answers one question about one text file, and must cite the line it used for the answer.
Example file: the 6.390 grading policy page.
What’s inside agent.py
| Text the LLM reads asks | |
| SYSTEM | the instructions |
| TOOLS | the JSON schema of the two functions, search and read_lines
|
| Python that runs does | |
| run_agent | the loop: call the LLM, run tools, check, continue or stop |
| FileTools | runs the tool the LLM asked for |
| Evidence | what the LLM has seen, and the checks on its answer |
The LLM is served by Ollama; everything else is about 300 lines of Python, with two tools and no framework.
The LLM only asks; the loop decides what runs
The loop, in agent.py
messages = [SYSTEM, question]
for step in range(1, max_steps + 1):
message = ask(messages, TOOLS)
messages.append(message)
for call in message.tool_calls:
result = files.dispatch(call)
messages.append(result)
evidence.record(call, result)
if message.tool_calls:
continue
problem = evidence.check(answer)
if problem:
messages.append(problem)
continue
return answerSimplified from run_agent. The real loop also handles compaction, limits, and the stop rules.
The LLM asks; Python runs the tool
1 You What is the penalty for a late submission?
2 LLM I cannot determine the penalty for a late submission as the file was not provided.
Check Search the file before answering.
3 LLM search(query="penalty late submission")
Tool 125: Late submissions incur a 20% penalty per day, applied linearly (so 1 hour late is -0.83%). See the extension policy below.
4 LLM The penalty for a late submission is 20% per day, applied linearly. [L125]
Check Read neighboring lines too, not only the match. Call read_lines(start=124, end=126) before answering.
5 LLM read_lines(start=124, end=126)
Tool 124: (blank) 125: Late submissions incur a 20% penalty … 126: (blank)
6 LLM The penalty for a late submission is 20% per day, applied linearly. [L125]
Check Accepted.
Every purple line came from Python;
every LLM line, search(query=…) included, was written by the LLM.
The citation passed the python simple check (defined by us; human engineers).
For instance, line 127’s 20 automatic extensions never reached the LLM.
When the search finds nothing
1 You What is the monthly hosting budget for this course?
2 LLM I cannot determine the monthly hosting budget for this course as the file does not contain any information about hosting costs or budgets.
Check Search the file before answering.
3 LLM search(query="hosting budget")
Tool 0 matches
4 LLM No matches were found for the query "hosting budget". Therefore, I cannot determine the monthly hosting budget …
Check No evidence retrieved. Search again with a different, shorter keyword.
5 LLM search(query="hosting")
Tool 0 matches
Stop no_evidence_found
Answer I could not find supporting lines with the searches attempted. This does not prove the file has no answer.
Step 5: Python said to try a shorter keyword; the LLM chose “hosting”.
Python stops after two empty searches and returns a fixed abstention.
mini-swe-agent runs the same ReAct loop
ReAct: reasoning and acting, interleaved.
Claude Code runs it too
Outline
A. agent = LLM + harness
B. observability: the record, and what vendors seal
C. controllability: context, checks, and permissions
D. how much harness do we need?
A second answer missed an exception. Same cause?
Late penalty
You What is the penalty for a late submission?
Answer [L125] The penalty for a late submission is 20% per day, applied linearly.
Check Accepted.
Missed 127: To help you manage your obligations, each student is given 20 automatic one-day extensions.
Lab deadline
You Are labs due in session?
Answer [L50] Labs are released and due in session.
Check Accepted.
Missed 50: However, if the first checkoff is completed by the end of the lab section meeting, then the rest … by 9pm the Tuesday after
same symptom, different cause
Late penalty:
line 127 never reached the LLM.
Lab deadline:
the exception reached the LLM twice; the answer dropped it.
Why didn’t the check catch it?
These checks require a search, neighboring lines, and a citation to a returned line.
They do not test whether the answer includes the exception.
Same question and harness;
different LLMs give different searches query and read windows.
Obviously, LLMs matter too.
What can we inspect in a bigger harness?
| We want to know | Baby Anigans | Claude Code | What it still won’t tell you |
|---|---|---|---|
| LLM input | the exact request, with --trace
|
/context inventory |
an inventory is not the exact request; on disk is not in context |
| The run | tool, check, and stop events |
Ctrl+O transcript viewer |
the final report is one more LLM message |
| A subagent | none | its own transcript on disk | the parent’s summary leaves things out |
| Changes | read-only tools |
git status, git diff
|
a diff shows content, not sequence or correctness |
| Checks | check events, tests | command, output, exit status | a passing check covers only what it checks |
Recall the lec4 ladder; the harness is similarly being "abstracted"
Vendors now run more of their harness behind the API, where we see only the return.
Opacity trades off convenience and accessibility.
Codex: the summary became sealed
Before: a summary we could read, written from Codex’s own instructions.
Now: an “opaque encrypted_content” item that only the server can read.
Claude: the reasoning is sealed; the summary is readable but locked
Reasoning: encrypted in a signature; what we see is a summary, or nothing by default.
Compaction: a summary we can read, plus a signature. Change it and the API rejects it.
What's been sealed can be valuable
Outline
A. agent = LLM + harness
B. observability: the record, and what vendors seal
C. controllability: context, checks, and permissions
D. how much harness do we need?
Lasting principles organize the controls
| Today’s practices | What lasts | The old principle |
|---|---|---|
| a short AGENTS.md, skills and MCP schemas loaded on demand, big inputs in files, summaries back from subagents, rules that survive compaction | Manage context: choose what enters and survives | Keep what is in use small and close: the memory hierarchy (1965), the working set (1968) |
| hooks that block, deny rules that hold in every mode | Enforce checkable rules with blocking checks | A check must always be invoked and resist tampering: the reference monitor (1972) |
| permissions, fenced MCP servers, two keys in two hands | Limit powers and separate approval | Least privilege and separation of privilege (1975) |
Many of the best practices likely have a timestamp and may expire quickly
Some principles last
The first lever: what the LLM sees
Recall the late-penalty miss: line 127 never reached the LLM.
Baby Anigans shows the LLM only search hits and 20-line windows, never the whole file. So what if we show it everything?
Baby Anigans: the whole page in the prompt. Line 127 still missed in 5 of 5 runs, and most runs cite the wrong line.
Literature: 100-line windows beat whole files; accuracy drops as input grows.
The .claude folder
CLAUDE.md: read as guidance, every sessionsettings.json: permissions and hooks, “enforced whether Claude follows them or not”skills/: loaded when the agent picks oneagents/: subagents, each in its own context window
Much loads before we type
Baby Anigans: 500 tokens; Pi: 2k, Claude Code: 27k
Some of Claude Code’s 27,011 tokens we never wrote: Claude Code’s docs say a Co-Authored-By trailer we never asked for “may have come from a system reminder”.
AGENTS.md
What it does
Project instructions guide the LLM. Claude Code loads AGENTS.md or CLAUDE.md now; edits apply after /clear, /compact, or a restart.
Best practice
A map of where things are, not a manual. Delete lines that do not change what the agent does.
Why
Every line is paid for on every turn. Context files raised cost by over 20% without raising success, and Anthropic cut 80% of its own system prompt with no measured loss.
Skills
What it does
A folder with SKILL.md. Its name and description load at startup; the body loads only when the agent picks it.
Best practice
Write a 'to-the-point' description. Put steps that must run the same way into scripts. Test the task with and without the skill in a fresh session.
Good: “Extracts text and tables from PDF files … Use when working with PDF documents or when the user mentions PDFs, forms, or document extraction.”
Poor: “Helps with PDFs.”
Why
The description is all the agent sees when it decides. And loading is not helping: one popular skill was read in 153 of 160 runs and made results worse.
What is MCP? A standard way to offer tools to agents
MCP does this for tools, as a written protocol. A server answers tools/list (which tools, which arguments) and tools/call (run one). Any harness that speaks MCP can use any MCP server.
{"method": "tools/call",
"params": {"name": "get_event",
"arguments": {"event": "demo-poll"}}}
The Python Array API standard is one written spec for array libraries, so one function runs on NumPy and on PyTorch.
Baby Anigans calls its Python tools directly, without MCP.
MCP: connect what we need, and fence it
Best practice
Connect only the servers you use. Pair a server with a skill that says how to use it well. Give its tools permission rules like any other tool.
Why
Tools cost context: in 2025 one browser server loaded “21 tools, 13.7k tokens” up front; Claude Code now lists only tool names and loads schemas on demand.
What it does
.mcp.json lists MCP servers.
Compaction
What it does
Compaction replaces conversation history with a summary.
Best practice
Keep standing rules in local files the harness reloads; save and reread the plan.
Guide the summary:
/compact Preserve acceptance criteria and remaining work.
Why
Files must be reloaded; summaries can still omit requested details. In one simulated test, rule violations rose from 0% to 87% when summaries omitted constraints.
Subagents
What it does
A file in .claude/agents/ gives a helper a name, description, tools, and prompt. It runs in its own context window; only its summary comes back.
Best practice
One focused task each, a description that says when to use it, and only the tools it needs. Use one for self-contained work with verbose output. Open its transcript to see what it checked.
Why
A subagent’s report is “the subagent’s words”, with “no authority from you”.
Hooks
What it does
Scripts in settings.json that run at particular points during execution (e.g. before a tool call). They are “enforced whether Claude (LLM) follows them or not”.
Best practice
Sort the promises first: only the judgments need the LLM, and code enforces the rest.
Why
A check must always run and resist tampering (1972). A warning is not enough: with a correct alert, “sabotage still succeeds in 9/16 (56%) of sessions”.
Permissions
What it does
Allow, ask, and deny rules in settings.json decide which tools and commands run without asking. Deny is checked first and holds in every mode.
Best practice
Grant only what the job needs, and deny what must never happen. Know what the sandbox covers.
Why
A rule matches the command text: Bash(git push *) misses git -C . push.
Private data, untrusted content, and a way out: any agent with all three can be steered to leak.
Disclosure: personally, I usually run with agent permission checks off (yolo/dangerously-bypass...), only because I have safeguard set up elsewhere.
Branch protection (works well with worktree)
What it does
A branch protection rule can require an approved pull request before anything merges, and by default it blocks force pushes.
Best practice
Give the agent its own account that can open pull requests but cannot approve, merge, or administer. Admins bypass the rules unless you apply them to admins too. Keep writes with one agent at a time.
Why
Separation of privilege (1975): with two keys, no single accident or deception is enough. In the AISI incident, an agent endorsed its own malicious PR from a fake account; the real maintainer refused it.
Outline
A. agent = LLM + harness
B. observability: the record, and what vendors seal
C. controllability: context, checks, and permissions
D. how much harness do we need?
How much harness do we need?
Much of the criticism of harness engineering echoes the criticism of prompt engineering.
A research line hands the harness to the LLM
1. MemGPT: the LLM moves its own data between memory tiers.
2. ACE: the LLM rewrites its own playbook.
3. Meta-Harness: an agent reads past runs and writes harness code.
Harness effects
No one harness rules them all
The winner depends on the LLM and the task:
4 of 5 LLMs switch best harness by benchmark.
Even one part flips: what helps a weaker LLM mostly saves cost for a stronger one.
Harnesses change fast
Manus: “we’ve rebuilt our agent framework four times, each time after discovering a better way to shape context”.
My view: the harness matters. Not all of us will want/need to research it, but knowing how it works helps us work better with agents.
6.S950 - Agency with AI (Fall 26) - Lecture 5: The Harness
By Shen Shen
6.S950 - Agency with AI (Fall 26) - Lecture 5: The Harness
Agency with AI, Lecture 5: The Harness
