How Do They Do What They Do?
Work in Progress
Dan Ryan
March 2026
A just so story about how intellectual work gets done.
Let's imagine there is a community of collaborative scholars,
each with their own specialty and expertise.
One member, the principal investigator (PI), conjures up
a project and writes a brief document describing the
what and the why and the how.
Together they will write this grant!
The PI solicits feedback from four members of the community, each with their own specialized wisdom.
"We want to investigate whether ancient agricultural practices contain embedded knowledge about soil ecology that modern science has overlooked."
Specialist Annotations
Specialist Annotations
...
This process is repeated across many panels of specialists. Each panel member sees the annotated text passed from the previous panel.
"I want to investigate whether ancient agricultural practices contain embedded knowledge about soil ecology that modern science has overlooked."
"I want to investigate* whether ancient* agricultural practices*
contain embedded*
knowledge* about soil ecology*
that modern* science has overlooked*."
"I want to investigate* whether ancient* agricultural practices*
contain embedded*
know^ledge* about soil+ecology*
that modern* science has overlooked*."
Specialist Annotations
Specialist Annotations
...
BEST NEXT WORD
After all the panels have contributed to the annotation of the original document the PI selects a single best next word to append to the document.
Absolutely
BEST NEXT WORD
APPEND
=<STOP>?
RESPOND
=<STOP>?
YES
NO
The extended document is then fed into the system again and the process is repeated until the best next word is determined to be <STOP>.
"I want to investigate whether ancient agricultural practices contain embedded knowledge about soil ecology that modern science has overlooked."
Absolutely
Absolutely
Absolutely! Investigating ancient agricultural practices for overlooked ecological insights is a fascinating research direction. One possible way to frame this is through the lens of mycorrhizal networks, which traditional knowledge systems often preserved through polyculture and minimal soil disturbance.
1. The Subterranean Network. Ancient "slow" agriculture often relied on the symbiotic relationship between plant roots and fungi. These common mycelial networks function as a biological "internet," redistributing nutrients and water across diverse crop species based on environmental signals.
2. Modern Disruption. Modern industrial methods, particularly deep tilling and heavy synthetic fertilization, often "mute" the chemical signaling (such as strigolactones) that plants use to recruit these fungal partners. This effectively fragments the ancient, living network that sustained soil health for millennia.
3. Research Focus. A key investigation could explore how modern science might "re-connect" these fragmented systems by studying the resilient, high-density fungal architectures found in surviving traditional agricultural sites. I can provide more specific case studies on these fungal social networks if you’d like.<STOP>
BEST NEXT WORD
APPEND
=<STOP>?
RESPOND
=<STOP>?
YES
NO
But isn't it just a "stochastic parrot"?
BEST NEXT WORD
"I want to investigate whether ancient agricultural practices contain embedded knowledge about soil ecology that modern science has overlooked."
Physicist notes "complexity, emergence; distributed knowledge over generations"
Legal expert: not much to say
Historian notes "revisionist approach to history of science; methodological debates"
Humanist notes "tension in the framing; overlooked suggests deficit narrative; ancient/modern binary"
"I want to investigate*
whether ancient*
agricultural practices*
contain embedded*
knowledge* about soil
ecology* that modern*
science has overlooked*."
Physicist notes "complexity, emergence; distributed knowledge over generations"
The original document (prompt)
Specialists notice different things and add annotations.
The annotated document (each word slightly "accented" by what the specialists saw)
Panel #1
BEST NEXT WORD
ETC.
Philosopher of science notes "emergence" and "revisionism" and thinks "Kuhn, Polyani"
Historian: "revisionist" plus "binary" suggests "post-colonial studies"
Biologist likes "distributed knowledge" and "soil networks" and thinks "mycorrihizal networks!"
Ancestral wisdom expert thinks "embedded knowledge needs unpacking"
"I want to investigate*
whether ancient*
agricultural practices*
contain embedded*
knowledge* about soil
ecology* that modern*
science has overlooked*."
Panel #2
BEST NEXT WORD
"I want to investigate whether ancient agricultural practices contain embedded knowledge about soil ecology that modern science has overlooked."
Biologist likes "distributed knowledge" and "soil" and adds "reminds me of mycorrihizal networks!"
"I want to investigate* whether ancient* agricultural practices* contain embedded* knowledge* about soil ecology* that modern* science has overlooked*."
Physicist adds accents of "complexity, emergence; distributed knowledge over generations"
abstract claim
→
concrete example
We can see recurrent patterns in how specialists in layer 2 react to annotations of specialists in layer 1.
These are interpretive circuits.
The second specialist's annotation tweaks the document in the direction of mentioning mushrooms.
BEST NEXT WORD
The panelists appear to be "cooperating" but they only "talk" through the document.
How do they learn to do this?
They develop these reflexes through a "Calculus of Blame."
Professor X gets advice on a grant proposal from valued colleagues.
The Calculus of Blame
The grant is rejected. What went wrong?
Parse the referee reports. Think about the advice I got. How the score might be different if I'd taken Eva's advice a bit more to heart? Or if I had relied less on Ben's suggestions? How much would the outcome have changed in response to small changes in how I weighed the advice of each of my colleagues?
When I share the news, my colleagues do the same thing, questioning the rubrics that informed their advice to me. Which considerations steered them away from giving me better advice? They figure out how to tweak their rubrics so as to give better advice next time.
The panels are the layers of a transformer model.
The specialists are attention heads.
By training only on "next word prediction" they learn rubrics that allow them to participate in circuits that extract and generate semantic meaning that the transformer turns into the text output we see.
THE END
"I want to investigate whether ancient agricultural practices contain embedded knowledge about soil ecology that modern science has overlooked."
investigate
Each word is represented by a set of semantic weights called an embedding vector.
| +0.92 |
| +0.43 |
| +1.04 |
| +0.85 |
| -0.76 |
| +0.97 |
| 0.68 |
| 0.59 |
| +0.71 |
| +0.31 |
| +0.81 |
| +1.0 |
Agency. Is this an action performed by a conscious actor?
Temporality. Does it imply a long duration (ancient) vs. a short one?
Epistemology. Does it relate to the seeking or creation of knowledge?
Formality. Is the register academic/professional vs. casual?
Physicality. Does it involve manual labor vs. intellectual labor?
Scrutiny. Does it imply looking "closely" or in high detail?
Uncertainty. Does the word imply we don't know the answer yet?
CollaborationIs this typically a solitary or group activity?
Scientificity. Does it map to the "hard sciences" or "physics" subspace?
Historicity. Does it map to the "humanities" or "historical" subspace?
Complexity. Does the action involve many moving parts or "emergence"?
Intentionality. Is the action purposeful and directed toward a goal?
In reality there are many more components (on the order of 1000) and they are not as humanly interpretable as shown here.
"I want to investigate whether ancient agricultural practices contain embedded knowledge about soil ecology that modern science has overlooked."
...
investigate
overlooked
to
I
want
So, the input document (prompt) is a stack of embedding vectors.
...
investigate
overlooked
to
I
want
...
...
embedding dimension
context window
embedding dimension
context window
The specialist's rubric specifies which semantic dimensions it pays attention to for each position in the input.
DOCUMENT
SPECIALIST'S "RUBRIC"
In reality, the specialist (head) rubric has three parts
...
...
...
For each position in the input, what semantic dimensions do I care about?
How do I weigh what I find in various dimensions of other words to inform what I care about?
How do I compute the value of what I find?
"QUERY"
"KEY"
"VALUE"
Interpretability = What's Going On Inside
or
How is the LLM doing that?!
I
Absolutely, I'd love to be your conceptual tutor...
want
to
learn
about
...
intrepretability
How Do They Do What They Do
By Dan Ryan
