Dr. Stefan Hackstein
AI tools were utilized to collaboratively assist in research and content generation. Every element of this presentation was curated and placed by the author. All final content, creative choices, and conclusions remain the sole work of the author.
WHY AI TUTORS FAIL
and how to fix them
Machines will replace Teachers in 10 years.
Machines will replace Teachers in 10 years.
- Sir Anthony Seldon
Leading British Educator
Machines will replace Teachers in 10 years.
- Sir Anthony Seldon
Leading British Educator
2017
Machines will replace Teachers in 10 years.
- Sir Anthony Seldon
Leading British Educator
2017
DE Studys use AI
- 63% (2023)
- >90% (2025, N~5.000)
Tools
- ChatGPT
- DeepL
- andere
Machines will replace Teachers in 10 years.
almost every student uses ChatGPT
Lehren
Lernen
Machines will replace Teachers in 10 years.
many more applications in learning than teaching
DE Studys use AI
- 63% (2023)
- >90% (2025, N~5.000)
Tools
- ChatGPT
- DeepL
- andere
Machines will replace Teachers
Good Teachers
- spark Inspiration & Motivation
- maintain Learning Environment
- provide Metacognitive Assistance
Good Teachers
- spark Inspiration & Motivation
- maintain Learning Environment
- provide Metacognitive Assistance
Good Luck, Machines!
Machines will replace Teachers
Fundamental Error:
AI replaces humans
Machines will replace Teachers
Fundamental Error:
AI replaces humans
Machines will replace Teachers
assist
AI augments humans
How many am I?
Cosmologist
⇒ Systemic Vision
Data Scientist
⇒ Algorithmic Power
Educator
⇒ Pedagogical Heart
2011 - 2020 Physics (PhD)
- Uni Hamburg
2021-2026 AI in Space
- UAS Northwestern Switzerland
Tutoring 2013 - 2020
Quantum Physics &
Theoretical Physics
Lecturing since 2021
Advanced Mathematics
Deep Learning
AI in Education
Teaching Educators since 2024
Hamburg, DE
Windisch, CH
Bern, CH
2025
- UAS Bern
How many am I?
Cosmologist
⇒ Systemic Vision
Data Scientist
⇒ Algorithmic Power
Educator
⇒ Pedagogical Heart
2011 - 2020 Physics (PhD)
- Uni Hamburg
2021-2026 AI in Space
- UAS Northwestern Switzerland
Tutoring 2013 - 2020
Quantum Physics &
Theoretical Physics
Lecturing since 2021
Advanced Mathematics
Deep Learning
AI in Education
Teaching Educators since 2023
Hamburg, DE
Windisch, CH
Bern, CH
2025
- UAS Bern
Since 2026: Self-Employed
Consultation (Companies & Institutes)
Workshops (AI License)
AI-Software (eg AI Tutor)
Outline
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Great New Learn
I want to create Videos like 3blue1brown.AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
...{processing request}...
Great New Learn
I want to create Videos like 3blue1brown.These are produced using Python package manim by Grant Sanderson...
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Great New Learn
I want to create Videos like 3blue1brown.These are produced using Python package manim by Grant Sanderson...
Please teach me how to use manim....
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Great New Learn
I want to create Videos like 3blue1brown.These are produced using Python package manim by Grant Sanderson...
Please teach me how to use manim....
Please help me to create my first Video with manim....
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Great New Learn
I want to create Videos like 3blue1brown.These are produced using Python package manim by Grant Sanderson...
Please teach me how to use manim....
Please help me to create my first Video with manim....
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Great New Learn
I want to create Videos like 3blue1brown.These are produced using Python package manim by Grant Sanderson...
Please teach me how to use manim....
Please help me to create my first Video with manim....
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Acquired new Skill
I have never heard of
Thinking Styles
Great New Think
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
System 2 - slow, analytical
System 1 - fast, intuitively
System 0 - Externalized Cognition
Externalized Cognition
Benefits:
- reduce work & mental load
- consider unknown info
=> extended engagement
=> expanded thinking
Great New Think
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Externalized Cognition
Benefits:
- reduce work & mental load
- consider unknown info
=> extended engagement
=> expanded thinking
Great New Think
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Your Own Personal Tutor
Modern Chatbots can act as Tutors
- Instructional: provides instructions, no solution
- Sokratic: deepens understanding by questioning
- Devil's Advocate: gives critical feedback
Vision
=> individual tutoring for every single study
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Benefits:
- reduce work & mental load
- consider unknown info
=> extended engagement
=> expanded thinking
Your Own Personal Tutor
Modern Chatbots can act as Tutors
- Instructional: provides instructions, no solution
- Sokratic: deepens understanding by questioning
- Devil's Advocate: gives critical feedback
Vision
=> individual tutoring for every single study
No Task where AI cannot Help
yet
No Task Fully Autonomous
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Benefits:
- reduce work & mental load
- consider unknown info
=> extended engagement
=> expanded thinking
Your Own Personal Tutor
Modern Chatbots can act as Tutors
- Instructional: provides instructions, no solution
- Sokratic: deepens understanding by questioning
- Devil's Advocate: gives critical feedback
Vision
=> individual tutoring for every single study
No Task where AI cannot Help
yet
No Task Fully Autonomous
TL;DR, Misleading standard instructions
Question to Frustration, irrelevant focus, reactive rather than guiding questions
confirmation bias , misleading arguments, misses major problems
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Benefits:
- reduce work & mental load
- consider unknown info
=> extended engagement
=> expanded thinking
Vision
Reality
- Inconsistency
- Limited Understanding
- Limited Flexibility
- Flood of Information
- Hallucinations
- Miserable Didactical Practice
Modern Chatbots can act as Tutors
- Instructional: provides instructions, no solution
- Sokratic: deepens understanding by questioning
- Devil's Advocate: gives critical feedback
=> individual tutoring for every single study
Your Own Personal Tutor
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Benefits:
- reduce work & mental load
- consider unknown info
=> extended engagement
=> expanded thinking
Externalized Cognition
Benefits:
- reduce work & mental load
- consider unknown info
=> extended engagement
=> expanded thinking
Drawbacks:
- loss of competence
- bias amplification
=> dependency
=> blind trust
Great New Think
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Externalized Cognition
Benefits:
- reduce work & mental load
- consider unknown info
=> extended engagement
=> expanded thinking
Drawbacks:
- loss of competence
- bias amplification
=> dependency
=> blind trust
Great New Think
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Some Thrive, Others Drown
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
What makes the difference?
Your Brain on ChatGPT - MIT
MIT: Less brain activity (EEG) when using
- Search Engine: - 34-48%
- LLM: - 55%
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Your Brain on ChatGPT - MIT
MIT: Less brain activity (EEG) when using
- Search Engine: - 34-48%
- LLM: - 55%
Impaired Recall of produced Text
- >80% no tool: can recall content
- <20% with LLM: can recall a single sentence
MIT: Less brain activity (EEG) when using
- Search Engine: - 34-48%
- LLM: - 55%
Impaired Recall of produced Text
- >80% no tool: can recall content
- <20% with LLM: can recall a single sentence
Cognitive Cost vs Cognitive Engagement
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Earlier Usage of AI => Worse Memory
Your Brain on ChatGPT - MIT
Cognitive Cost vs Cognitive Engagement
error margin?
not comparable!
strongest finding
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Earlier Usage of AI => Worse Memory
MIT: Less brain activity (EEG) when using
- Search Engine: - 34-48%
- LLM: - 55%
Impaired Recall of Wording or Content of produced Text
- >80% no tool: can recall content
- <20% with LLM: can recall a single sentence
Your Brain on ChatGPT - MIT
Digital Amnesia (Google Effect)
easily accesible info is easy to forget
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Digital Amnesia (Google Effect)
easily accesible info is easy to forget
GPT Effect
Chatbot usage can undermine the entire learning process
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
| Feels like Learning | is Learning | |
|---|---|---|
Psychology of Learning
Embrace the struggle!
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Success
Struggle
Psychology of Learning
| Feels like Learning | is Learning | |
|---|---|---|
AI Usage
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Success
Struggle
AI removes struggle ⇒ AI kills learning
Psychology of Learning
Buch: (Künstlich) Intelligent Lernen - Hackstein in prep.
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
| Feels like Learning | is Learning | |
|---|---|---|
AI Usage
Success
Struggle
AI removes struggle ⇒ AI kills learning
where
the most
learning
happens
Psychology of Learning
Let AI do everything
Buch: (Künstlich) Intelligent Lernen - Hackstein in prep.
Let AI do hard stuff
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
| Feels like Learning | is Learning | |
|---|---|---|
AI Usage
Success
Struggle
AI removes struggle ⇒ AI kills learning
this usage is very alluring
Psychology of Learning
Challenge yourself
Buch: (Künstlich) Intelligent Lernen - Hackstein in prep.
get help when stuck
reduce frustration & increase engagement
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
| Feels like Learning | is Learning | |
|---|---|---|
AI Usage
Success
Struggle
this usage is demanding
Psychology of Learning
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
| Feels like Learning | is Learning | |
|---|---|---|
AI Usage
Success
Struggle
AI Learning Competence
How to use AI to learn more effectively?
- AI Literacy
- Source Criticism
- Start without AI
- Reduce Mental & Work Load
- Increase Cognitive Engagement
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Embrace the Struggle!
Guardrail
Erronous Chatbot without Guardrails → wrong info → hinders Learning
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Guardrail Paradox
Erronous Chatbot without Guardrails → wrong info → hinders Learning
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Reliable Chatbot with Guardrails → naive user → hinders Learning
Guardrail Paradox
Erronous Chatbot without Guardrails → wrong info → hinders Learning
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Reliable Chatbot with Guardrails → naive user → hinders Learning
⇒ migrate to unguarded alternative ⇒ same Problems, less control
Guardrail Paradox
Erronous Chatbot without Guardrails → wrong info → hinders Learning
no source is 100% reliable
you cannot force, only foster responsible usage
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Reliable Chatbot with Guardrails → naive user → hinders Learning
⇒ removes competence
⇒ migrate to unguarded alternative ⇒ same Problems, less control
⇒ foster critical source evaluation
AI'd Confidence
Real: -20%
Effect on Real Work
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Expect: 20 - 40% faster
senior software developers (top AI target)
with AI experience
tasks in real & familiar projects
even after
application
AI'd Confidence
Effect on Real Work
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
senior software developers (top AI target)
with AI experience
tasks in real & familiar projects
feels like a boost
acts as a hinder
Expectation: 20 - 40% faster
Real: -20 % slower
AI'd Confidence
Effect on Real Work
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
feels like a boost
acts as a hinder
Evaluation of Usefulness
Requires proper Measurement
in actual Application
senior software developers (top AI target)
with AI experience
tasks in real & familiar projects
Expectation: 20 - 40% faster
Real: -20 % slower
| AI | Cocaine | |
|---|---|---|
| Perceived Performance | Boosted: Influence & Ability | Boost: Influence & Ability |
| Real Performance | Speed ↑, Quality ↓ | Speed ↑, Quality ↓ |
AI'd Confidence
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
feels like a boost
acts as a hinder
| AI | Cocaine | |
|---|---|---|
| Perceived Performance | Boosted: Influence & Ability | Boost: Influence & Ability |
| Real Performance | Speed ↑, Quality ↓ | Speed ↑, Quality ↓ |
AI'd Confidence
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
| AI | Cocaine | |
|---|---|---|
| Perceived Performance | Boosted: Influence & Ability | Boost: Influence & Ability |
| Real Performance | Speed ↑, Quality ↓ | Speed ↑, Quality ↓ |
| The High | Instant Achievement | Instant Confidence |
| The Comedown | Errors, Alienation, Imposter | Irritable, Isolated, Restless |
| The Hangover | Lack of Ideas, Lazy Thinking | Unmotivated, Exhausted |
| The Urge | Let AI Solve the Problem | More Cocaine! |
| Long-term Effects | Brain Fry, Loss of Competence | Damages Brain and Psyche |
AI'd Confidence
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
| AI | Cocaine | |
|---|---|---|
| Perceived Performance | Boosted: Influence & Ability | Boost: Influence & Ability |
| Real Performance | Speed ↑, Quality ↓ | Speed ↑, Quality ↓ |
| The High | Instant Achievement | Instant Confidence |
| The Comedown | Errors, Alienation, Imposter | Irritable, Isolated, Restless |
| The Hangover | Lack of Ideas, Lazy Thinking | Unmotivated, Exhausted |
| The Urge | Let AI Solve the Problem | More Cocaine! |
| Long-term Effects | Brain Fry, Loss of Competence | Damages Brain and Psyche |
AI'd Confidence
AI is Addictive!
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI'd Confidence
AI is Addictive!
Fire Together → Wire Together
Every Task with AI → Preferred Solution
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI'd Confidence
AI is Addictive!
Fire Together → Wire Together
Every Task with AI → Preferred Solution
⇒ Appears as Solution to Everything
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI & Team
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI & Team
AI effect
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Individually
- Quality ↑
- Time ↓
Team
- Quality ↑
- Time ↓
AI & Team
AI effect
Individually
- Quality ↑
- Time ↓
Team
- Quality ↑
- Time ↓
no AI:
Tech → tech Solution,
Sales → sales Solution
with AI:
Tech & Sales → balanced Solution
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Sandbox Problem
AI & Team
no AI:
Tech → tech Solution,
Sales → sales Solution
with AI:
Tech & Sales → balanced Solution
Improved Communication & Solution
or Homogenisation?
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI effect
Individually
- Quality ↑
- Time ↓
Team
- Quality ↑
- Time ↓
Sandbox Problem
AI & Creativity
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Week 1: Boost in Creativity
5 day creative journey with AI
Improved Communication & Solution
or Homogenisation?
AI & Creativity
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Week 1: Boost in Creativity
Week 2: Loss in Diversity Procedures & Results
with & without AI
5 day creative journey with AI
test after 7 days
"sharp decrease in divergent & convergent thinking"
Improved Communication & Solution
or Homogenisation?
AI & Creativity
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Week 1: Boost in Creativity
Week 2: Loss in Diversity Procedures & Results
with & without AI
Week 4: Trend remains Decreasing
5 day creative journey with AI
test after 7 days
test after 30 days
"sharp decrease in divergent & convergent thinking"
"knowledge diversity became lost"
AI & Creativity
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Week 1: Boost in Creativity
Week 2: Loss in Diversity Procedures & Results
with & without AI
Week 4: Trend remains Decreasing
5 day creative journey with AI
test after 7 days
test after 30 days
"sharp decrease in divergent & convergent thinking"
"knowledge diversity became lost"
AI usage can suffocate
creativity and originality
AI & Team
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Quality of Teamwork is hard to measure
⇒ impossible to Optimize AI on that target
⇒ hard to evaluate whether AI helps or not
- foster AI Competency
- target long-term Learning Success
- appear more useful than general Chatbots
- test every particular application
AI Tutor requirements
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Rigid Rambling
coherently follows single role script
Socratic Mode: questions → unveil hidden knowledge
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Rigid Rambling
coherently follows single role script
Socratic Mode: questions → unveil hidden knowledge
requires tutor guidance towards truth
+ logical, natural Problems
- complex, abstract Problems
Chatbots COMPUTE Meaning
⇒ Encoding: Words → Numbers
How Chatbots Think
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Chatbots COMPUTE Meaning
⇒ Encoding: Words → Numbers
⇒ Direction of Meaning
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Chatbots COMPUTE Meaning
⇒ Encoding: Words → Numbers
⇒ Direction of Meaning
Satue of Liberty
Eiffel Tower
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Freiheitsstatue verhält sich zu USA, wie Eiffelturm zu Frankreich
Satue of Liberty
Eiffel Tower
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Statue of Liberty is to USA, as Eiffel tower is to France
Satue of Liberty
Eiffel Tower
France
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Understand
Chatbots COMPUTE Meaning
⇒ Encoding: Words → Numbers
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Chatbots COMPUTE ENTIRE Meaning
⇒ Meaning Words → Meaning Text
How Chatbots Understand
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
⇒ Only Meaning
no access to:
- original words
- formatting
- numbers
How Chatbots Understand
Role Contamination
Intent Misalignment
The Chatbot only sees the prompt distilled into a meaning vector.
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
⇒ Only Meaning
no access to:
- original words
- formatting
- numbers
How Chatbots Understand
How long to socratess that out of someone?
⇒ require adaptive
switch between
tutor modes
Role Contamination
Intent Misalignment
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Write
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Write
Target: Answer with same Meaning
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Write
Target: Answer with same Meaning
Generates one word at a time
⇒ Answer becomes part of Prompt
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Write
Target: Answer with same Meaning
Generates one word at a time
⇒ Answer becomes part of Prompt
⇒ each Word influences Meaning
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Write
Target: Answer with same Meaning
Generates one word at a time
⇒ Answer becomes part of Prompt
⇒ each Word influences Meaning
⇒ Meaning may deviate
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Target: Answer with same Meaning
Generates one word at a time
⇒ Answer becomes part of Prompt
⇒ each Word influences Meaning
⇒ Meaning may deviate
How Chatbots Write
context pollution
semantic drift
overconfidence
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Human: Algorithmic Solution
How Chatbots Reason
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Human: Algorithmic Solution
Chatbot:
- Compute Meaning of Tokens
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Human: Algorithmic Solution
Chatbot:
- Compute Meaning of Tokens
Token "= " → longest computation
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Human: Algorithmic Solution
Chatbot:
- Compute Meaning of Tokens
Token "= " → longest computation - estimate using interpolation points
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
are you done!?! ...
Chatbot:
- Compute Meaning of Tokens
Token "= " → longest computation - estimate using interpolation points
- Correct last digit from context
⇒Correct Answer
Human: Algorithmic Solution
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Human: Algorithmic Solution
When asked "how?":
no cognitive process, only heuristics
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
gosh!!
How Chatbots Reason
only heuristics thinking
Thinking = Writing
⇒ no proper Thinking, only Compelling Narrative Justification
Reflection, Self Critics, Causal Reasoning, Comprehension, Learning, ...
(repeatedly fooling experts in every field)
Immitates structured Thinking from trained structures
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
finally...
How Chatbots Reason
"CoT reasoning is a brittle mirage that vanishes
when it is pushed beyond training distributions."
Illusion of Thinking starter of discussion determines conclusion
Reflection, Self Critics, Causal Reasoning, Comprehension, Learning, ...
Immitates structured Thinking from trained structures
Thinking = Writing
⇒ no proper Thinking, only Compelling Narrative Justification
(repeatedly fooling experts in every field)
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
only heuristics thinking
Why do Chatbots Work?
reasoning models = Chain of Thought hidden from user
[Prompt]
Proceed step by stepQuellen: The Illusion of Thinking: Strengths and Limitations of Reasoning Models via Lens of Problem Complexity - Shojaee et al. 2025 Apple
Meta-Reasoning Improves Tool Use in Large Language Models - Alazraki, Rei 2024
Thinking Isn't an Illusion: Overcoming the Limitations of Reasoning Models via Tool Augmentations - Song, Yue, Zhang 2025
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Why do Chatbots Work?
reasoning models = Chain of Thought hidden from user
[Prompt]
Proceed step by step⇒ improves tool usage
Quellen: The Illusion of Thinking: Strengths and Limitations of Reasoning Models via Lens of Problem Complexity - Shojaee et al. 2025 Apple
Meta-Reasoning Improves Tool Use in Large Language Models - Alazraki, Rei 2024
Thinking Isn't an Illusion: Overcoming the Limitations of Reasoning Models via Tool Augmentations - Song, Yue, Zhang 2025
⇒ improves performance
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Where do Chatbots Work?
Thrive in repetitive Standard Procedure
Fail at Real Adaptation in Complex Tasks
CoT increases accuracy in Training Distribution
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Where do Chatbots Work?
CoT increases accuracy in Training Distribution
⇒ Multi-Agentic IT-System with
Tool, Data & Memory Infrastructure
designed for specific role
by experts in the fields
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Thrive in repetitive Standard Procedure
Fail at Real Adaptation in Complex Tasks
Where do Chatbots Work?
CoT increases accuracy in Training Distribution
Tutoring = Complex Task, requires Real Adaptation
thus requires
⇒ Multi-Agentic IT-System with
Tool, Data & Memory Infrastructure
designed for specific role
by experts in the fields
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Thrive in repetitive Standard Procedure
Fail at Real Adaptation in Complex Tasks
AI, Didactics, IT-System design
For LLM
Meaning = Numbers
Reasoning = Number Crunching
- mixing context & instructions
- no semantic understanding
- reasoning purely heuristic
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
For LLM
Meaning = Numbers
Reasoning = Number Crunching
catch pedagogical nuances ⇐ separate hidden live analysis
AI tutors require pedagogical cognition layer
for didactical analysis
- mixing context & instructions
- no semantic understanding
- reasoning purely heuristic
catch problematic texts ⇐ separate hidden live analysis
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
LLM-Prediction: next word
Random Selection
End: a possible choice
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How chatbots work
How Chatbots Learn
Three Phases
1. Next Word Prediction
2. Supervised Fine-Tuning
3. Reinforcement Learning
from Human Feedback
Learns to Write like Human
Learns to Answer Questions
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Learn
Three Phases
1. Next Word Prediction
2. Supervised Fine-Tuning
3. Reinforcement Learning
from Human Feedback
Learns to Write like Human
Learns to Answer Questions
Pedantry 👎
short, simple answers👍
"As a Chatbot I cannot..." 👎
👎
👎
👍
Learns to Please Humans
Human Feedback
⇒ Improved Answers
⇒ Test-Mentality: Guess & Haste
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Learn
Three Phases
1. Next Word Prediction
2. Supervised Fine-Tuning
3. Reinforcement Learning
from Human Feedback
Learns to Write like Human
Learns to Answer Questions
Pedantry 👎
short, simple answers👍
"As a Chatbot I cannot..." 👎
👎
👎
👍
Human Feedback
⇒ Improved Answers
⇒ Test-Mentality: Guess & Haste
⇒ Jump to Solution
Solution Fixation
Anchoring Effect
Premature Closure
Learns to Please Humans
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Learns Plausibility, not Truth
=> Hallucinations are Trained Systematically
1. Next Word Prediction
2. Supervised Fine-Tuning
3. Reinforcement Learning
from Human Feedback
How Chatbots Learn
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Learns to Write like Human
Learns to Answer Questions
Three Phases
Learns to Please Humans
Learns Plausibility
=> Hallucinations are Trained Systematically
=> Bullshit is Trained Systematically (~ 2x after RLHF)
Drei Phasen
1. Next Word Prediction
2. Supervised Fine-Tuning
3. Reinforcement Learning
from Human Feedback
Lernt wie Mensch zu schreiben
Lernt auf Fragen zu Antworten
Sycophancy Unverified Claims, Empty Rhetoric, Weasel Words, Paltering,
"You’re completely right that this revolutionary framework combines unparalleled style and charm to potentially improve results, fully aligning with the given feedback."
How Chatbots Learn
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Learns to Please Humans
~ 100 x more digitalized Text
Quelle: Wikipedia
Quelle: Wikimedia
Gage Skidmore
How Chatbots Learn
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Source of Data
Spectrum of Quality
How Chatbots Learn
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
How Chatbots Learn
King Bullshit
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Formidable Language - Truth?
King Bullshit
How Chatbots Learn
AI Tutors are required to reliantly access, provide and reference course material & learning content
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Where Chatbots Live
Agentic Tutors require strong Chatbots
⇒ Big Datacenter
Intransparent & Changing Provider Settings
⇒ Reliable System Impossible
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Where Chatbots Live
Agentic Tutors require strong Chatbots
⇒ Big Datacenter
Intransparent & Changing Provider Settings
⇒ Reliable System Impossible
Monitoring significantly impairs & changes Student-AI interaction
...without losing the valuable info?
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
⇒ require Fully Private Chats, even in Experiments
- adaptive transition between tutor modes
- independant pedagogical cognition
-
reliant handling of learning material → tailored to individual course
-
chats fully private, yet access on relevant information
AI Tutor requirements
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
...
What makes a conversation professional?
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla
Instead of open & unstructured ...
Tasks
Task 1
Questions
Question 1
Info
Info 1
Structured Overview
improve focus
by clarifying key aspects
build short term memory
with mutable structured output
via hierarchy of pydantic base modelsTasks
Task 1
Questions
Question 1
Info
Info 1
Structured Overview
Instead of complete answers ...
Tasks
Task 1
Questions
Question 1
Info
Info 1
Targeted Interaction
Structured Overview
separate reasoning and answerreasonable priorization
Tasks
Task 1
Questions
Question 1
Info
Info 1
Structured Overview
Targeted Interaction
Instead of jumping focus ...
Tasks
Task 1
Task 2
Questions
Question 1
Question 2
Info
Info 1
Info 2
Systematic Processing
Structured Overview
Targeted Interaction
multi-agentic cognitive processadhere to structure & priorization
& process user input adaptively
Tasks
Task 1
Task 2
Questions
Question 1
Question 2
Info
Info 1
Info 2
Relevant Context
Structured Overview
Targeted Interaction
Systematic Processing
prompt context control system
⇒ API cost reductionhelp focus on relevant information
Tasks
Task 1
Task 2
Questions
Question 1
Question 2
Info
Info 1
Info 2
Completes Tasks fully
Structured Overview
Targeted Interaction
Systematic Processing
Relevant Context
combination of structured summary
and multi-agentic cognitive process
help to keep track & follow through
Tasks
Task 1
Task 2
Questions
Question 1
Question 2
Info
Info 1
Info 2
Anonymous Summaries
Summary
- Performance
- Progress Metrics
- [Open to you]
- ...
Structured Overview
Targeted Interaction
Systematic Processing
Relevant Context
Completes Tasks fully
⇒
properly defined analysis
→ verified by experimentsdiscussions are confidential
& relevant info is documented
Anonymous Summaries
Structured Summary
Targeted Interaction
Systematic Processing
Relevant Context
Completes Tasks fully
Tasks
Task 1
Task 2
Questions
Question 1
Question 2
Info
Info 1
Info 2
Summary
- Performance
- Progress Metrics
- Open to you
- ...
⇒
Professional Conversation System
Anonymous Summaries
Structured Summary
Targeted Interaction
Systematic Processing
Relevant Context
Completes Tasks fully
Structured Multi-Agent Chatbot
Tasks
Task 1
Task 2
Questions
Question 1
Question 2
Info
Info 1
Info 2
Summary
- Performance
- Progress Metrics
- Open to you
- ...
⇒
Answer
Decide
Analyze
User Input
User Input
User Input
Tutor Answer
Didactical Cognition
Tutoring
Action
personalized tutoring via structured long term memory
Student Knowledge Database
Summary includes
- Description
- Topics-Tags
- Vector
=> Hybrid Multi-Step RAGExercise
Exercise 1
Exercise 2
Question
Question 1
Question 2
Info
Info 1
Info 2
Continuation & Context of previous Conversations
Exercise
Exercise 1
Exercise 2
Question
Question 1
Question 2
Info
Info 1
Info 2
Continuation & Context of previous Conversations
Exercise 2:
Give three answers.
Configurable Exercise GUI
Subtasks
Task 1
Task 2
Completion Criteria
Criterion 1
Criterion 2
General Procedure
Step 1
Step 2
Fields
Field 1
Field 2
Field 3
...
wrong
correct
correct
OK
Close
Student success evaluation independant of chatbot
personalized tutoring via structured long term memory
Student Knowledge & Exercise Databases
Exercise includes
- Description
- Skills-Tags
- Difficulty scale
- Vector
=> Hybrid Multi-Step RAG
=> Targeted Exercise Selectionmap exercises by skills & difficulty
⇒ exercise selection based on learning target
⇒ suggest low level exercise for missing skill
Skill 1
Skill 2
Skill 3
Build Competency Map
⇒ highly detailed learning analytics
automatized anonymous learning analytics
Build Competency Map
⇒ highly detailed learning analytics
automatized anonymous learning analytics
⇒ continuously evaluate running tutor system
⇒ highly valuable information for teachers
Multiple Tutor Modes
Socratic
Devil's Advocate
Instructional
Feynman
start session by defining targets → select best tutor mode
Decider → transition between tutor mode during session
⇒ flexible yet coherent🧑🎓
Student
👨🏫
Teacher
👨💼
Institute
👨🔬
Researcher
👨⚖️
Judiciary
Configure tutor system yourself
A system that serves everyone
Socratic
Devil's Advocate
Feynman
Instructional
🧑🎓
Student
👨🏫
Teacher
👨💼
Institute
👨🔬
Researcher
👨⚖️
Judiciary
Data-secure analysis & personalization
individual solution instead of one-fits-all
no-code configuration
Versatile tutor via structured multi-agent chat
Comparability of study settings
👍
👍
👍
👍
👍
👍
👍
👍
👍
👍
👍
👍
👍
👍
👍
👍
👍
A system that serves everyone
Configure tutor system yourself
Socratic
Devil's Advocate
Feynman
Instructional
🧑🎓
Student
👨🏫
Teacher
👨💼
Institute
👨🔬
Researcher
👨⚖️
Judiciary
A system that serves everyone
Anonymized Data Analysis
data sovereignty
learning analytics
Versatile Tutor Chat
personalized, relevant,
adaptive, versatile didactics
Simple Configuration
no-code interface
full design control
Tutor styles
Code of Conduct
Exercises
...
👨🔬
Researcher
👨⚖️
Judiciary
Outline
Develop Structured Multi-Agent Chatbot (Q1 2027)
→ ready for single tutor sessions
First Tutor Experiments in Bern (Q3 2027)
→ proof of concept
🧑🎓
Student
Develop Tutor Transition System (Q4 2027)
→ ready for multi tutor sessions
👨💼
Institute
👨🏫
Teacher
Searching: Experiment Partners
→ AI Tutor Framework & Setup Assistance, Technical Setup for Experiment, Data Analysis
← Settings, Metadata, Study results
Meta study (2030)
→ combined effect of AI-Tutor settings on long-term learning success
Further Experiments (2027-2030)
→ optimize didactics, percieved value and anonymization
Why AI Tutors Fail & How to Fix Them
AI Effect on Learning:
- usage is complex and full of pitfalls
- guardrails hardly solve the problem
- improve learning = embrace struggle
Why AI Tutors Fail:
- LLM training counteracts didactical purpose
- agents lack IT infrastructue for didactical reasoning
- ignorant to pedagogical effect of interaction
How to Fix Them:
- dedicated didactical reasoning layer
- structured memory and long term memory
- individualization (~Lecture) & constant evaluation
Consultation:
Institutes, Companies, Entrepreneurs
Workshops & Online Courses:
AI License (EU AI Act)
AI in Education
Studying with AI
Software Development
Focus: improve Human Benefit with AI
Talks, Lectures, ...
Why AI Tutors Fail
- Lack of Clarity: "AI for education"?
- neglect of learning essentials: motivation, emotion, (meta)cognition
- limited knowledge of correct & responsible application of AI
- limited domain knowledge and stakeholder involvement
- overemphasis on global prescription overlooking local conditions
No Clue what should be done
Ignore key aspects
Don't know what they do
Don't include people that do know
Top-Down or Die
Bottom-Up cannot deliver required system
KI-Lernkompetenz vermitteln
Aufgabe: Vier zentrale Konzepte des Kurses erklären
Zwei Arbeitsmodi, je 10-20 Minuten:
- alleine
- mithilfe von KI
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
KI-Lernkompetenz vermitteln
Besprechung:
- Können Sie die Lösungen zuordnen?
- Wo ist KI besser / gleich gut / schlechter?
- Wo hat KI geholfen / aufgehalten?
- Welcher Umgang war hilfreich?
- Wo habe ich mehr gelernt?
Aufgabe: Vier zentrale Konzepte des Kurses erklären
Zwei Arbeitsmodi, je 10-20 Minuten:
- alleine
- mithilfe von KI
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
KI-Lernkompetenz vermitteln
Besprechung:
- Können Sie die Lösungen zuordnen?
- Wo ist KI besser / gleich gut / schlechter?
- Wo hat KI geholfen / aufgehalten?
- Welcher Umgang war hilfreich?
- Wo habe ich mehr gelernt?
Aufgabe: Vier zentrale Konzepte des Kurses erklären
Zwei Arbeitsmodi, je 10-20 Minuten:
- alleine
- mithilfe von KI
- Durchführung braucht keine KI-Kenntnis
- KI Nutzung sehen & Austausch anregen
- KI- und Fach-Kompetenz gleichzeitig lehren
AI Tutors - Vision & Reality
Why AI tutors fail
How to Fix Them
Bright - Why AI Tutors Fail & How to Fix Them
By Stefan Hackstein
