Ahcène Boubekki
Line Clemmensen
UCPH/Pioneer
UCPH/DTU
Safety in Mental Health Chatbots
technical considerations
ELIZA, etc.
ELIZA
ELIZA (Weizenbaum, MIT, 1966).
Rule-based chatbot
Manually updatable
Different scripts (DOCTOR, GIRL, etc.)
Computational Therapy?
Risks?
S. Ciston, et al., Inventing ELIZA: How the First Chatbot Shaped the Future of AI, The MIT Press (2026)
ELIZA
Guardrails, Refusals, and CHANGE
If misunderstanding, enter the NONE rule:
S. Ciston, et al., Inventing ELIZA: How the First Chatbot Shaped the Future of AI, The MIT Press (2026)
Does not break the illusion
Can be controlled using rules to avoid topics
ELIZA
Weizenbaum's warnings
Current Regulation
Machine passing as human
Exploiting vulnerable users
Sharing sensitive data
Machine therapy
Crisis and risks
Human judgment
Weizenbaum's warnings
[specialists in human to human interaction] view the simplest mechanical parody of a single interviewing technique as having captured anything of the essence of a human encounter...
S. Ciston, et al., Inventing ELIZA: How the First Chatbot Shaped the Future of AI, The MIT Press (2026)
EU AI Act, etc.
EU AI Act
NEDA's Tessa chatbot (2023)
An eating disorders chatbot offered dieting advice.
Parents sue OpenAI over ChatGPT's role in son's suicide
Raine v. OpenAI (2025, ongoing)
Character.AI and Google (2026)
Character.AI and Google agree to settle lawsuits over teen mental health harms and suicides.
Instagram@neda, 16.07.2022
Washington Post, 27.12.2025
The Guardian, 08.01.2026
EU AI Act
New Legislative Framework
1. Law
2. Standards
Already regulated domains (Annex I)
8 high-risk domains (Annex III)
High-risk vs the Rest
In-effect Requirements
A5: Prohibited practices (manipulation...)
A4: AI Literacy
GPAI obligations (testing, report...)
A50: transparency (inform, watermark...)
EU AI Act
Weizenbaum's warnings
Machine passing as human
Exploiting vulnerable users
Sharing sensitive data
Machine therapy
Crisis and risks
Human judgment
Current Regulation
Art. 50(1): inform user of AI
Art. 5(1): Ban manipulation, etc.
GDPR Art. 9
EU MDR: software with medical purpose is medical device (from 2028)
None in EU AI Act
SB 243: prevent suicidal ideation
Art. 14: human oversight of
high-risk systems
LLM, etc.
LLM
Query
Main Agent
RAG
Memory
Planner
Censor
Output
Web
Tools
Scripts
etc.
RAG
RAG
LLM: Hallucinations
Mitigation strategies:
Reward abstention during training.
Threshold on confidence.
Whitelist of resources.
Exclude outputs from memory.
Ensemble of censors.
Hard-coded fallback/exit.
Diagnostic:
Models are trained and evaluated that reward confident guessing over admitting uncertainty.
(Kalai et al. 2025)
LLM: Sycophancy
Mitigation strategies:
No persona.
Sycophancy benchmarks.
No training on engagement.
Session limits, breaks, etc.
Limit memory of emotions.
Diagnostic:
Sycophancy is learned from us: models are trained on human preferences, and humans tend to prefer answers that agree with them. (Sharma et al., 2023).
Ahcène Boubekki
Line Clemmensen
UCPH/Pioneer
UCPH/DTU
Safety in Mental Health Chatbots
technical considerations
D3A: Safety in Mental Health Chatbots
By ahcene
