All Student Models are Wrong
But Some are Useful
Shayan Doroudi
December 3, 2018
University of California, Irvine
Student
Model
Data
Instructional Policy
Activity
Response
Millions of students
Model-Based Instructional Sequencing
Over 500,000 students/year
25 million active monthly users
~12 million active monthly users
Student
Model
Data
Model-Based Instructional Sequencing in 1960s
Photos from suppes-corpus.stanford.edu
“We assume that a mathematical model of learning will provide an approximate description of the student's learning, and the task for a theory of instruction is then to settle the question of how the instructional sequence of concepts, skills, and facts should be organized to optimize for a given student his rate of learning.”
Suppes (1974)
The Place of Theory in Educational Research
AERA Presidential Address
“It would be my prediction that we will see increasingly sophisticated theories of instruction in the near future.”
Student
Model
Data
Instructional Policy
We haven’t seen “increasingly sophisticated theories of instruction”
Student
Model
Data
Instructional Policy
“All Models are Wrong
But Some are Useful”
George Box, 1979
Statistical Models of Student Learning
≠
How Students Learn
2. Relying on a wrong model can have adverse consequences for students.
1. Undesirable behaviors of student models can be explained by using a wrong model.
I will demonstrate...
Three Consequences of a Wrong Model
Misguided Inferences about Learning
Misguided Instructional Decisions
Inequitable Outcomes
All Student Models are Wrong
But Some are Useful
Data-Driven Models + Learning Theory
Misguided Instructional Decisions
Inequitable Outcomes
Misguided Inferences about Learning
Robustness to How Students Learn
Student Models
Bayesian Knowledge Tracing (BKT)
Corbett and Anderson, 1994
Bayesian Knowledge Tracing (BKT)
Data
Student A:
Student B:
Student C:
...
Bayesian Knowledge Tracing (BKT)
Data
Student A:
Student B:
Student C:
...
Machine Learning
Bayesian Knowledge Tracing (BKT)
Addition
Subtraction
Multiplication
Mastery Learning
Keep giving practice opportunities on a skill/concept until student reaches mastery:
P(\text{Learned}) > 0.95
Then move onto the next skill/concept
Corbett and Anderson, 1994
Data
Mastery
Learning
Rosen et al., 2018
Ritter et al., 2007
Corbett and Anderson, 1994
Additive Factor Model (AFM)
Cen, 2009
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - \beta + \gamma t))}
\(\theta\) - Student Ability \(\sim \mathcal{N}(0, 1)\)
\(\beta\) - Item Difficulty
\(\gamma\) - Learning Rate
P(Correct)
\theta - \beta + \gamma t
All Student Models are Wrong
But Some are Useful
Data-Driven Models + Learning Theory
Misguided Instructional Decisions
Inequitable Outcomes
Misguided Inferences about Learning
Robustness to How Students Learn
Semantics of a BKT Model
75% of skills in a middle school mathematics tutoring system had P(guess) > 0.5 or P(slip) > 0.5
Baker, Corbett, and Aleven, 2008:
High guess and slip parameters are the result of BKT being unidentifiable.
Beck and Chang, 2007:
BKT Model actually is identifiable.
Semantics of a BKT Model
Doroudi and Brunskill, 2017:
High guess and slip parameters could be due to
fitting the wrong model.
Doroudi and Brunskill, Educational Data Mining 2017, Best Paper Nominee
Data
“Analysis of specification error relates to a rhetorical strategy in which we
suggest a model as the “true” one for sake of argument,
determine how our working model differs from it
and what the consequences of the difference(s) are,
and thereby get some sense of
how important the mistakes we will inevitably make may be.”
Otis Dudley Duncan, 1975
Semantics of a Wrong Model
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + 0.1i))}
\theta \sim \mathcal{N}(0, 1)
P(Correct)
\theta - \beta + \gamma t
Doroudi and Brunskill, Educational Data Mining 2017, Best Paper Nominee
Semantics of a Wrong Model
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + 0.1i))}
\theta \sim \mathcal{N}(0, 1)
500 students
20 practice opportunities
High P(slip)!
P(Correct)
\theta - \beta + \gamma t
Doroudi and Brunskill, Educational Data Mining 2017, Best Paper Nominee
Not
Learned
Learned
Semantics of a Wrong Model
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + 0.1i))}
100 students
200 practice opportunities
\theta \sim \mathcal{N}(0, 1)
High P(guess)!
Doroudi and Brunskill, Educational Data Mining 2017, Best Paper Nominee
P(Correct)
\theta - \beta + \gamma t
Not
Learned
Learned
Takeaway Message
Researcher
Incorrect inference
(e.g., throw out questions)
Data
Relying on the semantics of a wrong model can lead to incorrect inferences about student learning.
Takeaway Message
All Student Models are Wrong
But Some are Useful
Data-Driven Models + Learning Theory
Misguided Instructional Decisions
Inequitable Outcomes
Misguided Inferences about Learning
Robustness to How Students Learn
Data
Mastery
Learning
Misguided Notion of Mastery
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + 0.1i))}
\theta \sim \mathcal{N}(0, 1)
Average P(Correct)
at Mastery:
0.54
P(Correct)
\theta - \beta + \gamma t
Mastery
Learning
Declare
Mastery
Data
Declare mastery early
Takeaway Message
Could have cascading effects that could impede on future learning.
Data
New Instructional Policy
Going Beyond Mastery Learning
New
Model
Fractions
Tutor
Fractions
Tutor
Around 1000 students
Data
New Instructional Policy
Experiment
New
Model
Fractions
Tutor
Fractions
Tutor
Fractions
Tutor
vs.
Baseline Policy
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Simulated Experiment
New
Model
New
Model
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Data
New Instructional Policy
New
Model
Fractions
Tutor
vs.
Baseline Policy
| Baseline Policy |
Adaptive Policy | |
|---|---|---|
| Simulated Results | 5.9 ± 0.9 | 9.1 ± 0.8 |
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Posttest Scores (out of 16 points)
Simulation Results
| Baseline Policy |
Adaptive Policy | |
|---|---|---|
| Simulated Results | 5.9 ± 0.9 | 9.1 ± 0.8 |
| Experimental Results | 5.5 ± 2.6 | 4.9 ± 1.8 |
Posttest Scores (out of 16 points)
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Experiment Results
Single Model Simulation
Fitted
Model
Fitted
Model
Chi et al., 2011
Rowe et al., 2014
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Instructional
Policy
Robust Simulation
Fitted
Model
Instructional
Policy
“True”
Model
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Robust Evaluation
| Baseline Policy |
Adaptive Policy | |
|---|---|---|
| New Model | 5.9 ± 0.9 | 9.1 ± 0.8 |
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Posttest Scores (out of 16 points)
Robust Evaluation
| Baseline Policy |
Adaptive Policy | |
|---|---|---|
| New Model | 5.9 ± 0.9 | 9.1 ± 0.8 |
| Bayesian Knowledge Tracing | 6.5 ± 0.8 | 7.0 ± 1.0 |
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Posttest Scores (out of 16 points)
Robust Evaluation
| Baseline Policy |
Adaptive Policy | |
|---|---|---|
| New Model | 5.9 ± 0.9 | 9.1 ± 0.8 |
| Bayesian Knowledge Tracing | 6.5 ± 0.8 | 7.0 ± 1.0 |
| Deep Knowledge Tracing | 9.9 ± 1.5 | 8.6 ± 2.1 |
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Posttest Scores (out of 16 points)
Takeaway Message
Using a wrong student model can lead to incorrect inferences about the efficacy of an instructional policy.
All Student Models are Wrong
Semantics of a Wrong Model
But Some are Useful
Robust Models
Data-Driven Models + Learning Theory
Misguided Instructional Decisions
Inequitable Outcomes
Mastery learning intends to give each student
the right amount of instruction.
Equity of Mastery Learning
“The [BKT] model overestimates the true learning and performance parameters for below-average students who make many errors. While these students receive more remedial exercises than the above average students, they nevertheless receive less remedial practice than they need and perform worse on the test than expected.”
Corbett and Anderson, 1994
Corbett and Anderson, 1994
Equity of Mastery Learning
Corbett and Anderson, 1994
Doroudi and Brunskill, Learning Analytics & Knowledge 2019
Even after individualizing BKT parameters, they found that
low-performing students do worse on the test.
This inequity could be due to fitting the wrong model.
Doroudi and Brunskill, 2019:
Solution: Individualize BKT parameters for different students.
Corbett and Anderson, 1994
Equity of Mastery Learning
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + {\color{red}{0.1}}t))}
\theta \sim \mathcal{N}(0, 1)
200 students
20 practice opportunities
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + {\color{red}{0.05}}t))}
200 students
20 practice opportunities
Fast Learners
Slow Learners
Doroudi and Brunskill, Learning Analytics & Knowledge 2019
Equity of Mastery Learning
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + {\color{red}{0.1}}t))}
\theta \sim \mathcal{N}(0, 1)
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + {\color{red}{0.05}}t))}
Average P(Correct)
at Mastery:
0.56
Average P(Correct) at Mastery:
0.45
Mastery
Learning
Mastery
Learning
Fast Learners
Slow Learners
Doroudi and Brunskill, Learning Analytics & Knowledge 2019
Consider how
(1) algorithms,
(2) machine learning,
(3) technology design, and
(4) socio-cultural forces
combine to affect equity in
learning technologies.
Equity of Learning Technologies
All Student Models are Wrong
But Some are Useful
Data-Driven Models + Learning Theory
Misguided Instructional Decisions
Inequitable Outcomes
Misguided Inferences about Learning
Robustness to How Students Learn
Robust Evaluation Matrix
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
| Student Models | Policy 1 |
Policy 2 |
Policy 3 |
|---|---|---|---|
| Student Model 1 | |||
| Student Model 2 | |||
| Student Model 3 |
\(V_{SM_1,P_1}\) \(V_{SM_1,P_2}\) \(V_{SM_1,P_3}\)
\(V_{SM_2,P_1}\) \(V_{SM_2,P_2}\) \(V_{SM_2,P_3}\)
\(V_{SM_3,P_1}\) \(V_{SM_3,P_2}\) \(V_{SM_3,P_3}\)
Robust Evaluation Matrix
| Student Models |
Baseline Policy |
Adaptive Policy |
|---|---|---|
| New Model | 5.9 ± 0.9 | 9.1 ± 0.8 |
| Bayesian Knowledge Tracing | 6.5 ± 0.8 | 7.0 ± 1.0 |
| Deep Knowledge Tracing | 9.9 ± 1.5 | 8.6 ± 2.1 |
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Posttest Scores (out of 16 points)
Robust Evaluation Matrix
| Student Models |
Baseline Policy |
Adaptive Policy | Awesome Policy |
|---|---|---|---|
| New Model | 5.9 ± 0.9 | 9.1 ± 0.8 | 16 |
| Bayesian Knowledge Tracing | 6.5 ± 0.8 | 7.0 ± 1.0 | 16 |
| Deep Knowledge Tracing | 9.9 ± 1.5 | 8.6 ± 2.1 | 16 |
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
Posttest Scores (out of 16 points)
Robust Evaluation Matrix
Doroudi, Aleven, and Brunskill, Learning @ Scale 2017
| Student Models | Policy 1 |
Policy 2 |
Policy 3 |
|---|---|---|---|
| Demographic 1 | |||
| Demographic 2 | |||
| Demographic 3 |
\(V_{SM_1,P_1}\) \(V_{SM_1,P_2}\) \(V_{SM_1,P_3}\)
\(V_{SM_2,P_1}\) \(V_{SM_2,P_2}\) \(V_{SM_2,P_3}\)
\(V_{SM_3,P_1}\) \(V_{SM_3,P_2}\) \(V_{SM_3,P_3}\)
Can tell us which policies are equitable
Robust Evaluation Matrix
|
Student Models |
Mastery Learning BKT |
|---|---|
| AFM - Fast Learners | 56% |
| AFM - Slow Learners | 45% |
Doroudi and Brunskill, Learning Analytics & Knowledge 2019
Robust Evaluation Matrix
|
Student Models |
Mastery Learning BKT |
|---|---|
| AFM - Fast Learners | 56% |
| AFM - Slow Learners | 45% |
| BKT - Fast Learners | 98%* |
| BKT - Slow Learners | 97.3%* |
*Percent of students who are in learned state.
Doroudi and Brunskill, Learning Analytics & Knowledge 2019
Robust Evaluation Matrix
|
Student Models |
Mastery Learning
BKT |
Mastery Learning AFM |
|---|---|---|
| AFM - Fast Learners | 56% | 96% |
| AFM - Slow Learners | 45% | 95% |
| BKT - Fast Learners | 98%* | |
| BKT - Slow Learners | 97.3%* |
*Percent of students who are in learned state.
Doroudi and Brunskill, Learning Analytics & Knowledge 2019
Robust Evaluation Matrix
|
Student Models |
Mastery Learning
BKT |
Mastery Learning AFM |
|---|---|---|
| AFM - Fast Learners | 56% | 96% |
| AFM - Slow Learners | 45% | 95% |
| BKT - Fast Learners | 98%* | 99.8%* |
| BKT - Slow Learners | 97.3%* | 99.5%* |
*Percent of students who are in learned state.
Doroudi and Brunskill, Learning Analytics & Knowledge 2019
All Student Models are Wrong
Misguided Inferences about Learning
But Some are Useful
Robustness to How Students Learn
Data-Driven Models + Learning Theory
Misguided Instructional Decisions
Inequitable Outcomes
Student
Model
Big
Data
Student
Model
Data
Theory
Doroudi, Aleven, and Brunskill, In Submission
Integrating Data with Theory
Cognitive
(Information Processing)
DistributedCognition
Constructivism
Socio-Cultural
Situated Cognition
“It can be argued that there is a trade-off between accounting for the subjective experience of doing mathematics and the precision inherent in expressing models in the syntax of computer formalisms.”
Paul Cobb, 1987
“It is desirable to formulate situative models that are specific enough to implement them as simulation programs”
James Greeno, 1998
Theory-Model Gap
Socio-Cultural
Model
Cognitive
Model
Data
Robustness to Learning Theories
Policy 1 |
Policy 2 |
Policy 3 |
|
|---|---|---|---|
| Cognitive Model | |||
| Constructivist Model | |||
| Socio-Cultural Model |
Robustness to Learning Theories
\(V_{SM_1,P_1}\) \(V_{SM_1,P_2}\) \(V_{SM_1,P_3}\)
\(V_{SM_2,P_1}\) \(V_{SM_2,P_2}\) \(V_{SM_2,P_3}\)
\(V_{SM_3,P_1}\) \(V_{SM_3,P_2}\) \(V_{SM_3,P_3}\)
The Bigger Picture
Research Landscape
Properties of Models of Learning
Sequencing Instruction
Learner-Generated Content
Doroudi, Aleven, & Brunskill - L@S '17
Doroudi & Brunskill - LAK '19
Doroudi, Aleven, & Brunskill - In Submission
Doroudi et al. - EDM '15
Doroudi et al. - EDM '16
Doroudi, Thomas, and Brunskill - UAI '17
*Best Paper*
Doroudi & Brunskill - EDM '17
*Best Paper Nominee*
Doroudi et al. - CHI '16
Doroudi et al. - ICLS '18
Research Landscape
Properties of Models of Learning
Sequencing Instruction
Learner-Generated Content
Doroudi, Aleven, & Brunskill - L@S '17
Doroudi & Brunskill - LAK '19
Doroudi, Aleven, & Brunskill - In Submission
Doroudi et al. - EDM '15
Doroudi et al. - EDM '16
Doroudi, Thomas, and Brunskill - UAI '17
*Best Paper*
Doroudi & Brunskill - EDM '17
*Best Paper Nominee*
Doroudi et al. - CHI '16
Doroudi et al. - ICLS '18
This Talk
Assess the robustness of various student models and instructional policies
Future Directions
Study the equitability of learning technologies, including how algorithms interact with socio-cultural factors
Work with online education providers to study how the
consequences in this talk affect actual students
Build student models for settings that we care about
by bridging the theory-model gap
To build more robust data-driven learning technologies while advancing the science of learning
Vision
Acknowledgements
The research reported here was supported, in whole or in part, by the Institute of Education Sciences, U.S. Department of Education, through Grants R305A130215 and R305B150008 to Carnegie Mellon University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Dept. of Education.
Some of the work reported here was written in papers with co-authors Emma Brunskill and Vincent Aleven. I thank Emma Brunskill, Ken Holstein, and Petr Johanes for discussions that influenced this work.
References
Box, G. E. (1979). Robustness in the strategy of scientific model building. In Robustness in statistics (pp. 201-236).
Cen, H. (2009). Generalized learning factors analysis: improving cognitive models with machine learning (Doctoral dissertation). Carnegie Mellon University, Pittsburgh, PA.
Chi, M., VanLehn, K., Litman, D., & Jordan, P. (2011). Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies. User Modeling and User-Adapted Interaction, 21(1-2), 137-180.
Cobb, P. (1990). A constructivist perspective on information-processing theories of mathematical activity. International Journal of Educational Research, 14(1), 67-92.
Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4(4), 253-278.
Doroudi, S., & Brunskill, E. (2017, June). The misidentified identifiability problem of Bayesian Knowledge Tracing. In Proceedings of the 10th International Conference on Educational Data Mining. International Educational Data Mining Society.
References
Doroudi, S. & Brunskill, E. (2019, March). Fairer but not fair enough: On the equitability of knowledge tracing. To appear in Proceedings of the 9th International Learning Analytics & Knowledge Conference. ACM.
Doroudi, S., Aleven, V., & Brunskill, E. (2017, April). Robust evaluation matrix: Towards a more principled offline exploration of instructional policies. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale (pp. 3-12). ACM.
Doroudi, S., Aleven, V. & Brunskill, E. (2018). Where's the reward? A review of reinforcement learning for instructional sequencing. Manuscript in submission.
Duncan, O. D. (1975). Introduction to structural equation models. Elsevier.
Greeno, J. G. (1998). The situativity of knowing, learning, and research. American psychologist, 53(1), 5.
Rowe, J. P., Mott, B. W., & Lester, J. C. (2014). Optimizing Player Experience in Interactive Narrative Planning: A Modular Reinforcement Learning Approach. AIIDE, 3, 2.
Backup Slides
Data
New Instructional Policy
Review of Data-Driven Instruction
Student
Model
vs.
Baseline Policy
Doroudi, Aleven, and Brunskill, In Submission
Better understand researchers' beliefs about learning and computational modeling via interviews (ongoing work).
Bridging the Theory-Model Gap
Use agent-based modeling and social simulation to model socio-cultural and situative theories.
Assess robustness of models under different conceptions of learning.
At least 95% of students learn the skill
Mastery
Learning
Four Clusters of Studies
Doroudi, Aleven, and Brunskill, In Submission
aprender
to learn
Paired-Association Tasks
Concept Learning Tasks
Sequencing Activity Types
Sequencing Interdependent Content
Four Clusters of Studies
Doroudi, Aleven, and Brunskill, In Submission
reading
Paired-Association Tasks
Concept Learning Tasks
Sequencing Activity Types
Sequencing Interdependent Content
Four Clusters of Studies
Doroudi, Aleven, and Brunskill, In Submission
Worked Example
Problem
Solving
\(x^2 - 4 = 12\)
Solve for \(x\):
\(x^2 - 4 = 12\)
\(x^2 = 4 + 12\)
\(x^2 = 16\)
\(x = \sqrt{16} = \pm4\)
\(x^2 - 4 = 12\)
Solve for \(x\):
Paired-Association Tasks
Concept Learning Tasks
Sequencing Activity Types
Sequencing Interdependent Content
Five Clusters of Studies
Doroudi, Aleven, and Brunskill, In Submission
Paired-Association Tasks
Concept Learning Tasks
Sequencing Activity Types
Sequencing Interdependent Content
Four Clusters of Studies
Doroudi, Aleven, and Brunskill, In Submission
| Data-Driven Policy Outperformed Baseline |
Mixed Results |
Data-Driven Policy Did Not Outperform Baseline |
|
|---|---|---|---|
| Paired-Association Tasks | 10 | 0 | 3 |
| Concept Learning Tasks | 2 | 3 | 0 |
| Sequencing Activity Types | 4 | 4 | 0 |
| Sequencing Interdependent Content | 0 | 0 | 6 |
Four Clusters of Studies
Doroudi, Aleven, and Brunskill, In Submission
Paired-Association Tasks
Concept Learning Tasks
Sequencing Activity Types
Sequencing Interdependent Content
Use Psychologically-Inspired Models
Spacing Effect
Worked-Example Effect
Use Data-Driven
Models
We attempt to treat the same problem with several alternative models each with different simplifications but with a common...assumption. Then, if these models, despite their different assumptions, lead to similar results, we have what we can call a robust theorem that is relatively free of the details of the model.
Hence, our truth is the intersection of independent lies.
- Richard Levins, 1966
Importance Sampling
Estimator that gives unbiased and consistent estimates for a policy!
Can have very high variance when policy is different from prior data.
-
Example: Worked example or problem-solving?
20 sequential decisions ⇒ need over 2^{20}$20\(2^{20}\) students!
Importance sampling can prefer the worse of two policies more often than not (Doroudi, Thomas, and Brunskill, 2017).
Doroudi, Thomas, and Brunskill, Uncertainty in Artificial Intelligence 2017, Best Paper
200 students
20 practice opportunities
Fast Learners
Slow Learners
Doroudi and Brunskill, Learning Analytics & Knowledge 2019
UCI Talk
By Shayan Doroudi
UCI Talk
Invited talk given at UCI on December 3rd, 2018.
