Fairer but Not Fair Enough
On the Equitability of
Knowledge Tracing
LAK 2019
Shayan Doroudi & Emma Brunskill
The Promise of Mastery Learning
One Size Fits All
Mastery Learning
“The [BKT] model overestimates the true learning and performance parameters for below-average students who make many errors. While these students receive more remedial exercises than the above average students, they nevertheless receive less remedial practice than they need and perform worse on the test than expected.”
Corbett and Anderson, 1995
“17% of students would be expected to have a probability of mastery of only 60% or less when the population model would expect the student is at a probability of mastery of 95% or higher”
Lee and Brunskill, 2012
The Reality of Mastery Learning
Over 500,000 students/year
~12 million active monthly users
Why Does Mastery Learning Matter?
Bayesian Knowledge Tracing
N-Consecutive Correct in a Row
Over 50,000
students/year
Defining Equitable Outcomes
Primary Equity Concern:
Students from different demographics should be equally likely to learn all skills.
Secondary Equity Concern:
Students from different demographics should get comparable amounts of unnecessary practice.
Disagree? Great! Let's discuss.
Reinterpreting Mastery Learning
One Size Fits All
Mastery Learning
Equitable
Inequitable
“The [BKT] model overestimates the true learning and performance parameters for below-average students who make many errors. While these students receive more remedial exercises than the above average students, they nevertheless receive less remedial practice than they need and perform worse on the test than expected.”
Corbett and Anderson, 1994
“17% of students would be expected to have a probability of mastery of only 60% or less when the population model would expect the student is at a probability of mastery of 95% or higher”
Lee and Brunskill, 2012
The Reality of Mastery Learning
Inequitable
Mastery learning can be inequitable when the assumptions of our models are not accurate.
Outline
Background
Lack of Individualization
Model Misspecification
Background
Bayesian Knowledge Tracing (BKT)
Corbett and Anderson, 1995
Cognitive Mastery Learning
Keep giving practice opportunities on a skill/concept until student reaches mastery:
P(\text{Learned}) > 0.95
Then move onto the next skill/concept
Corbett and Anderson, 1995
Lack of Individualization
200 students
20 practice opportunities
Fast Learners
Slow Learners
How Fair Is One-Size-Fits-All?
Fairer than One-Size-Fits-All!
But Not Fair Enough!
Inequitable!
\left.\right\}
Solution: Individualization?
Even after individualizing BKT parameters, they found that
low-performing students do worse on the test.
Solution: Individualize BKT parameters for different students.
Corbett and Anderson, 1995
Why?
- Model not individualized well enough?
- BKT might not accurately explain how students learn
Model Misspecification
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + {\color{red}{0.1}}t))}
\theta \sim \mathcal{N}(0, 1)
200 students
20 practice opportunities
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + {\color{red}{0.05}}t))}
200 students
20 practice opportunities
Fast Learners
Slow Learners
Equity of Wrong Model
Equity of Wrong Model
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + {\color{red}{0.1}}t))}
\theta \sim \mathcal{N}(0, 1)
P(\text{Correct}) = \dfrac{1}{1 + \exp(-(\theta - 2 + {\color{red}{0.05}}t))}
Average P(Correct)
at Mastery:
0.56
Average P(Correct) at Mastery:
0.45
Mastery
Learning
Mastery
Learning
Fast Learners
Slow Learners
How to Find Equitable Models
We need to look for better student models, not in terms of accuracy or even mastery on average
Mastery learning with AFM appears to be more equitable even when students learn according to BKT
...but at the expense of giving more extra practice
but in terms of equity
To do so, we need to reason about how student models perform under model misspecification
When the model is wrong, mastery learning might not be as fair as it promises
Conclusions
-
Lack of individualization
-
Model mispecification
Mastery learning is more fair than not using mastery learning, but we should aim higher!
Let's look for models that are equitable regardless of how students actually learn.
This research was supported in part by a Google grant and a Microsoft Research Faculty Fellowship.
Backup Slides
Fairness of Various Algorithms
|
Student Models |
Mastery Learning BKT |
|---|---|
| AFM - Fast Learners | 56% |
| AFM - Slow Learners | 45% |
Equity Under Model Mismatch
|
Student Models |
Mastery Learning BKT |
|---|---|
| AFM - Fast Learners | 56% |
| AFM - Slow Learners | 45% |
| BKT - Fast Learners | 98%* |
| BKT - Slow Learners | 97.3%* |
*Percent of students who are in learned state.
Equity Under Model Mismatch
|
Student Models |
Mastery Learning
BKT |
Mastery Learning AFM |
|---|---|---|
| AFM - Fast Learners | 56% | 96% |
| AFM - Slow Learners | 45% | 95% |
| BKT - Fast Learners | 98%* | |
| BKT - Slow Learners | 97.3%* |
*Percent of students who are in learned state.
Equity Under Model Mismatch
Equity Under Model Mismatch
|
Student Models |
Mastery Learning
BKT |
Mastery Learning AFM |
|---|---|---|
| AFM - Fast Learners | 56% | 96% |
| AFM - Slow Learners | 45% | 95% |
| BKT - Fast Learners | 98%* | 99.8%* |
| BKT - Slow Learners | 97.3%* | 99.5%* |
*Percent of students who are in learned state.
LAK 2019
By Shayan Doroudi
LAK 2019
Presentation on LAK 2019 paper "Fairer but Not Fair Enough: On the Equitability of Knowledge Tracing"
