Where's The Reward?

A Review of Reinforcement Learning for
Instructional Sequencing

Shayan Doroudi, Vincent Aleven, Emma Brunskill

AIED 2020 Journal Track

Over the past 50 years, how successful has reinforcement learning been in discovering useful adaptive instructional policies?

Research Questions

Under what conditions is reinforcement learning most likely to be successful in advancing instructional sequencing?

Research Questions

Overview

• Reinforcement Learning: Towards a "Theory of Instruction"
• Historical Perspective
• Review of Empirical Studies
• Discussion: Where's the Reward?
• Planning for the Future

Overview

• Reinforcement Learning: Towards a "Theory of Instruction"
• Historical Perspective
• Review of Empirical Studies
• Discussion: Where's the Reward?
• Planning for the Future

Reinforcement Learning (RL)

Markov Decision Process (MDP)

• Set of States \(S\)

• Set of Actions \(A\)

• Transition Matrix \(T\)

• Reward function \(R\)

• Horizon \(H\)

Theory of Instruction

Atkinson's (1972) “Ingredients for a Theory of Instruction”:

taken in conjunction with methods for deriving optimal strategies

• A model of the learning process.

• Specification of admissible instructional actions.

• Specification of instructional objectives

• A measurement scale that permits costs to be assigned to each of the instructional actions and and payoffs to the achievement of instructional objectives.

Overview

• Reinforcement Learning: Towards a "Theory of Instruction"
• Historical Perspective
• Review of Empirical Studies
• Discussion: Where's the Reward?
• Planning for the Future

First Wave: 1960s-70s

• Teaching machines were popular in late 50s-early 60s.
• Computers! → Computer-Assisted Instruction
• Dynamic Programming and Markov Decision Processes
• Mathematical Psychology: studying mathematical models of learning

“The mathematical techniques of optimization used in theories of instruction draw upon a wealth of results from other areas of science, especially from tools developed in mathematical economics and operations research over the past two decades, and it would be my prediction that we will see increasingly sophisticated theories of instruction in the near future.”

Suppes (1974)

The Place of Theory in Educational Research

AERA Presidential Address

RL's Status in Education Research

The Dark Ages

c. 1975 - 1999

Seemingly no work applying reinforcement learning to instructional sequencing during this time.

The Three Waves

Overview

• Reinforcement Learning: Towards a "Theory of Instruction"
• Historical Perspective
• Review of Empirical Studies
• Discussion: Where's the Reward?
• Planning for the Future

Searched for all empirical studies (as of December 2018) that compared one or more RL-induced policies (broadly conceived) to one or more baseline policies.

Review of Empirical Studies

Five Clusters of Studies

Overview

• Reinforcement Learning: Towards a "Theory of Instruction"
• Historical Perspective
• Review of Empirical Studies
• Discussion: Where's the Reward?
• Planning for the Future

The Role of Theory

Use Psychologically-Inspired Models

Spacing Effect

Concept Learning Tasks

Sequencing Activity Types

Sequencing Interdependent Content

Paired-Associate Learning Tasks

When Has RL Been Successful?

Students' prior knowledge can effect how much instructional sequencing matters.

Overview

• Reinforcement Learning: Towards a "Theory of Instruction"
• Historical Perspective
• Review of Empirical Studies
• Discussion: Where's the Reward?
• Planning for the Future

Planning for the Future

Data-driven models should be combined with insights from psychological theories.

Planning for the Future

Machine intelligence should be combined with insights from human intelligence.

[T]he development of a theory of instruction cannot progress if one holds the view that a complete theory of learning is a prerequisite.

Atkinson (1972)

Acknowledgements

The research reported here was supported, in whole or in part, by the Institute of Education Sciences, U.S. Department of Education, through Grants R305A130215 and R305B150008 to Carnegie Mellon University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Dept. of Education.

Inclusion Criteria

We consider any papers published before December 2018 where:

• There is (implicitly) a model of the learning process, where different instructional actions probabilistically change the state of a student.
• There is an instructional policy that maps past observations from a student (e.g., responses to questions) to instructional actions.
• Data collected from students are used to learn either:
  • the model
  • an adaptive policy
• If the model is learned, the instructional policy is designed to (approximately) optimize that model according to some reward function

What's Not Included?

• Adaptive policies that use hand-made or heuristic decision rules (rather than data-driven/optimized decision rules)
• Experiments that do not control for everything other than sequence of instruction
• Experiments that use RL for other educational purposes, such as:
  • generating data-driven hints (Stamper et al., 2013) or
  • giving feedback (Rafferty et al., 2015)

The Role of Theory

Psychological theory can help determine both when sequencing might matter as well as how to most effectively leverage RL for instructional sequencing.

Case Study

• Fractions Tutor
• Two experiments testing RL-induced policies (both no sig difference)
• Off-policy policy evaluation

Fractions Tutor

Experiment 1

• Used prior data to fit G-SCOPE Model (Hallak et al., 2015).

• Used G-SCOPE Model to derive two new Adaptive Policies.

• Wanted to compare Adaptive Policies to a Baseline Policy (fixed, spiraling curriculum).

• Simulated both policies on G-SCOPE Model to predict posttest scores (out of 16 points).

Single Model Simulation

• Used by Chi, VanLehn, Littman, and Jordan  (2011) and Rowe, Mott, and Lester (2014) in educational settings.
• Rowe, Mott, and Lester (2014): New adaptive policy estimated to be much better than random policy.
• But in experiment, no significant difference found (Rowe and Lester, 2015).

Importance Sampling

• Estimator that gives unbiased and consistent estimates for a policy!
• Can have very high variance when policy is different from prior data.
• Example: Worked example or problem-solving?
  • 20 sequential decisions ⇒ need over \(2^{20}\) students
  • 50 sequential decisions ⇒ need over \(2^{50}\) students!
• Importance sampling can prefer the worse of two policies more often than not (Doroudi et al., 2017b).

Doroudi, Thomas, and Brunskill, UAI 2017, Best Paper

Robust Evaluation Matrix

\(V_{SM_1,P_1}\)

\(V_{SM_1,P_2}\)

\(V_{SM_2,P_2}\)

\(V_{SM_3,P_2}\)

\(V_{SM_1,P_3}\)

\(V_{SM_2,P_3}\)

\(V_{SM_3,P_3}\)

Experiment 2

• Used Robust Evaluation Matrix to test new policies
• Found that a New Adaptive Policy that was very simple but robustly expected to do well:
  • sequence problems in increasing order of avg. time
  • skip any problems where students have demonstrated mastery of all skills (according to BKT)
• Ran an experiment testing New Adaptive Policy

Experiment 2

Experiment 2:
Insights

• Even though we did robust evaluation, two things were not considered adequately:

  • How long each problem takes per student

  • Student population mismatch

Overview

• Reinforcement Learning: Towards a "Theory of Instruction"
• Part 1: Historical Perspective
• Part 2: Systematic Review
  • Discussion: Where's the Reward?
• Part 3: Case Study: Fractions Tutor and Policy Selection
• Planning for the Future

Planning for the Future

• Data-Driven + Theory-Driven Approach

  • Reinforcement learning researchers should work with learning scientists and psychologists.

  • Work on domains where we have or can develop decent cognitive models.

  • Work in settings where the set of actions is restricted but that are still meaningful
    (e.g., worked examples vs. problem solving)

  • Compare to good baselines based on learning sciences (e.g., expertise reversal effect)

• Do thoughtful and extensive offline evaluations.

• Iterate and replicate! Develop theories of instruction that can help us see where the reward might be.

Is Data-Driven Sufficient?

• Might we see a revolution in data-driven instructional sequencing?
  • More data
  • More computational power
  • Better RL algorithms
• Similar advances have recently revolutionized the fields of computer vision, natural language processing, and computational game-playing.
• Why not instruction?
  • Learning is fundamentally different from images, language, and games.
  • Baselines are much stronger for instructional sequencing.

So, where is the reward?

• In the coming years, will likely see both purely data-driven (deep learning) approaches as well as theory+data-driven approaches to instructional sequencing.
  • Only time can tell where the reward lies, but our robust evaluation suggests combining theory and data.
  • By reviewing the history and prior empirical literature, we can have a better sense of the terrain we are operating in.

So, where is the reward?

• Applying RL to instructional sequencing has been rewarding in other ways:
  • Advances have been made to the field of RL.
    • The Optimal Control of Partially Observable Markov Processes
    • Our work on importance sampling (Doroudi et al., 2017b)
  • Advances have been made to student modeling.