Reinforcement Learning for Instructional Sequencing:
Learning from Its Past to Meet the Challenges of the Future

Shayan Doroudi

RL4ED Workshop @ EDM 2021

Over the past 50 years, how successful has RL been in discovering useful adaptive instructional policies?

Over the past 50 years, how successful has RL been in discovering useful adaptive instructional policies?

Learning from the Past

Searched for all empirical studies (as of December 2018) that compared one or more RL-induced policies (broadly conceived) to one or more baseline policies.

Review of Empirical Studies

The Role of Theory

Use Psychologically-Inspired Models

Spacing Effect

Concept Learning Tasks

Sequencing Activity Types

Sequencing Interdependent Content

Paired-Associate Learning Tasks

What are the challenges that we must face if we are to use RL productively in education going forward?

Meeting the Challenges of the Future

Three Challenges

Three Challenges

Effectiveness

Evaluation

External Validity

Effectiveness

How do we most effectively leverage RL to enhance student learning?

Effectiveness

How do we most effectively leverage RL to enhance student learning?

Data-driven models should be combined with insights from psychological theories.

Choice of models

Theory can inform:

Evaluation

We need to compare RL-induced policies to the most effective baseline policies we can think of.

How do we know if RL actually enhances student learning?

For sequencing activity types, we could compare to policies based on principles such as the expertise-reversal effect.

For sequencing interdependent content, we could compare to heuristics that sequence based-on expert-curated knowledge maps.

Evaluation

Even when RL-induced policies outperform baselines, there could be at least three reasons:

How do we know if RL actually enhances student learning?

Evaluation

In order to determine if adaptivity matters, we can use yoked control designs.

How do we know if RL actually enhances student learning?

Kalyuga, S., & Sweller, J. (2005). Rapid dynamic assessment of expertise to improve the efficiency of adaptive e-learning. Educational Technology Research and Development, 53(3), 83-93.

External Validity

How do we ensure RL-based systems actually enhance student learning in the real world?

External Validity

We need to take into account the realities of real-world educational environments.

How do we ensure RL-based systems actually enhance student learning in the real world?

Just because an RL-based system is shown to be effective does not necessarily mean it will get used.

[T]he development of a theory of instruction cannot progress if one holds the view that a complete theory of learning is a prerequisite.

Atkinson (1972)

Acknowledgements

The research reported here was supported, in whole or in part, by the Institute of Education Sciences, U.S. Department of Education, through Grants R305A130215 and R305B150008 to Carnegie Mellon University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Dept. of Education.

Inclusion Criteria

We consider any papers published before December 2018 where:

• There is (implicitly) a model of the learning process, where different instructional actions probabilistically change the state of a student.
• There is an instructional policy that maps past observations from a student (e.g., responses to questions) to instructional actions.
• Data collected from students are used to learn either:
  • the model
  • an adaptive policy
• If the model is learned, the instructional policy is designed to (approximately) optimize that model according to some reward function

What's Not Included?

• Adaptive policies that use hand-made or heuristic decision rules (rather than data-driven/optimized decision rules)
• Experiments that do not control for everything other than sequence of instruction
• Experiments that use RL for other educational purposes, such as:
  • generating data-driven hints (Stamper et al., 2013) or
  • giving feedback (Rafferty et al., 2015)

“The mathematical techniques of optimization used in theories of instruction draw upon a wealth of results from other areas of science, especially from tools developed in mathematical economics and operations research over the past two decades, and it would be my prediction that we will see increasingly sophisticated theories of instruction in the near future.”

Suppes (1974)

The Place of Theory in Educational Research

AERA Presidential Address

RL's Status in Education Research

The Role of Theory

Psychological theory can help determine both when sequencing might matter as well as how to most effectively leverage RL for instructional sequencing.

Case Study

• Fractions Tutor
• Two experiments testing RL-induced policies (both no sig difference)
• Off-policy policy evaluation

Fractions Tutor

Experiment 1

• Used prior data to fit G-SCOPE Model (Hallak et al., 2015).

• Used G-SCOPE Model to derive two new Adaptive Policies.

• Wanted to compare Adaptive Policies to a Baseline Policy (fixed, spiraling curriculum).

• Simulated both policies on G-SCOPE Model to predict posttest scores (out of 16 points).

Single Model Simulation

• Used by Chi, VanLehn, Littman, and Jordan  (2011) and Rowe, Mott, and Lester (2014) in educational settings.
• Rowe, Mott, and Lester (2014): New adaptive policy estimated to be much better than random policy.
• But in experiment, no significant difference found (Rowe and Lester, 2015).

Importance Sampling

• Estimator that gives unbiased and consistent estimates for a policy!
• Can have very high variance when policy is different from prior data.
• Example: Worked example or problem-solving?
  • 20 sequential decisions ⇒ need over \(2^{20}\) students
  • 50 sequential decisions ⇒ need over \(2^{50}\) students!
• Importance sampling can prefer the worse of two policies more often than not (Doroudi et al., 2017b).

Doroudi, Thomas, and Brunskill, UAI 2017, Best Paper

Robust Evaluation Matrix

\(V_{SM_1,P_1}\)

\(V_{SM_1,P_2}\)

\(V_{SM_2,P_2}\)

\(V_{SM_3,P_2}\)

\(V_{SM_1,P_3}\)

\(V_{SM_2,P_3}\)

\(V_{SM_3,P_3}\)

Experiment 2

• Used Robust Evaluation Matrix to test new policies
• Found that a New Adaptive Policy that was very simple but robustly expected to do well:
  • sequence problems in increasing order of avg. time
  • skip any problems where students have demonstrated mastery of all skills (according to BKT)
• Ran an experiment testing New Adaptive Policy

Experiment 2

Experiment 2:
Insights

• Even though we did robust evaluation, two things were not considered adequately:

  • How long each problem takes per student

  • Student population mismatch

Overview

• Reinforcement Learning: Towards a "Theory of Instruction"
• Part 1: Historical Perspective
• Part 2: Systematic Review
  • Discussion: Where's the Reward?
• Part 3: Case Study: Fractions Tutor and Policy Selection
• Planning for the Future

Planning for the Future

• Data-Driven + Theory-Driven Approach

  • Reinforcement learning researchers should work with learning scientists and psychologists.

  • Work on domains where we have or can develop decent cognitive models.

  • Work in settings where the set of actions is restricted but that are still meaningful
    (e.g., worked examples vs. problem solving)

  • Compare to good baselines based on learning sciences (e.g., expertise reversal effect)

• Do thoughtful and extensive offline evaluations.

• Iterate and replicate! Develop theories of instruction that can help us see where the reward might be.

Is Data-Driven Sufficient?

• Might we see a revolution in data-driven instructional sequencing?
  • More data
  • More computational power
  • Better RL algorithms
• Similar advances have recently revolutionized the fields of computer vision, natural language processing, and computational game-playing.
• Why not instruction?
  • Learning is fundamentally different from images, language, and games.
  • Baselines are much stronger for instructional sequencing.

So, where is the reward?

• In the coming years, will likely see both purely data-driven (deep learning) approaches as well as theory+data-driven approaches to instructional sequencing.
  • Only time can tell where the reward lies, but our robust evaluation suggests combining theory and data.
  • By reviewing the history and prior empirical literature, we can have a better sense of the terrain we are operating in.

So, where is the reward?

• Applying RL to instructional sequencing has been rewarding in other ways:
  • Advances have been made to the field of RL.
    • The Optimal Control of Partially Observable Markov Processes
    • Our work on importance sampling (Doroudi et al., 2017b)
  • Advances have been made to student modeling.