CS 4/5789: Introduction to Reinforcement Learning

Lecture 7: Policy Evaluation

Prof. Sarah Dean

MW 1:25-2:40pm
155 Olin Hall

Reminders

• Homework
  • PSet 2 due today
  • Peer groups assigned Fri
  • Project 2 released today
• In-class Exam on 9/23
• Review session Monday

Agenda

1. Exam Details
2. Example
3. Exact Policy Evaluation
4. Iterative Policy Evaluation

Exam 1

• When:  Wednesday 9/23 during lecture 1:25-2:40pm
  • first 10/last 5 buffer for distribution/collection
• Where: Olin 155 or SDS ATP
• What: (Logistics)
  • One hour pen and paper exam
  • Closed book, no electronics
    • if we see a phone, we will take your exam*
  • One page cheat sheet allowed
  • Assigned seats (alphabetical by netID)
  • No bathroom breaks*

Help me check the seat map!

Exam 1 Content

• Everything up until (including) today, but emphasis on Lectures 1-6
  • Primary source: PSets 1&2, Lecture Notes 1 (minus 1.5)
  • Supporting source: Lecture Notes 2 up thru 2.2, Projects 1&2 (no code-specific questions)
• Four questions: one T/F, one MC, two short answer
• Short answer questions will have partial credit and may require explanations or explicit step-by-step derivation, shorter/simpler than standard homework question
• Design: exam takes 30-40 min for knowledgeable TA

How can we evaluate policies over indefinite horizons?

Agenda

1. Exam Details
2. Example
3. Exact Policy Evaluation
4. Iterative Policy Evaluation

Think/Pair/Share

Suppose our policy always takes a fixed action \(a\).

Define a vector \(U\in\mathbb R^{|\mathcal S|}\) such that $$\mathbb E_{s'\sim P(\cdot|s,a)}[V(s')] = U^\top V$$

Transition Matrix

Given a policy \(\pi(\cdot)\) and a transition function \(P(\cdot\mid \cdot,\cdot)\)

• Define the transition matrix \(P_{\pi} \in \mathbb R^{S\times S}\)
• At row \(s\) and column \(s'\), entry is \(\mathbb E_{a\sim\pi(s)}[P(s'\mid s,a)]\)

\(P_{\pi}=\)

\(s\)

\(s'\)

\(\mathbb E_{a\sim\pi(s)}[P(s'\mid s,a)]\)

Bellman Consistency