CS 4/5789: Introduction to Reinforcement Learning

Lecture 8: Iterative Policy Evaluation & Review

Prof. Sarah Dean

MW 1:25-2:40pm
155 Olin Hall

Reminders

  • Homework
    • Peer groups assigned
    • Project 2 released
  • In-class Exam on Wednesday

Agenda

  1. Iterative Policy Evaluation
  2. Exam Details
  3. Review (your questions)

Transition Matrix

Given a policy \(\pi(\cdot)\) and a transition function \(P(\cdot\mid \cdot,\cdot)\)

  • Define the transition matrix \(P_{\pi} \in \mathbb R^{S\times S}\)
  • At row \(s\) and column \(s'\), entry is \(\mathbb E_{a\sim\pi(s)}[P(s'\mid s,a)]\)

\(P_{\pi}=\)

\(s\)

\(s'\)

\(\mathbb E_{a\sim\pi(s)}[P(s'\mid s,a)]\)

Bellman Consistency

  • The matrix vector form of the Bellman Equation is $$V^{\pi} = R^{\pi} + \gamma P_{\pi} V^\pi$$

\(s\)

\(s'\)

\(P(s'\mid s,\pi(s))\)

\(=\)

\(+\gamma\)

\(V^\pi(s)\)

\(r(s,\pi(s))\)

Exam 1

  • When:  Wednesday 9/23 during lecture 1:25-2:40pm
    • first 10/last 5 buffer for distribution/collection
  • Where: Olin 155 or through SDS ATP (Malott Hall)
  • What: (Logistics)
    • One hour pen and paper exam
    • Closed book, no electronics
      • if we see a phone, we will take your exam*
    • One page cheat sheet allowed
    • Assigned seats (alphabetical by netID)
    • No bathroom breaks*

*accommodations possible, fill in form ASAP

Check the seat map! Submit seating request!

Exam 1 Content

  • Everything up until (including) today, but emphasis on Lectures 1-6
    • Primary source: PSets 1&2, Lecture Notes 1 (minus 1.5)
    • Supporting source: Lecture Notes 2 up thru 2.2, Projects 1&2 (no code-specific questions)
  • Four questions: one T/F, one MC, two short answer
  • Short answer questions will have partial credit and may require explanations or explicit step-by-step derivation, shorter/simpler than standard homework question
  • Design: exam takes 30-40 min for knowledgeable TA