Sarah Dean PRO
asst prof in CS at Cornell
CS 4/5789: Introduction to Reinforcement Learning
Lecture 8: Iterative Policy Evaluation & Review
Prof. Sarah Dean
MW 1:25-2:40pm
155 Olin Hall
Reminders
- Homework
- Peer groups assigned
- Project 2 released
- In-class Exam on Wednesday
Agenda
- Iterative Policy Evaluation
- Exam Details
- Review (your questions)
Transition Matrix
Given a policy \(\pi(\cdot)\) and a transition function \(P(\cdot\mid \cdot,\cdot)\)
- Define the transition matrix \(P_{\pi} \in \mathbb R^{S\times S}\)
- At row \(s\) and column \(s'\), entry is \(\mathbb E_{a\sim\pi(s)}[P(s'\mid s,a)]\)
\(P_{\pi}=\)
\(s\)
\(s'\)
\(\mathbb E_{a\sim\pi(s)}[P(s'\mid s,a)]\)
Bellman Consistency
- The matrix vector form of the Bellman Equation is $$V^{\pi} = R^{\pi} + \gamma P_{\pi} V^\pi$$
\(s\)
\(s'\)
\(P(s'\mid s,\pi(s))\)
\(=\)
\(+\gamma\)
\(V^\pi(s)\)
\(r(s,\pi(s))\)
Exam 1
- When: Wednesday 9/23 during lecture 1:25-2:40pm
- first 10/last 5 buffer for distribution/collection
- Where: Olin 155 or through SDS ATP (Malott Hall)
- What: (Logistics)
- One hour pen and paper exam
- Closed book, no electronics
- if we see a phone, we will take your exam*
- One page cheat sheet allowed
- Assigned seats (alphabetical by netID)
- No bathroom breaks*
*accommodations possible, fill in form ASAP
Check the seat map! Submit seating request!
Exam 1 Content
- Everything up until (including) today, but emphasis on Lectures 1-6
- Primary source: PSets 1&2, Lecture Notes 1 (minus 1.5)
- Supporting source: Lecture Notes 2 up thru 2.2, Projects 1&2 (no code-specific questions)
- Four questions: one T/F, one MC, two short answer
- Short answer questions will have partial credit and may require explanations or explicit step-by-step derivation, shorter/simpler than standard homework question
- Design: exam takes 30-40 min for knowledgeable TA
Fa26 CS 4/5789: Lecture 8
By Sarah Dean
Private
