CS 4/5789: Introduction to Reinforcement Learning

Lecture 2: Policies and Estimation

Prof. Sarah Dean

MW 1:25-2:40pm
155 Olin Hall

Reminders

• Homework
  • PSet 1 released today, due next Wednesday
  • Project 1 released by Friday, due 9/11
• Office Hours: mine are Wednesday 3-4pm, see Canvas/Ed
• Enrollment: join waitlist, come to lecture, do assignments, there should be plenty of space

Agenda

1. RL Setting

2. Estimation and Variance

3. Off-Policy Estimation

Given data, how do we estimate the expected reward of a given policy?

Setting

action

observation (context)

reward

1. Agent observes environment
2. Agent takes action
3. Environment returns reward
4. Repeat

1. RL Setting

2. Estimation 

3. Off-Policy

Setting

action

observation (context)

\(a_t\sim \pi_\theta(x_t)\)

reward

\(x_t\sim \mu\)

\(r_t\sim R(x_t,a_t)\)

1. Agent observes environment
2. Agent takes action
3. Environment returns reward
4. Repeat

1. RL Setting

2. Estimation 

3. Off-Policy

Example

• context: \(x\sim \mathcal \mu\)
  • user profile (demographics, view history)
  • distribution \(\mathcal \mu\) is the user base
• action: \(a\)
  • recently released movie
• policy:  \(\pi_\theta(x)\)
  • movie recommender network
  • Think/Pair/Share: ways to ensure outputs are valid probabilities?
• reward:  \(r\sim R(x,a)\)
  • whether user clicks on recommendation

personalized Netflix banner

1. RL Setting

2. Estimation 

3. Off-Policy

\(\mathcal{L}(\theta) = \mathbb{E}_{x,y\sim\mathcal{D}}[\ell(f_\theta(x), y)]\)

RL for Supervised Learning?

\(J(\theta)=\mathbb E_{x\sim \mu, a\sim \pi_\theta(x), r\sim R(x,a) }\left[r\right]\)

1. RL Setting

2. Estimation 

3. Off-Policy

• Collect data with \(a\sim\pi_\theta\): \((x_1,a_1, r_1),(x_2,a_2,r_2),...,(x_n,a_n,r_n)\)
• \(\displaystyle \hat J_\theta = \frac{1}{n} \sum_{i=1}^n r_i \) is unbiased estimate of \(J(\theta)\)
• Variance & decomposition

1. RL Setting

2. Estimation

3. Off-Policy

Monte Carlo Estimation

• Suppose \(x\) are drawn uniform from \(\{0,1\}\) and \(\mathcal A = \{0,1\}\) with $$R(x,a)=\begin{cases}\text{Bernoulli}(p) & x=a \\ \text{Bernoulli}(0) & x\neq a \end{cases}$$

• Estimate the expected lift instead $$r\rightarrow r - b(x)$$
• Does this change the objective?
• Why might it reduce the variance?

1. RL Setting

2. Estimation

3. Off-Policy

Variance Reduction

• Fact (reweighting trick): Suppose that \(\rho,\rho'\) are distributions, and \(u\sim\rho'\). As long as \(\rho'(u)>0\) whenever \(\rho(u)>0\),
  \[\mathbb E_{u\sim\rho'}\left[\frac{\rho(u)}{\rho'(u)}u\right] = \int_{u}\frac{\rho(u)}{\rho'(u)}u \rho'(u)du=\mathbb E_{u\sim\rho}[u]\]
• Off-policy estimate \(\hat J_\theta'\)
• Importance weights and variance

1. RL Setting

2. Estimation

3. Off-Policy

Off-Policy Estimation

Recap

Given data \(\{(x_i,a_i,r_i)\}_{i=1}^n\) collected with \(a\sim\pi'\) , how do we estimate the expected reward \( J(\theta)=\mathbb E_{x\sim \mu, a\sim \pi_\theta(x), r\sim R(x,a) }\left[r\right]\)?

$$\hat J_\theta' = \frac{1}{n} \sum_{i=1}^n \frac{\pi_\theta(a_i|x_i)}{\pi'(a_i|x_i)} (r_i - b(x_i))$$