CS 4/5789: Introduction to Reinforcement Learning

Lecture 1: Introduction

Prof. Sarah Dean

MW 1:25-2:40pm
155 Olin Hall

Agenda

1. What is Reinforcement Learning (RL)?

2. Logistics and Syllabus

3. Types of Machine Learning (ML)

4. RL Setting

AlphaGo

Robotic Manipulation

Algorithmic Media Feeds

Chatbots and Assistants

RL is for Decision-Making

Decision-Making

observation

action

reward

AlphaGo

Robotic Manipulation

Media Feeds

Chatbot

?

?

?

?

?

?

?

?

?

Agenda

1. What is Reinforcement Learning (RL)?

2. Logistics and Syllabus

3. Types of Machine Learning (ML)

4. RL Setting

Logistics

• Instructor: Prof. Sarah Dean
• Grad TAs: Yujin Kim, Akanksha Sarkar, Haruka Kiyohara
• TAs: Guohao Chen, Bodong Liu, Jason Wu, Alex Noviello, Aidan Wu, Myles Pasetsky, Leon Jiao

• Contact: Ed Discussion
• Instructor Office Hours: TBA
• TA Office Hours: TBA, see Ed Discussion

Waitlist and Enrollment

There should be plenty of space!

Course staff do not manage waitlist and enrollment.

CS enrollment policies:
https://www.cs.cornell.edu/courseinfo/enrollment

Exams

• Two in class, closed book prelims (15% of final grade each)
  • Wednesday September 23rd
  • Wednesday October 28th
• Final exam during finals period, time TBD (20% of grade)

Homework

• Homework assignments (30% of final grade)
  • ~5 problem sets (math), groups of up to four
    • Individual oral peer assessments
  • ~5 programming projects (coding), groups of up to two
    • Autograded tests + TA graded report
• 5789: Paper review assignments (after Unit 1) 
• Collaboration Policy: outside of formal group, you may discuss problems but do not look at, copy, or lightly adapt a completed solution (including AI).
• Late Policy: six slip days, use no more than three at once (see Syllabus)

Oral Peer Assessment

• Goal: assess and encourage real understanding, not just correct looking solutions
• Assessing your peers: 10% of final grade
• Meet with small group of peers (randomly assigned)
• Each student explains their solution and answers questions about their reasoning
• Peers determine pass/fail assessment and provide qualitative feedback on level of understanding

Participation

Participation is 10% of final grade, /20 points

• Lecture participation = 1pt each
  • Poll Everywhere: PollEv.com/sarahdean011
• Helpful posts on Ed Discussions = 2pt each
  • TA endorsement

Schedule

• Unit 1: Policy Optimization
  • Estimation, Iterative Optimization, Policy Gradients
• Unit 2: Markov Decision Processes
  • Value Iteration, Policy Iteration, Learning from Data
• Unit 3: Exploration

  • Bandits, Receding Horizon Planning, Learning from Experts

Prerequisites

Machine learning (e.g., CS 3780)

Background in probability, linear algebra, and programming.

Materials

Lecture Slides and Notes on Canvas

Agenda

1. What is Reinforcement Learning (RL)?

2. Logistics and Syllabus

3. Types of Machine Learning (ML)

4. RL Setting

Types of Machine Learning

1. Unsupervised Learning
2. Supervised Learning
3. Reinforcement Learning

Unsupervised Learning

Examples: clustering, PCA, image generation

• Goal:
  • summarization
• Dataset:
  • information about many instances
  • \(\{x_1, x_2, \dots x_N\}\)
• Evaluation:
  • qualitative

Supervised Learning

Examples: classification, regression, conditional generation

• Goal:
  • prediction
• Dataset:
  • each instance has features and label
  • \(\{(x_1, y_1), \dots (x_N,y_N)\}\)
• Evaluation:
  • accuracy, \(y\) vs. \(\hat y\)

Reinforcement Learning

• Goal:
  • action or decision
• Dataset:
  • history of observations, actions, and rewards
  • \(\{(x_1, a_1, r_1),...,(x_N, a_N, r_N)\}\)
  • may be sequentially dependent
• Evaluation:
  • cumulative reward

Types of Machine Learning

1. Unsupervised Learning (descriptive)
   • summarize unstructured data \(\{x_i\}_{i=1}^N\)
2. Supervised Learning (predictive)
   • predict labels from features \(\{(x_i, y_i)\}_{i=1}^N\)
3. Reinforcement Learning (prescriptive)
   • choose actions that lead to high reward \(\{(x_i, a_i, r_i)\}_{i=1}^N\)
   • data depends on the decisions

Agenda

1. What is Reinforcement Learning (RL)?

2. Logistics and Syllabus

3. Types of Machine Learning (ML)

4. RL Setting

RL Setting

1. Agent observes environment
2. Agent takes action
   • according to policy \(\pi\)
3. Environment returns reward
   • dependent on context/action
4. Repeat

Example

• context: \(x\sim \mathcal D\)
  • user profile (demographics, view history)
  • distribution \(\mathcal D\) is the user base
• action: \(a\)
  • recently released movie
• policy:  \(\pi(x)\)
  • movie recommendation (personalized)
• reward:  \(r\sim R(x,a)\)
  • whether user clicks on recommendation
  • distribution \(R(x,a)\) depends on user and movie

personalized Netflix banner

Recap

action

observation (context)

\(a_t\)

reward

\(x_t\)

\(r_t\)

\(\sim R(x_t,a_t)\)

\(\sim \pi(x_t)\)

\(\sim \mathcal D\) (for now)

\(\displaystyle \max_\pi ~~ \mathbb E_{x_t\sim \mathcal D, a_t\sim \pi(x_t), r_t\sim R(x_t,a_t) }\left[\sum_t r_t\right]\)

1. What is Reinforcement Learning (RL)?
2. Logistics and Syllabus
3. Types of Machine Learning (ML)
4. RL Setting