CS 4/5789: Introduction to Reinforcement Learning
Lecture 1: Introduction
Prof. Sarah Dean
MW 1:25-2:40pm
155 Olin Hall
Agenda
1. What is Reinforcement Learning (RL)?
2. Logistics and Syllabus
3. Types of Machine Learning (ML)
4. RL Setting
AlphaGo
Robotic Manipulation
Algorithmic Media Feeds
Chatbots and Assistants
...
observation
action
reward
a policy maps observation to action
design policy to achieve high reward
RL is for Decision-Making
reaction
adaptation
Decision-Making
observation
action
reward
AlphaGo
\(\theta_t-\theta_*\)
Robotic Manipulation
Media Feeds
Chatbot
?
?
?
?
?
?
👍 👎
?
?
?
💬
Agenda
1. What is Reinforcement Learning (RL)?
2. Logistics and Syllabus
3. Types of Machine Learning (ML)
4. RL Setting
Logistics
- Instructor: Prof. Sarah Dean
- Grad TAs: Yujin Kim, Akanksha Sarkar, Haruka Kiyohara
- TAs: Guohao Chen, Bodong Liu, Jason Wu, Alex Noviello, Aidan Wu, Myles Pasetsky, Leon Jiao
- Contact: Ed Discussion
- Instructor Office Hours: TBA
- TA Office Hours: TBA, see Ed Discussion
Waitlist and Enrollment
There should be plenty of space!
Course staff do not manage waitlist and enrollment.
CS enrollment policies:
https://www.cs.cornell.edu/courseinfo/enrollment
Exams
- Two in class, closed book prelims (15% of final grade each)
- Wednesday September 23rd
- Wednesday October 28th
- Final exam during finals period, time TBD (20% of grade)
Homework
- Homework assignments (30% of final grade)
- ~5 problem sets (math), groups of up to four
- Individual oral peer assessments
- ~5 programming projects (coding), groups of up to two
- Autograded tests + TA graded report
- ~5 problem sets (math), groups of up to four
- 5789: Paper review assignments (after Unit 1)
- Collaboration Policy: outside of formal group, you may discuss problems but do not look at, copy, or lightly adapt a completed solution (including AI).
- Late Policy: six slip days, use no more than three at once (see Syllabus)
Oral Peer Assessment
- Goal: assess and encourage real understanding, not just correct looking solutions
- Assessing your peers: 10% of final grade
- Meet with small group of peers (randomly assigned)
- Each student explains their solution and answers questions about their reasoning
- Peers determine pass/fail assessment and provide qualitative feedback on level of understanding
Participation
Participation is 10% of final grade, /20 points
- Lecture participation = 1pt each
- Poll Everywhere: PollEv.com/sarahdean011
- Helpful posts on Ed Discussions = 2pt each
- TA endorsement
Schedule
- Unit 1: Policy Optimization
- Estimation, Iterative Optimization, Policy Gradients
- Unit 2: Markov Decision Processes
- Value Iteration, Policy Iteration, Learning from Data
- Unit 3: Exploration
-
Bandits, Receding Horizon Planning, Learning from Experts
-
Prerequisites
Machine learning (e.g., CS 3780)
Background in probability, linear algebra, and programming.
Materials
Lecture Slides and Notes on Canvas
Agenda
1. What is Reinforcement Learning (RL)?
2. Logistics and Syllabus
3. Types of Machine Learning (ML)
4. RL Setting
Types of Machine Learning
- Unsupervised Learning
- Supervised Learning
- Reinforcement Learning
Unsupervised Learning
Examples: clustering, PCA, image generation
- Goal:
- summarization
- Dataset:
- information about many instances
- \(\{x_1, x_2, \dots x_N\}\)
- Evaluation:
- qualitative
"descriptive"
Supervised Learning
Examples: classification, regression, conditional generation
- Goal:
- prediction
- Dataset:
- each instance has features and label
- \(\{(x_1, y_1), \dots (x_N,y_N)\}\)
- Evaluation:
- accuracy, \(y\) vs. \(\hat y\)
"predictive"
Reinforcement Learning
- Goal:
- action or decision
- Dataset:
- history of observations, actions, and rewards
- \(\{(x_1, a_1, r_1),...,(x_N, a_N, r_N)\}\)
- may be sequentially dependent
- Evaluation:
- cumulative reward
"presciptive"
Types of Machine Learning
- Unsupervised Learning (descriptive)
- summarize unstructured data \(\{x_i\}_{i=1}^N\)
- Supervised Learning (predictive)
- predict labels from features \(\{(x_i, y_i)\}_{i=1}^N\)
- Reinforcement Learning (prescriptive)
- choose actions that lead to high reward \(\{(x_i, a_i, r_i)\}_{i=1}^N\)
- data depends on the decisions
Difficulties of Decision-Dependence
Unlike other types of ML, in RL data is not just drawn "i.i.d." from some distribution
- May start with no data
- Actions have consequences
- Solving task may require long sequence of correct actions
\(a_t\)
\(x_t, r_t\)
\(a_{t+1}\)
\(x_{t+1}, r_{t+1}\)
\(...\)
ML Specification
- Specifying supervised learning problem
- feature space \(\mathcal X\) and label space \(\mathcal Y\)
- distribution over feature and labels \((x,y)\sim \mathcal D\)
- often empirical, i.e. defined by a dataset
- pick a loss function to determine accuracy
Agenda
1. What is Reinforcement Learning (RL)?
2. Logistics and Syllabus
3. Types of Machine Learning (ML)
4. RL Setting
RL Setting
action
observation (context)
\(a_t\)
reward
\(x_t\)
\(r_t\)
- Agent observes environment
- Agent takes action
- according to policy \(\pi\)
- Environment returns reward
- dependent on context/action
- Repeat
\(\sim R(x_t,a_t)\)
\(\sim \pi(x_t)\)
\(\sim \mathcal D\) (for now)
Goal: achieve high cumulative reward: \(\sum_t r_t\)
\(\displaystyle \max_\pi ~~ \mathbb E_{x_t\sim \mathcal D, a_t\sim \pi(x_t), r_t\sim R(x_t,a_t) }\left[\sum_t r_t\right]\)
Example
- context: \(x\sim \mathcal D\)
- user profile (demographics, view history)
- distribution \(\mathcal D\) is the user base
- action: \(a\)
- recently released movie
- policy: \(\pi(x)\)
- movie recommendation (personalized)
- reward: \(r\sim R(x,a)\)
- whether user clicks on recommendation
- distribution \(R(x,a)\) depends on user and movie
personalized Netflix banner
Recap
action
observation (context)
\(a_t\)
reward
\(x_t\)
\(r_t\)
\(\sim R(x_t,a_t)\)
\(\sim \pi(x_t)\)
\(\sim \mathcal D\) (for now)
\(\displaystyle \max_\pi ~~ \mathbb E_{x_t\sim \mathcal D, a_t\sim \pi(x_t), r_t\sim R(x_t,a_t) }\left[\sum_t r_t\right]\)
- What is Reinforcement Learning (RL)?
- Logistics and Syllabus
- Types of Machine Learning (ML)
- RL Setting