Supervised Learning Problem

Supervised Learning case:

Given Dataset                                   

Learn a function that will predict y from X: 

e. g. find parameters theta that will minimize:                                where      is a loss function  

Standart Assumptions:

• Samples in Dataset are I.I.D
• We have ground truth labels

No ground truth answers

You don't have answers at all

No ground truth answers

Your answers are not good enough 

Choice matters

Assume that we have expert trajectories, i.e. sufficiently good answers:

• Treat trajectories as a dataset:                             \(D = \{(x_1, a_1), .. (x_N, a_N)\}\) 
• Train with Supervised Learning

Will this work?

Choice matters

Assume that we have expert trajectories, i.e. sufficiently good answers:

Yes, but only if for any possible behavior there is a close example from the data

• Treat trajectories as a dataset:                             \(D = \{(x_1, a_1), .. (x_N, a_N)\}\) 
• Train with Supervised Learning

Will this work?

Choice matters

New Plan (DAGGER algorithm):

1. Train a model from human trajectories :         \(D_0 = \{(x_1, a_1), .. (x_N, a_N)\}\)
2. Run the model to get new tajectories:              \(D' = \{(x_1, ?), .. (x_N, ?)\}\)
3. Ask humans to label \(D'\) with actions \(a_t\)
4. Aggregate:
5. Repeat

Choice matters

But this is really hard to do:     3. Ask humans to label \(D'\) with actions \(a_t\)

Huge datasets and huge models 

Do we still need Reinforcement Learning?

Impressive results in image and text generation with SL!

Huge datasets and huge models 

SL results

RL results

Reinforcement learning

If You know what you want, but don't know how to do it...    USE REWARDS!

WIN: +1

LOSE: -1

Assumptions:

• It's easy to compute reward
• You can express your goals with rewards!

Reinforcement Learning Problem

You have Agent and Environment that interact with each other:

• Agent's actions change the state environment
• After each action agent receives new state and reward

Interaction with environment  is typically divided into episodes.

Reinforcement Learning Problem

Agent has a policy:

Agent learns its policy via Trial and Error!

The goal is to find a policy that maximizes total expected reward:

Reinforcement Learning Problem

Environment and Observation

What should an agent observe?

• Wheel speed
• Acceleration
• LiDAR
• Battery

• Map of the apartment
• Location

Markovian Propery

Task: Open the red door with the key

Details: Agent starts at random location

Actions:

• move up/left/right/down
• pick up an object
• apply an object to the door              (when near the door)

Markovian Propery

Which observations are enough to learn the optimal policy?

1. Agent's coordinates, and previous action
2. Full image of the maze
3. Agent's coordinates and does it has key

Markov Decision Process

MDP is a 5-tuple \(<S,A,R,T, \gamma >\):

• \(S\) is a set of states
• \(A\) is a set of actions
• \(R: S \times A \to \mathbb{R}\) is a reward function
• \(T: S \times A \times S \to [0,1]\) is a transition function \(T(s,a,s\prime) =  P(S_{t+1}=s\prime|S_t=s, A_t=a)\)
• \(\gamma \in [0, 1] \) is a discount factor

Given Agent's policy \(\pi\), RL objective become:

Discount factor \(\gamma\) determines how much we should care about the future!

Multi-Armed Bandits

MDP for Multi-Armed Bandits:

1. Only one state                                           
2. Rewards are imediate (follows from 1.)

Reinforcement Learning: SL as RL

You can formulate SL problem as RL problem!

Given Dataset:  

We consider X_i as states, and y_i as correct actions!

Then the reward function will be \(R(X_i, a_i)  = 1\). if \(a_i = y_i\) else 0.  

Exploration-Exploitation Dilemma

fox

bread

truck

dog

0.

0.

0.

1.

We have ground truth labels

bread

-3

We have rewards

Reward Specification Problem

Reward is a proxy for you goal, but they are not the same!

Goal: Train a bot to win the game!

Rewards: 

• +100 for the first place
• +5 for additional targets along the course

Credit Assignment Problem

Your data is not i.i.d. Previous actions affect future states and rewards.

Credit Assignment Problem:

                            How to determine which actions are responsible for the outcome? 

• Agent makes a move at step 8
• At step 50 agent looses: R = -1
• Was it a good move?

Distributional shift: In case of Deep RL

The training dataset is changing with the policy.

This can lead to a catastrophic forgetting problem:

               Agent unlearns it's policy in some parts of the State Space

Conclusion

• Reinforcement Learning uses less feedback information than Supervised Learning
• RL allows us to approach a wider set of tasks
• RL is a harder problem with it's own additional challenges