Introduction to​
Reinforcement Learning

Lecture 1

Tabular Value-based RL

 

 

made by Pavel Temirchev

Background

  • Basic Machine Learning (Linear Algebra, Probability, Stochastic Optimization, Neural Networks)
     
  • Basic Scientific Python (numpy, matplotlib)
     
  • Desirable, but not mandatory: PyTorch

What problems does RL solve?

Assume we have:

  • Environment (with its state)
  • Agent
  • Agent's actions within the environment

We want an agent to act optimally in the environment.

Forgotten something?

The measure of optimality - REWARD

s_t
r_t
a_t
s_{t+1}

Agent

Environment

s_t

Example 1: Robotics (baking pancakes)

Environment state:
 Agent's actions:
 Rewards:
  • coordinates x, y, z of all joints 
  • voltage applied to all joints

Pancake flipped +1

Pancake baked +10

Pancake failed -10

Example 2: Optimal control for a well

Environment state:
 Agent's actions:
 Rewards:
  • flow characteristics around the well
  • rock properties
  • fluid properties
  • ...
  • bottom-hole pressure applied

\( r_t = q_o - \alpha q_w - \beta q_g \)

Example 3: Self-driving cars

Environment state:
 Agent's actions:
 Rewards:
  • images from the camera
  • measures from sensors
  • gas, break applied    

Destination point achieved +1

Rules break -1

Accident -10

Example 4: Chess

Environment state:
 Agent's actions:
 Rewards:
  • coordinates x, y of all figures
  • step applied for a figure

Win +1

Lose -1

Some formal definitions

s_t \in \mathcal{S}
a_t \in \mathcal{A}
r_t \in \mathbb{R}
s_{t+1} \sim p(s_{t+1}|s_t, a_t)
a_t \sim \pi(a_t|s_t)
r_t = r(s_{t+1}, s_t, a_t)

The set of the environment states

The set of agent's actions

The reward (is a scalar)

- transition probabilities

- agent's policy (behavior, strategy)

- reward function

Reminder: Tabular Definition of Functions

If the domain of function \(f\) is finite, then it can be written as a table:

f
x_0
x_1
x_2
x_3
x_4
f(x_0)
f(x_1)
f(x_2)
f(x_3)
f(x_4)
\mathcal{X}
x_0
x_1
\mathcal{X}
f(x_0, y_0)
f(x_1, y_0)
f(x_1, y_1)
f(x_0, y_1)
y_0
y_1
\mathcal{Y}
f

Markov Decision Processes (finite-time):

\tau = \{s_0, a_0, s_1, \dots, s_t, a_t, s_{t+1}, \dots, a_{T-1}, s_T \}
\mathbb{E}_{\tau \sim p(\tau)} \sum_{t=0}^T r_t \rightarrow \max_\pi
p(\tau) = \prod_{t=0}^T p(s_{t+1}|s_t, a_t) \pi(a_t|s_t)

Aim is to maximize expected cumulative reward:

Trajectory:

New states are dependent only on the previous state and action made

(not on the history!)

It is enough to make decisions solely using the current state (not the history!)

Markov Decision Processes (infinite-time):

\mathbb{E}_{\tau \sim p(\tau)} \sum_{t=0}^\infty r_t

What if the process is infinite?

T \rightarrow +\infty

Then

can be unbounded

Let's make it bounded again!

\mathbb{E}_{\tau \sim p(\tau)} \sum_{t=0}^\infty \gamma^t r_t < \infty

Introduce                                  - discount factor

0 \le \gamma < 1

Then, given bounded reward:

Discounted sum of rewards:

r_0 + \gamma r_1 + \gamma^2 r_2 + \gamma^3 r_3 + \dots

Cake today is better

then cake tomorrow

Encourages the agent to

get rewards faster!

Cake eating problem:

\mathcal{A} = \{ \text{eat}, \text{do nothing} \}
r(\text{eat}) = 1
r(\text{do nothing}) = 0

At which time-step you should eat the cake?

Episode terminates after eating

Why not Supervised Machine Learning?

Advertisement problem:

Need a dataset:

(s_0, a_0) \rightarrow p(click)
(s_0, a_1) \rightarrow p(click)
\dots
(s_N, a_N) \rightarrow p(click)

No way to take time dependencies into account!

And no dataset too!

State-value function (v-function)

V^\pi(s_t) = \mathbb{E}_{a_t} \mathbb{E}_{s_{t+1}} \Bigg[ r_t + \mathbb{E}_{a_{t+1}} \mathbb{E}_{s_{t+2}} \Big[ \gamma r_{t+1} + \mathbb{E}_{a_{t+2}} \mathbb{E}_{s_{t+3}}[\dots]\Big]\Bigg]
V^\pi(s_t) = \sum_{a_t} \pi(a_t|s_t) \sum_{s_{t+1}} p(s_{t+1}|s_t,a_t)\Bigg[ r_t + \dots \Bigg]
V^\pi(s_{T-1}) = \sum_{a_{T-1}} \pi(a_{T-1}|s_{T-1}) \sum_{s_T} p(s_T|s_{T-1},a_{T-1}) r_T
s_{T-1}
a_{T-1}
s_{T}
V^\pi(s_t) = \mathbb{E}_{a_t} \mathbb{E}_{s_{t+1}} \Bigg[ r_t + \gamma V^\pi(s_{t+1}) \Bigg]

Action-value function (q-function)

Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1}} \Bigg[ r_t + \mathbb{E}_{a_{t+1}} \mathbb{E}_{s_{t+2}} \Big[ \gamma r_{t+1} + \mathbb{E}_{a_{t+2}} \mathbb{E}_{s_{t+3}}[\dots]\Big]\Bigg]
Q^\pi(s_{T-1}, a_{T-1}) = \sum_{s_T} p(s_T|s_{T-1},a_{T-1}) r_T
s_{T-1}
a_{T-1}
s_{T}
Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1}} \Bigg[ r_t + \gamma \mathbb{E}_{a_{t+1}} Q^\pi(s_{t+1}, a_{t+1}) \Bigg]

Optimality Bellman Equation

Q^{\pi^*}(s_t, a_t) = \mathbb{E}_{s_{t+1}} \Big[ r_t + \gamma \max_a Q^{\pi^*}(s_{t+1}, a) \Big]

Theorem (kinda):

If the Q-function of a policy \(\pi^*\) for any state-action pair \( (s_t, a_t) \) is equal to:

then the policy \(\pi^*\) - is the optimal policy

and  \(Q^{\pi^*} = Q^*\) - is the optimal state-value function

If the optimal Q-function for any state-action pair \( (s_t, a_t) \) is known (it is just a table)

Then recovering optimal policy is easy: 

a_t = \arg\max_a Q^*(s_t, a)

Dynamic Programming

for Optimality Bellman Equation

The loop:

Q(s, a) \leftarrow 0 \;\;\;\; \forall(s, a)
Q(s, a) \leftarrow \mathbb{E}_{s'} \Big[ r + \gamma \max_{a'} Q(s', a') \Big]
while not convergent:
​   for      :
      for      :
s \in \mathcal{S}
a \in \mathcal{A}

Will converge to the optimal Q-function

What's wrong with this algorithm? Can you use it in practice?

It requires us to know the transition probabilities \( p(s'|s, a) \)

Q-learning

Dynamic Programming with Monte-Carlo sampling

REMINDER: Monte-Carlo estimate of an expectation

Q(s, a) \leftarrow 0 \;\;\;\; \forall(s, a)
Q(s_t, a_t) \leftarrow \alpha Q^{old}(s_t, a_t) + (1-\alpha) \big[r_t + \gamma \max_{a} Q(s_{t+1}, a) \big]

Will (with some dirty hacks) converge to the optimal Q-function

What's wrong with this algorithm? Can you use it in practice?

\mathbb{E}_{x\sim p(x)} f(x) \approx \frac{1}{N} \sum_{i=0}^N f(x_i),\;\;\;\; x_i \sim p(x)
while not convergent:
​   for t=0 to T:
a_t = \arg\max_a Q(s_t, a)
(r_t, s_{t+1}) \sim \texttt{environment}(a_t)
Q^{old}(s_t, a_t) \leftarrow Q(s_t, a_t)

will be discussed at the next slide

use exponential averaging as MC estimate

Q-learning

Exploration vs. Exploitation

Let's learn how to move forward:

x
0
r_t = x_{t+1} - x_t

Q-function estimate:

Q(s_0, a=\text{fall}) = 0
Q(s_0, a=\text{step}) = 0
\pi(a|s_0)=\arg\max_aQ(s_0, a) = \text{fall}
\Delta x
Q(s_0, a=\text{fall}) = \Delta x

Solution (\(\epsilon\)-greedy strategy): add noise to \( \pi \) :

make random action with probability \( \epsilon \)

Q-learning

Graphical representation of the learning loop

behave

collect

update \(Q\)-function

set optimal policy \(\pi\)

\pi(a|s)=\arg\max_aQ(s, a)

set behavioral policy \(\pi_b\)

\pi_b =
\{
\pi, \;\;\;\text{w.p.} \;\;(1-\epsilon)
rand, \;\;\;\text{w.p.} \;\;\epsilon

Q-learning example

Windy Gridworld Navigation Problem

wind

s_t = (x, y)
r(s_{goal}) = 1; \;\;\;\;0 \;\;else
\gamma = 0.99
\alpha = 0.9
\epsilon_0 = 1
\epsilon_{i+1} = 0.99\epsilon_i

Actions:

States:

Rewards:

Discounting:

Learning rate:

\(\epsilon\)-greedy strategy:

(decrease \(\epsilon\) after each episode)

Approximate Q-learning

(one step behind neural networks)

|\mathcal{A}|\cdot |\mathcal{S}| = 18\cdot 256^{192\times 160\times3} \approx 10^{185000} >> 10^{86}

Why do we need approximations? All is good, stop.

number of atoms in the Universe

For Atari Games:

DQN (Deep Q-Network)

Thank you for your attention!

and visit our seminars at

Deep RL Reading Group

telegram: https://t.me/theoreticalrl

R.S. Sutton and A.G. Barto, Reinforcement Learning: An Introduction

P. Shvechikov and A. Panin, Practical Reinforcement Learning

Links (clickable):

for reading:

 

online course:

 

Made with Slides.com