Introduction to
Reinforcement Learning
part 1 (probably the single one)
Tabular Value-based RL
made by Pavel Temirchev
Deep RL
reading group
What problems does RL solve?
Assume we have:
- Environment (with its state)
- Agent
- Agent's actions within the environment
We want an agent to act optimally in the environment.
Forgotten something?
The measure of optimality - REWARD
st
s_t
rt
r_t
at
a_t
st+1
s_{t+1}
Agent
Environment
st
s_t
Example 1: Robotics (baking pancakes)
Environment state:
Agent's actions:
Rewards:
- coordinates x, y, z of all joints
- voltage applied to all joints
Pancake flipped +1
Pancake baked +10
Pancake failed -10
Example 2: Optimal control for a well
Environment state:
Agent's actions:
Rewards:
- flow characteristics around the well
- rock properties
- fluid properties
- ...
- bottom-hole pressure applied
rt=qo−αqw−βqg
Example 3: Self-driving cars
Environment state:
Agent's actions:
Rewards:
- images from the camera
- measures from sensors
- gas, break applied
Destination point achieved +1
Rules break -1
Accident -10
Example 4: Chess
Environment state:
Agent's actions:
Rewards:
- coordinates x, y of all figures
- step applied for a figure
Win +1
Lose -1
Some formal definitions
st∈S
s_t \in \mathcal{S}
at∈A
a_t \in \mathcal{A}
rt∈R
r_t \in \mathbb{R}
st+1∼p(st+1∣st,at)
s_{t+1} \sim p(s_{t+1}|s_t, a_t)
at∼π(at∣st)
a_t \sim \pi(a_t|s_t)
rt=r(st+1,st,at)
r_t = r(s_{t+1}, s_t, a_t)
The set of the environment states
The set of agent's actions
The reward (is a scalar)
- transition probabilities
- agent's policy (behavior, strategy)
- reward function
Reminder: Tabular Definition of Functions
If the domain of function f is finite, then it can be written as a table:
f
f
x0
x_0
x1
x_1
x2
x_2
x3
x_3
x4
x_4
f(x0)
f(x_0)
f(x1)
f(x_1)
f(x2)
f(x_2)
f(x3)
f(x_3)
f(x4)
f(x_4)
X
\mathcal{X}
x0
x_0
x1
x_1
X
\mathcal{X}
f(x0,y0)
f(x_0, y_0)
f(x1,y0)
f(x_1, y_0)
f(x1,y1)
f(x_1, y_1)
f(x0,y1)
f(x_0, y_1)
y0
y_0
y1
y_1
Y
\mathcal{Y}
f
f
Markov Decision Processes (finite-time):
τ={s0,a0,s1,…,st,at,st+1,…,aT−1,sT}
\tau = \{s_0, a_0, s_1, \dots, s_t, a_t, s_{t+1}, \dots, a_{T-1}, s_T \}
Eτ∼p(τ)∑t=0Trt→maxπ
\mathbb{E}_{\tau \sim p(\tau)} \sum_{t=0}^T r_t \rightarrow \max_\pi
p(τ)=∏t=0Tp(st+1∣st,at)π(at∣st)
p(\tau) = \prod_{t=0}^T p(s_{t+1}|s_t, a_t) \pi(a_t|s_t)
Aim is to maximize expected cumulative reward:
Trajectory:
New states are dependent only on the previous state and action made
(not on the history!)
It is enough to make decisions solely using the current state (not the history!)
Markov Decision Processes (infinite-time):
Eτ∼p(τ)∑t=0∞rt
\mathbb{E}_{\tau \sim p(\tau)} \sum_{t=0}^\infty r_t
What if the process is infinite?
T→+∞
T \rightarrow +\infty
Then
can be unbounded
Let's make it bounded again!
Eτ∼p(τ)∑t=0∞γtrt<∞
\mathbb{E}_{\tau \sim p(\tau)} \sum_{t=0}^\infty \gamma^t r_t < \infty
Introduce - discount factor
0≤γ<1
0 \le \gamma < 1
Then, given bounded reward:
Discounted sum of rewards:
r0+γr1+γ2r2+γ3r3+…
r_0 + \gamma r_1 + \gamma^2 r_2 + \gamma^3 r_3 + \dots
Cake today is better
then cake tomorrow
Encourages the agent to
get rewards faster!
Cake eating problem:
A={eat,do nothing}
\mathcal{A} = \{ \text{eat}, \text{do nothing} \}
r(eat)=1
r(\text{eat}) = 1
r(do nothing)=0
r(\text{do nothing}) = 0
At which time-step you should eat the cake?
Episode terminates after eating
Why not Supervised Machine Learning?
Advertisement problem:
Need a dataset:
(s0,a0)→p(click)
(s_0, a_0) \rightarrow p(click)
(s0,a1)→p(click)
(s_0, a_1) \rightarrow p(click)
…
\dots
(sN,aN)→p(click)
(s_N, a_N) \rightarrow p(click)
No way to take time dependencies into account!
And no dataset too!
State-value function (v-function)
Vπ(st)=EatEst+1[rt+Eat+1Est+2[γrt+1+Eat+2Est+3[…]]]
V^\pi(s_t) = \mathbb{E}_{a_t} \mathbb{E}_{s_{t+1}} \Bigg[ r_t + \mathbb{E}_{a_{t+1}} \mathbb{E}_{s_{t+2}} \Big[ \gamma r_{t+1} + \mathbb{E}_{a_{t+2}} \mathbb{E}_{s_{t+3}}[\dots]\Big]\Bigg]
Vπ(st)=∑atπ(at∣st)∑st+1p(st+1∣st,at)[rt+…]
V^\pi(s_t) = \sum_{a_t} \pi(a_t|s_t) \sum_{s_{t+1}} p(s_{t+1}|s_t,a_t)\Bigg[ r_t + \dots \Bigg]
Vπ(sT−1)=∑aT−1π(aT−1∣sT−1)∑sTp(sT∣sT−1,aT−1)rT
V^\pi(s_{T-1}) = \sum_{a_{T-1}} \pi(a_{T-1}|s_{T-1}) \sum_{s_T} p(s_T|s_{T-1},a_{T-1}) r_T
sT−1
s_{T-1}
aT−1
a_{T-1}
sT
s_{T}
Vπ(st)=EatEst+1[rt+γVπ(st+1)]
V^\pi(s_t) = \mathbb{E}_{a_t} \mathbb{E}_{s_{t+1}} \Bigg[ r_t + \gamma V^\pi(s_{t+1}) \Bigg]
Action-value function (q-function)
Qπ(st,at)=Est+1[rt+Eat+1Est+2[γrt+1+Eat+2Est+3[…]]]
Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1}} \Bigg[ r_t + \mathbb{E}_{a_{t+1}} \mathbb{E}_{s_{t+2}} \Big[ \gamma r_{t+1} + \mathbb{E}_{a_{t+2}} \mathbb{E}_{s_{t+3}}[\dots]\Big]\Bigg]
Qπ(sT−1,aT−1)=∑sTp(sT∣sT−1,aT−1)rT
Q^\pi(s_{T-1}, a_{T-1}) = \sum_{s_T} p(s_T|s_{T-1},a_{T-1}) r_T
sT−1
s_{T-1}
aT−1
a_{T-1}
sT
s_{T}
Qπ(st,at)=Est+1[rt+γEat+1Qπ(st+1,at+1)]
Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1}} \Bigg[ r_t + \gamma \mathbb{E}_{a_{t+1}} Q^\pi(s_{t+1}, a_{t+1}) \Bigg]
Optimality Bellman Equation
Qπ∗(st,at)=Est+1[rt+γmaxaQπ∗(st+1,a)]
Q^{\pi^*}(s_t, a_t) = \mathbb{E}_{s_{t+1}} \Big[ r_t + \gamma \max_a Q^{\pi^*}(s_{t+1}, a) \Big]
Theorem (kinda):
If the Q-function of a policy π∗ for any state-action pair (st,at) is equal to:
then the policy π∗ - is the optimal policy
and Qπ∗=Q∗ - is the optimal state-value function
If the optimal Q-function for any state-action pair (st,at) is known (it is just a table)
Then recovering optimal policy is easy:
at=argmaxaQ∗(st,a)
a_t = \arg\max_a Q^*(s_t, a)
Dynamic Programming
for Optimality Bellman Equation
The loop:
Q(s,a)←0∀(s,a)
Q(s, a) \leftarrow 0 \;\;\;\; \forall(s, a)
Q(s,a)←Es′[r+γmaxa′Q(s′,a′)]
Q(s, a) \leftarrow \mathbb{E}_{s'} \Big[ r + \gamma \max_{a'} Q(s', a') \Big]
while not convergent:
for :
for :
s∈S
s \in \mathcal{S}
a∈A
a \in \mathcal{A}
Will converge to the optimal Q-function
What's wrong with this algorithm? Can you use it in practice?
It requires us to know the transition probabilities p(s′∣s,a)
Q-learning
Dynamic Programming with Monte-Carlo sampling
REMINDER: Monte-Carlo estimate of an expectation
Q(s,a)←0∀(s,a)
Q(s, a) \leftarrow 0 \;\;\;\; \forall(s, a)
Q(st,at)←αQold(st,at)+(1−α)[rt+γmaxaQ(st+1,a)]
Q(s_t, a_t) \leftarrow \alpha Q^{old}(s_t, a_t) + (1-\alpha) \big[r_t + \gamma \max_{a} Q(s_{t+1}, a) \big]
Will (with some dirty hacks) converge to the optimal Q-function
What's wrong with this algorithm? Can you use it in practice?
Ex∼p(x)f(x)≈N1∑i=0Nf(xi),xi∼p(x)
\mathbb{E}_{x\sim p(x)} f(x) \approx \frac{1}{N} \sum_{i=0}^N f(x_i),\;\;\;\; x_i \sim p(x)
while not convergent: for t=0 to T:
at=argmaxaQ(st,a)
a_t = \arg\max_a Q(s_t, a)
(rt,st+1)∼environment(at)
(r_t, s_{t+1}) \sim \texttt{environment}(a_t)
Qold(st,at)←Q(st,at)
Q^{old}(s_t, a_t) \leftarrow Q(s_t, a_t)
will be discussed at the next slide
use exponential averaging as MC estimate
Q-learning
Exploration vs. Exploitation
Let's learn how to move forward:
x
x
0
0
rt=xt+1−xt
r_t = x_{t+1} - x_t
Q-function estimate:
Q(s0,a=fall)=0
Q(s_0, a=\text{fall}) = 0
Q(s0,a=step)=0
Q(s_0, a=\text{step}) = 0
π(a∣s0)=argmaxaQ(s0,a)=fall
\pi(a|s_0)=\arg\max_aQ(s_0, a) = \text{fall}
Δx
\Delta x
Q(s0,a=fall)=Δx
Q(s_0, a=\text{fall}) = \Delta x
Solution (ϵ-greedy strategy): add noise to π :
make random action with probability ϵ
Q-learning
Graphical representation of the learning loop
behave
collect
update Q-function
set optimal policy π
π(a∣s)=argmaxaQ(s,a)
\pi(a|s)=\arg\max_aQ(s, a)
set behavioral policy πb
πb=
\pi_b =
{
\{
π,w.p.(1−ϵ)
\pi, \;\;\;\text{w.p.} \;\;(1-\epsilon)
rand,w.p.ϵ
rand, \;\;\;\text{w.p.} \;\;\epsilon
Q-learning example
Windy Gridworld Navigation Problem
wind
st=(x,y)
s_t = (x, y)
r(sgoal)=1;0else
r(s_{goal}) = 1; \;\;\;\;0 \;\;else
γ=0.99
\gamma = 0.99
α=0.9
\alpha = 0.9
ϵ0=1
\epsilon_0 = 1
ϵi+1=0.99ϵi
\epsilon_{i+1} = 0.99\epsilon_i
Actions:
States:
Rewards:
Discounting:
Learning rate:
ϵ-greedy strategy:
(decrease ϵ after each episode)
Approximate Q-learning
(one step behind neural networks)
∣A∣⋅∣S∣=18⋅256192×160×3≈10185000>>1086
|\mathcal{A}|\cdot |\mathcal{S}| = 18\cdot 256^{192\times 160\times3} \approx 10^{185000} >> 10^{86}
Why do we need approximations? All is good, stop.
number of atoms in the Universe
For Atari Games:
DQN (Deep Q-Network)
Thank you for your attention!
Links (clickable):
for reading:
online course:
Introduction to Reinforcement Learning part 1 (probably the single one) Tabular Value-based RL made by Pavel Temirchev Deep RL reading group
Intro to RL, lecture 2: Q-learning (ISP)
By cydoroga
Intro to RL, lecture 2: Q-learning (ISP)
- 576
