Curiosity-driven Exploration
by Self-Supervised
Prediction
report is made by
Pavel Temirchev
Deep RL
reading group
Reinforcement Learning preliminaries
AGENT
ENVIRONMENT
ACTION
OBSERVATION
REWARD
Reinforcement Learning preliminaries
at∈A
a_t \in \mathcal{A}
st∈S
s_t \in \mathcal{S}
rt=r(st,at)∈R
r_t = r(s_t, a_t) \in \mathbb{R}
st+1∼p(st+1∣st,at)
s_{t+1} \sim p(s_{t+1}|s_t, a_t)
Markov Decision Process:
at∼π(at∣st)
a_{t} \sim \pi(a_{t}|s_t)
- the set of actions
- the set of states of the environment
- reward function
- the policy
- transition probability
Reinforcement Learning preliminaries
Policy Gradient Methods:
We want to maximize the expected return:
R(θ)=Eπ(θ)∑trt→maxθ
R(\theta) = \mathbb{E}_{\pi(\theta)} \sum_t r_t \rightarrow \max_\theta
We parametrize the policy:
π(at∣st)=π(at∣st,θ)
\pi(a_t|s_t) = \pi(a_t|s_t, \theta)
Maximization is done by gradient ascent:
θ←θ+α∇θR
\theta \leftarrow \theta + \alpha\nabla_\theta R
Reinforcement Learning preliminaries
Advantage Actor-Critic Method:
Where A(a,s) - advantage function
Gradients:
∇θR=Eπ(θ)∇θlogπ(a∣s,θ)A(a,s)
\nabla_\theta R = \mathbb{E}_{\pi(\theta)} \nabla_\theta \log \pi(a|s,\theta)A(a, s)
A3C method was used in this work
For more see: https://arxiv.org/pdf/1602.01783.pdf
Problem Description
Commonly used exploration strategies:
- ϵ-greedy strategy
Take an action at=argmaxπ(at∣st) with probability 1−ϵ
Or take a random action with ϵ probability
- Boltzmann strategy
Take an action at with probability ∼exp(π(at∣st)/T)
Problem Description
These exploration strategies are not efficient if rewards are sparce
It is almost impossible to perform long complex sequence of actions
by random exploration
Curiosity-driven Exploration
Good exploration strategy should
encourage agent to:
- Explore 'novel' states
- Perform actions that reduce the uncertainty about the environment
Curiosity-driven Exploration
Сам себя не похвалишь - никто не похвалит
Curiosity-driven Exploration
rt=rti+rte
r_t = r^i_t + r^e_t
rti
r^i_t
rte
r^e_t
- the intrinsic reward
- the extrinsic reward
mostly, if not always, zero
- total reward
Curiosity-driven Exploration
Agent is composed of two modules:
- Generator of the intrinsic reward rti
- Policy π(θ) that outputs actions
Prediction error as curiosity reward
rti is based on how hard it is for the agent to predict the consequences of it's own actions
We need a model of the environmental dynamics that predicts st+1 given st and at
Prediction error as curiosity reward
Prediction in a raw state space S
is not efficient
Not all changes in the environment depend on agent's actions
or
affect the agent
Prediction error as curiosity reward
Inverse Dynamic Model
Forward Dynamic Model
Transforms raw state representation st
into ϕ(st) which depend only on such a parts of st that need to be controlled by agent
Tries to predict ϕ(st+1) given ϕ(st) and at
Inverse dynamics model
st+1
s_{t+1}
st
s_{t}
ϕ(st+1)
\phi (s_{t+1})
ϕ(st)
\phi (s_{t})
a^t
\hat{a}_t
a^t=g(st+1,st,θI)
\hat{a}_t = g(s_{t+1}, s_t, \theta_I)
minθILI(a^t,at)
\min_{\theta_I} L_I(\hat{a}_t, a_t)
LI
L_I
- cross entropy
Forward dynamics model
ϕ(st)
\phi (s_{t})
at
a_t
ϕ^(st+1)
\hat{\phi}(s_{t+1})
ϕ^(st+1)=f(st,a,θF)
\hat{\phi}(s_{t+1}) = f(s_t, a, \theta_F)
minθFLF(ϕ^(st+1),ϕ(st+1))
\min_{\theta_F} L_F(\hat{\phi}(s_{t+1}), \phi(s_{t+1}))
LF(ϕ^t+1,ϕt+1)=21∣∣ϕ^t+1−ϕt+1∣∣22
L_F(\hat{\phi}_{t+1}, \phi_{t+1}) = \frac{1}{2}||\hat{\phi}_{t+1} - \phi_{t+1}||^2_2
Intrinsic Curiosity Module (ICM)
rti=2η∣∣ϕ^t+1−ϕt+1∣∣22
r^i_t = \frac{\eta}{2}||\hat{\phi}_{t+1} - \phi_{t+1}||^2_2
η>0
\eta > 0
- the tradeoff between extrinsic and intrinsic reward
rt=rti+rte
r_t = r_t^i + r^e_t
- total reward
Optimization problem
L(θ)=[−λEπθP∑trt+βLF+(1−β)LI]
L(\theta) = \big[-\lambda \mathbb{E}_{\pi_{\theta_{P}}} \sum_t r_t + \beta L_F + (1-\beta)L_I \big]
θ={θP,θI,θF}
\theta = \{\theta_P, \theta_I, \theta_F\}
λ>0
\lambda > 0
0≤β≤1
0 \le \beta \le 1
- the tradeoff between policy gradient loss and intrinsic reward learning
L(θ)→minθ
L(\theta) \rightarrow\min_\theta
- the tradeoff between forward and inverse dynamics model learning
Experimental Setup
Environments
Doom 3-D navigation task
- DoomMyWayHome-v0 from OpenAI Gym
- Actions: left, right, forward and no-action
- Termination after reaching the goal or 2100 timesteps
- Sparse termination +1 reward and zero otherwise
Experimental Setup
Environments
Doom 3-D navigation task
Experimental Setup
Environments
Super Mario Bros
- Training on the first level and looking for generalization on the three subsequent levels
- 14 unique actions following:
Experimental Setup
Training Details
- RGB images → Grey-Scale 42x42 images
- st ← tensor of 4 last frames
- A3C was used with 12 workers and ADAM optimizer (parameters not shared)
- Action repeat of 4 in Doom training
- Action repeat of 6 in SuperMario training
- No action repeat at a testing time
Experimental Setup
Training Details
Policy and Value
st→ 4 Conv Layers:
- 32 filters
- 3x3 kernel size
- stride = 2
- padding = 1
- ELU nonlinearity
Conv output → LSTM:
- 256 units
Then two separate fully-connected layers for π(at∣st) and V(st)
Experimental Setup
Training Details
ELU={x,x≥0α(exp(x)−1),otherwise
ELU =
\begin{cases}
x, x \ge 0 \\
\alpha(\exp(x) - 1), otherwise
\end{cases}
Experimental Setup
Training Details
Experimental Setup
Training Details
Intrinsic Curiosity Module (ICM)
st converted to ϕ(st) using:
- 4 Conv Layers
- 32 filters each
- 3x3 kernel size
- stride = 2
- padding = 1
- ELU nonlinearity
ϕ(st) dimensionality is 288
Experimental Setup
Training Details
Intrinsic Curiosity Module (ICM)
Inverse Model
Forward Model
ϕt and ϕt+1 are concatenated and passed through FC layer of 256 units followed by an output softmax layer for a^t prediction
at and ϕt are concatenated and passed through FC layer of 256 units follower by an FC layer of 288 units for ϕ^t+1 prediction
Experiments
Doom: DENSE reward setting
Experiments
Doom: SPARSE reward setting
Experiments
Doom: VERY SPARSE reward setting
Experiments
Doom: Robustness to the noise
Input with noise sample:
Experiments
Doom: Robustness to the noise
Experiments
Comparison to the TRPO-VIME
For the SPARSE Doom reward setting
| Method | Mean (Median) Score at convergence |
|---|---|
| TRPO | 26.0% (0.0%) |
| A3C | 0.0% (0.0%) |
| VIME + TRPO | 46.1% (27.1%) |
| ICM + A3C | 100% (100%) |
Experiments
NO REWARD setting and GENERALIZATION
Experiments
Fine-tuning for unseen scenarios
Links
Blog (video is here):
Article:
Thanks for your
attention!
Curiosity-driven Exploration by Self-Supervised Prediction report is made by Pavel Temirchev Deep RL reading group
Curiosity-driven Exploration by Self-Supervised Prediction
By cydoroga
Curiosity-driven Exploration by Self-Supervised Prediction
- 745
