Using State Representation Learning with Robotic Priors
Detecting and Solving Multiple Tasks
A. Raffin, S. Höfer, R. Jonschkowski
Outline
I. Background Knowledge
II. Idea and Motivations
III. Our Method
IV. Experiments and Results
Reinforcement Learning (RL)
\pi(s) = a
r_{t}
Policy
a_t
Agent
Environment
reward
action
observation
o_{t}
s_{t}
state
Icons designed by Freepik and distributed by Flaticon
State Representation Learning (SRL)
\left( x_{agent}, y_{agent}\right)
Jonschkowski, R., & Brock, O. (2015). Learning state representations with robotic priors. Autonomous Robots, 39(3), 407-428.
observation
state
Neural Networks (1/2)
\left\{
\begin{array}{ll}
\text{Palaiseau}\\
\text{4th October 2016}\\
\text{10 a.m.}
\end{array}
\right.
f : X \rightarrow Y
Temperature ?
X
Y
\rightarrow
f = argmin_f \: \mathcal{L}(f, X, Y)
Loss function
Neural Networks (2/2)
z = \sigma(\sum_{i=1}^n w_{i} \times x_i)
Gating Connection
z = \begin{cases}
0 & \text{if g=0}\\
w_{xz}x & \text{if g=1}
\end{cases}
z = w_{(xg)z} \cdot (x \times g)
Outline
I. Background Knowledge
II. Idea and Motivations
III. Our Method
IV. Experiments and Results
Detecting and Solving Multiple Tasks (1/2)
Identify the current task
Learn task-specific state representation
Learn task-specific policies
Detecting and Solving Multiple Tasks (2/2)
Outline
I. Background Knowledge
II. Idea and Motivations
III. Our Method
IV. Experiments and Results
Learning Jointly a State Representation and a Task Detector (1/3)
Gate Layer
State
Observation
Learning Jointly a State Representation and a Task Detector (2/3)
Gate Layer
State
Observation
gate unit deactivated
1st Task
Learning Jointly a State Representation and a Task Detector (3/3)
Gate Layer
State
Observation
gate unit deactivated
2nd Task
Gated Network
Ouput of the gate layer
\sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^n e^{z_k}}
Task-Coherence Prior
Task-consistency
Task-separation
Same episode, same task
Different episode, task may change
during training, the task does not change within one episode
Losses for the Gate Layer (1/2)
Measure of similarity
Cross-entropy
\mathrm{H}(p, q) = -\sum_i p(i)\, \log q(i). \!
p,q: two probability distributions
\text{if p=q}, \quad \mathrm{H} = 0
Losses for the Gate Layer (2/2)
Separation loss
\mathrm{L_{separation}} = \exp(-\text{cross-entropy}_\text{between two episodes})
\mathrm{L_{consistency}} = \text{cross-entropy}_\text{with the same episode}
Consistency loss
Multi-Task Agent
Gate Layer
State
Observation
Action
Outline
I. Background Knowledge
II. Idea and Motivations
III. Our Method
IV. Experiments and Results
Multi-Task Slot Car Racing
Observation
Slot Car with Indicator: control the car corresponding to the color of the indicator
Slot Car with Indicator
Top down view
Input image
State Representation Learned
One task
Experimental Design
Training-Testing Cycle
Baselines
- "SRL Vanilla" (Robotic Priors Only)
- Principal Component Analysis
- Raw Observations
- Variations of our method
1. 4 training episodes of 100 steps each (2 per task)
2. Learn state representation + policy (KNN-Q)
3. Test on 10 episodes of 100 steps
Average reward per episode
10 test episodes (5 per task)
Results: Slot Car with Indicator
10 test episodes (5 per task)
Results: Slot Car Blue
Slot Car Blue: control the blue car
Results: What was learned
5 gate units
Weights Gate Layer
Weights Gate Tensor
Weights SRL Vanilla
Influence of Losses
Conclusion
Task-Specific State Representation
Task Detector
Transfer Learning
Task-Specific Policies
Future Work
Learning from experience
Output of the gate layer
h_g = softmax(\sum\limits_{i=1}^{n_{inputs}} v_{ig} \times x_i)
\sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^n e^{z_k}}
Softmax:
Maths details
v_{ig} : \text{weights from inputs to gate layer}
X : inputs
Maths details
Weight matrix gate (toy-task x-axis y-axis)
\begin{pmatrix}
0 & 0\\
0 & 0\\
0 & 0\\
0 & 0\\
-10 & 10
\end{pmatrix}
\begin{pmatrix}
x_{agent}\\
y_{agent}\\
x_{object}\\
y_{object}\\
bit_{task}
\end{pmatrix}
1st unit
2nd unit
Maths details
Output of the gate layer
h = \begin{pmatrix}
out_{unit1} & out_{unit2}
\end{pmatrix}
out_{unit1} + out_{unit2} = 1
Softmax:
\begin{pmatrix}
0 & 1
\end{pmatrix}
\begin{pmatrix}
1 & 0
\end{pmatrix}
Task 0
Task 1
Ex:
Maths details
Final Output (state representation)
state = \sum\limits_{k=1}^{n_{outputs}} \sum\limits_{g=1}^{n_{gates}} \sum\limits_{i=1}^{n_{inputs}} h_g \times w_{igk} \times x_i
h_g : \text{output of the gate unit g}
W : \text{tensor with the weights from inputs to output}
X : inputs
Maths details
Tensor example (toy-task x-axis y-axis)
\begin{pmatrix}
\begin{pmatrix}
1 \\
0
\end{pmatrix}\\\\
\begin{pmatrix}
0 \\
1
\end{pmatrix}\\\\
\begin{pmatrix}
0 \\
0
\end{pmatrix}\\\\
\begin{pmatrix}
0 \\
0
\end{pmatrix}\\\\
\begin{pmatrix}
0 \\
0
\end{pmatrix}
\end{pmatrix}
\begin{pmatrix}
x_{agent}\\\\
y_{agent}\\\\
x_{object}\\\\
y_{object}\\\\
bit_{task}
\end{pmatrix}
\begin{pmatrix}
1 & 0
\end{pmatrix}
Task 0
Ouput gate layer
\begin{pmatrix}
0 & 1
\end{pmatrix}
Task 1
Detecting And Solving Multiple Tasks
By Antonin Raffin
