Introduction to intrinsic motivations

Martin Biehl

  1. Background / introduction to intrinsic motivations
  2. Formal framework for intrinsic motivations
    1. Generative model
    2. Prediction / complete posterior
    3. Action selection
  3. Some intrinsic motivations:
    1. Free Energy Minimization
    2. Predictive information maximization
    3. Knowledge seeking
    4. Empowerment maximization
    5. Curiosity

Overview

Originally from psychology e.g. (Ryan and Deci, 2000):


activity for its inherent satisfaction rather than separable consequence


for the fun or challenge entailed rather than because of external products, pressures or reward


Examples (Oudeyer, 2008):

  • infants grasping, throwing, biting new objects,
  • adults playing crosswords, painting, gardening, reading novels...

Background on intrinsic motivations

But can always argue:

  • these things possibly increase the probability of survival in some way
  • "learned" by evolution
  • cannot be sure they have no purpose

 

 

Background on intrinsic motivations

Motivation is something that generates behavior for an agent (robot, living organism)

  • similar to reward function in reinforcement learning (RL)

Background on intrinsic motivations

Working definition compatible with Oudeyer (2008):

Motivation is intrinsic if it:

  • ``rewiring agnostic''.
  • embodiment independent,
  • semantic free / information theoretic,

This includes the approach by Schmidhuber (2010):

Motivation is intrinsic if it

  • rewards improvement of some model quality measure.

Background on intrinsic motivations

rewiring agnostic means

  • if we rewire the sensors or rewire the actuators the intrinsic motivation will still lead to similar behaviour

this implies

  • cannot assume that there is a special sensor that corresponds to reward
  • so reward functions of MDPs, POMDPs, and standard RL are not available

 

Background on intrinsic motivations

Embodiment independent means it should work (without changes) for any form of agent:

 

Background on intrinsic motivations

and produce "worthwhile" behavior

Embodiment independent means it should work (without changes) for any form of agent:

 

Background on intrinsic motivations

  • Should work for any number or kind of
    • sensors
    • actuators

Semantic free, information theoretic:

  • relations between sensors, actuators, and internal variables count
  • specific values don't


 

  • information theory quantifies relations
  • if \(f\) and \(g\) are bijective functions then $$\text{I}(X:Y)=\text{I}(f(X):g(Y))$$
  • so the values of \(X\) or \(Y\) can play no role in mutual information.

 

Background on intrinsic motivations

Background on intrinsic motivations

Another important but not defining feature is usually known from evolution:

 

Background on intrinsic motivations

Another important but not defining feature is usually known from evolution:

open endedness

 

The motivation should not vanish until the capacities of the agent are exhausted.

Background on intrinsic motivations

Applications of intrinsic motivations:

  • developmental robotics
  • sparse reward reinforcement learning problems
  • Human level and artificial general AI

Background on intrinsic motivations

Developmental robotics:

  • study developmental processes of infants
    • motor skill acquisition
    • language acquisition
  • implement similar processes in robots

Background on intrinsic motivations

Sparse reward reinforcement learning:

  • Add additional term rewarding model improvement / curiosity / control
  • when not obtaining reward this lets the agent find useful behaviour (hopefully)

Background on intrinsic motivations

AGI:

  • drive open ended and continual learning with intrinsic motivation
  • no limit?

Background on intrinsic motivations

Advantages of intrinsic motivations

  • scalability :
    • no need to design reward function for each environment
    • environment kind and size does not change reward function
    • agent complexity does not change reward function

Disadvantage:

  • often (very) hard to compute
  • too general, faster if available:
    • specifically designed (dense) reward
    • imitation learning

Examples:

  • hunger is not an intrinsic motivation
    • not embodiment (digestive system) independent
    • eating more doesn't improve our model of the world

Background on intrinsic motivations

Examples:

  • maximizing stored energy is closer to an intrinsic motivation
    • real world agents need energy but not virtual ones
    • doesn't directly improve the world model
    • but maybe indirectly
    • open ended?

Background on intrinsic motivations

Examples:

  • maximizing money is also close to an intrinsic motivation
    • but it only exists in some societies
    • may also indirectly improve our model
    • open ended?

Background on intrinsic motivations

Examples:

  • minimizing prediction error of the model is an intrinsic motivation
    • any agent that remembers its predictions can calculate the prediction error
    • reducing it improves the model (at least locally)

Background on intrinsic motivations

Background on intrinsic motivations

dark room problem

Examples:

  • minimizing prediction error of the model is an intrinsic motivation
    • any agent that remembers its predictions can calculate the prediction error
    • reducing it improves the model (at least locally)
    • not open ended

Solution for dark room problem

  • maximizing the decrease of the prediction error (prediction progress) is an intrinsic motivation
    • improves the predictions of the model in one area until more progress can be made in another
    • may be open ended

Background on intrinsic motivations

  1. Background to intrinsic motivations
  2. Formal framework for intrinsic motivations
    1. Generative model
    2. Prediction / complete posterior
    3. Action selection
  3. Some intrinsic motivations:
    1. Free Energy Minimization
    2. Predictive information maximization
    3. Knowledge seeking
    4. Empowerment maximization

Overview

2. Formal framework for intrinsic motivations

Remarks:

  • Intrinsic motivations quantify statistical relations between sensor values, actions, and beliefs.
  • Taking actions according to such measures requires to predict them.
  • Possible by using parameterized / generative model.
  • Encodes beliefs and predictions in probability distributions over parameters and latent variables.
  • Easy to rigorously express many intrinsic motivations.
  • Naive computation is intractable.
  • Making it tractable is not discussed.

2. Formal framework for intrinsic motivations

1. Generative model

  • Internal to the agent
  • For parameters write \(\Theta=(\Theta^1,\Theta^2,\Theta^3)\)
  • \(\xi=(\xi^1,\xi^2,\xi^3)\) are fixed hyperparameters that encode priors over the parameters

2. Formal framework for intrinsic motivations

2. Generative model

Model split up into three parts:

  1. sensor dynamics model \(\newcommand{\q}{\text{q}}\newcommand{\hs}{\hat{s}}\q(\hat{s}|\hat{e},\theta)\)
  2. environment dynamics model \(\newcommand{\q}{\text{q}}\newcommand{\ha}{\hat{a}}\q(\hat{e}'|\ha,\hat{e},\theta)\)
  3. initial environment distribution \(\newcommand{\q}{\text{q}}\newcommand{\hs}{\hat{s}}\q(\hat{e}|\theta)\)

2. Formal framework for intrinsic motivations

2. Generative model

Model split up into three parts:

  1. sensor dynamics model \(\newcommand{\q}{\text{q}}\newcommand{\hs}{\hat{s}}\q(\hat{s}|\hat{e},\theta)\)
  2. environment dynamics model \(\newcommand{\q}{\text{q}}\newcommand{\ha}{\hat{a}}\q(\hat{e}'|\ha,\hat{e},\theta)\)
  3. initial environment distribution \(\newcommand{\q}{\text{q}}\newcommand{\hs}{\hat{s}}\q(\hat{e}|\theta)\)

2. Formal framework for intrinsic motivations

2. Generative model

Model split up into three parts:

  1. sensor dynamics model \(\newcommand{\q}{\text{q}}\newcommand{\hs}{\hat{s}}\q(\hat{s}|\hat{e},\theta)\)
  2. environment dynamics model \(\newcommand{\q}{\text{q}}\newcommand{\ha}{\hat{a}}\q(\hat{e}'|\ha,\hat{e},\theta)\)
  3. initial environment distribution \(\newcommand{\q}{\text{q}}\newcommand{\hs}{\hat{s}}\q(\hat{e}|\theta)\)

2. Formal framework for intrinsic motivations

2. Prediction

So at \(t\) agent can plug its experience \(sa_{\prec t}\) into model

  • updates the probability distribution to a posterior
\newcommand{\tT}{{t:T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\hT}{{\hat{T}}} \newcommand{\diff}{\text{d}} \q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT,sa_\pt,\xi)
q(s^t:T^,e^0:T^,θ∣a^t:T^,sa≺t,ξ)\newcommand{\tT}{{t:T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\hT}{{\hat{T}}} \newcommand{\diff}{\text{d}} \q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT,sa_\pt,\xi)

                                        predicts consequences of \(\blue{\hat{a}_{t:\hat{T}}}\) for relations between:

\newcommand{\tT}{{t:T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\hT}{{\hat{T}}} \newcommand{\diff}{\text{d}} \q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT,sa_\pt,\xi)
q(s^t:T^,e^0:T^,θ∣a^t:T^,sa≺t,ξ)\newcommand{\tT}{{t:T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\hT}{{\hat{T}}} \newcommand{\diff}{\text{d}} \q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT,sa_\pt,\xi)
  • parameters \(\red{\Theta}\)
  • latent variables \(\red{\hat{E}_{0:\hat{T}}}\)
  • future sensor values \(\red{\hat{S}_{t:\hat{T}}}\)

2. Formal framework for intrinsic motivations

2. Prediction

Call \(\text{q}(\hat{s}_{t:\hat{T}},\hat{e}_{0:\hat{T}},\theta|\hat{a}_{t:\hat{T}},sa_{\prec t},\xi)\) the complete posterior.  

2. Formal framework for intrinsic motivations

2. Prediction

2. Formal framework for intrinsic motivations

3. Action selection

  • define intrinsic motivations as functions \(\mathfrak{M}\) of this posterior and a given sequence \(\hat{a}_{t:\hat{T}}\) of future actions:
\newcommand{\tT}{{t:T}}
\newcommand{\tT}{{t:T}}
\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\mot}{\mathfrak{M}} \mot(\q(.,.,.|.,sa_\pt,\xi),\ha_\thT)
M(q(.,.,.∣.,sa≺t,ξ),a^t:T^)\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\mot}{\mathfrak{M}} \mot(\q(.,.,.|.,sa_\pt,\xi),\ha_\thT)
  • Requirement is conditional probability \(\text{q}(\hat{s}_{t:\hat{T}},\hat{e}_{0:\hat{T}},\theta|\hat{a}_{t:\hat{T}})\) how this is obtained by agent does not matter.
  • drop \((sa_{\prec t},\xi)\) in following
  • To act find best sequence:
\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \ha^*_\thT(sa_\pt):=\argmax_{\ha_\thT} \mot(\q(.,.,.|.,sa_\pt,\xi),\ha_\thT)
a^t:T^∗(sa≺t):=argmaxa^t:T^M(q(.,.,.∣.,sa≺t,ξ),a^t:T^)\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \ha^*_\thT(sa_\pt):=\argmax_{\ha_\thT} \mot(\q(.,.,.|.,sa_\pt,\xi),\ha_\thT)

3. Some intrinsic motivations

1. Free energy minimization

Actions should lead to environment states expected to have precise sensor values.

\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \q(\he_\thT|\ha_\thT)= \int \sum_{\hs_\thT,\he_\pt} \q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT) \diff \theta
q(e^t:T^∣a^t:T^)=∫∑s^t:T^,e^≺tq(s^t:T^,e^0:T^,θ∣a^t:T^)dθ\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \q(\he_\thT|\ha_\thT)= \int \sum_{\hs_\thT,\he_\pt} \q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT) \diff \theta
\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \begin{aligned} \mot(\q(.,.,.|.,\xi),\ha_\thT) :&=-\HS_{\q}(\hS_\thT|\hE_\thT,\ha_\thT)\\ &= \sum_{\he_\thT} \q(\he_\thT|\ha_\thT) \sum_{\hs_\thT} \q(\hs_\thT|\he_\thT) \log \q(\hs_\thT|\he_\thT)\\ \end{aligned}
M(q(.,.,.∣.,ξ),a^t:T^):=−Hq(S^t:T^∣E^t:T^,a^t:T^)=∑e^t:T^q(e^t:T^∣a^t:T^)∑s^t:T^q(s^t:T^∣e^t:T^)log⁡q(s^t:T^∣e^t:T^)\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \begin{aligned} \mot(\q(.,.,.|.,\xi),\ha_\thT) :&=-\HS_{\q}(\hS_\thT|\hE_\thT,\ha_\thT)\\ &= \sum_{\he_\thT} \q(\he_\thT|\ha_\thT) \sum_{\hs_\thT} \q(\hs_\thT|\he_\thT) \log \q(\hs_\thT|\he_\thT)\\ \end{aligned}

Get \(\text{q}(\hat{e}_{t:\hat{T}}|\hat{a}_{t:\hat{T}})\) frome the complete posterior:

3. Some intrinsic motivations

1. Free energy minimization

  • random noise source are avoided
  • will get stuck in known "dark room traps"
    • we know $$\text{H}_{\text{q}}(\hat{S}_{t:\hat{T}}|\hat{a}_{t:\hat{T}})=0\Rightarrow\text{H}_{\text{q}}(\hat{S}_{t:\hat{T}}|\hat{E}_{t:\hat{t}},\hat{a}_{t:\hat{T}})=0$$
    • such an optimal action sequence \(\hat{a}_{t:\hat{T}}\) exists e.g. if there is a "dark room" in the environment
    • even if it cannot be escaped once entered
    • solved by adding KL divergence to constructed desired sensory experience
      • breaks purpose of intrinsic motivations (not scalable)
  • Free energy is not suitable for AGI
\newcommand{\hT}{{\hat{T}}} \newcommand{\thT}{{t:\hT}} \newcommand{\hTa}{{\hat{T}_a}} \newcommand{\thTa}{{t:\hTa}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\d}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\I}{\text{I}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \newcommand{\hA}{\hat{A}} \begin{aligned} \mot^{PI}(\d(.,.,.|.),\ha_\thT) :&= \I_{\q}(\hS_{t:t+k-1}:\hS_{t+k:t+2k-1}|\ha_\thT)\\ &=\sum_{\hs_{t:t+2k-1}} \q(\hs_{t:t+2k-1}|\ha_\thT) \log \frac{\q(\hs_{t+k:t+2k-1}|\hs_{t:t+k-1},\ha_\thT)}{\q(\hs_{t:t+k-1}|\ha_\thT)}\\ \end{aligned}
MPI(q(.,.,.∣.),a^t:T^):=Iq(S^t:t+k−1:S^t+k:t+2k−1∣a^t:T^)=∑s^t:t+2k−1q(s^t:t+2k−1∣a^t:T^)log⁡q(s^t+k:t+2k−1∣s^t:t+k−1,a^t:T^)q(s^t:t+k−1∣a^t:T^)\newcommand{\hT}{{\hat{T}}} \newcommand{\thT}{{t:\hT}} \newcommand{\hTa}{{\hat{T}_a}} \newcommand{\thTa}{{t:\hTa}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\d}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\I}{\text{I}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \newcommand{\hA}{\hat{A}} \begin{aligned} \mot^{PI}(\d(.,.,.|.),\ha_\thT) :&= \I_{\q}(\hS_{t:t+k-1}:\hS_{t+k:t+2k-1}|\ha_\thT)\\ &=\sum_{\hs_{t:t+2k-1}} \q(\hs_{t:t+2k-1}|\ha_\thT) \log \frac{\q(\hs_{t+k:t+2k-1}|\hs_{t:t+k-1},\ha_\thT)}{\q(\hs_{t:t+k-1}|\ha_\thT)}\\ \end{aligned}

3. Some intrinsic motivations

2. Predictive information maximization

Actions should lead to the most complex sensor stream:

  • Next \(k\) sensor values should have max mutual information with the subsequent \(k\).
  • Can get needed distributions from complete posterior.

3. Some intrinsic motivations

2. Predictive information maximization

  • random noise source are avoided as they produce no mutual information
  • will not get stuck in known "dark room traps"
    • from $$\text{H}_{\text{q}}(\hat{S}_{t+k:t+2k-1}|\hat{a}_{t:\hat{T}})=0\Rightarrow\text{I}_{\text{q}}(\hat{S}_{t:t+k-1},\hat{S}_{t+k:t+2k-1}|\hat{a}_{t:\hat{T}})=0$$
  • possible long term behavior:
    • ergodic sensor process
    • finds a subset of environment states that allows this ergodicity

3. Some intrinsic motivations

2. Predictive information maximization

Georg Martius, Ralf Der

\newcommand{\hT}{{\hat{T}}} \newcommand{\thT}{{t:\hT}} \newcommand{\hTa}{{\hat{T}_a}} \newcommand{\thTa}{{t:\hTa}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\d}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\I}{\text{I}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \newcommand{\hA}{\hat{A}} \begin{aligned} \mot^{KSA}(\q(.,.,.|.),\ha_\thT) :&= \I_{\q}(\hS_\thT:\Theta|\ha_\thT)\\ &=\sum_{\hs_\thT} \int \q(\hs_\thT,\theta|\ha_\thT) \log \frac{\q(\theta|\hs_\thT,\ha_\thT)}{\q(\theta)} \diff \theta \end{aligned}
MKSA(q(.,.,.∣.),a^t:T^):=Iq(S^t:T^:Θ∣a^t:T^)=∑s^t:T^∫q(s^t:T^,θ∣a^t:T^)log⁡q(θ∣s^t:T^,a^t:T^)q(θ)dθ\newcommand{\hT}{{\hat{T}}} \newcommand{\thT}{{t:\hT}} \newcommand{\hTa}{{\hat{T}_a}} \newcommand{\thTa}{{t:\hTa}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\d}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\I}{\text{I}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \newcommand{\hA}{\hat{A}} \begin{aligned} \mot^{KSA}(\q(.,.,.|.),\ha_\thT) :&= \I_{\q}(\hS_\thT:\Theta|\ha_\thT)\\ &=\sum_{\hs_\thT} \int \q(\hs_\thT,\theta|\ha_\thT) \log \frac{\q(\theta|\hs_\thT,\ha_\thT)}{\q(\theta)} \diff \theta \end{aligned}

3. Some intrinsic motivations

3. Knowledge seeking

Actions should lead to sensor values that tell the most about model parameters \(\Theta\):

  • Also known as information gain maximization
  • Can get needed distributions from complete posterior.

3. Some intrinsic motivations

3. Knowledge seeking

  • avoids random noise sources once they are known
  • similar to prediction progress
  • can rewrite as $$\text{H}_{\text{q}}(\Theta)-\text{H}_{\text{q}}(\Theta|\hat{S}_{t:\hat{T}},\hat{a}_{t:\hat{T}})$$
  • will not get stuck in known "dark room traps"
    • from $$\text{H}_{\text{q}}(\hat{S}_{t:\hat{T}}|\hat{a}_{t:\hat{T}})=0\Rightarrow\text{I}_{\text{q}}(\hat{S}_{t:\hat{T}},\Theta|\hat{a}_{t:\hat{T}})=0$$
  • possible long term behavior:
    • when model is known does nothing / random walk

3. Some intrinsic motivations

3. Knowledge seeking

Bellemare et al. (2016)

\newcommand{\hT}{{\hat{T}}} \newcommand{\thT}{{t:\hT}} \newcommand{\hTa}{{\hat{T}_a}} \newcommand{\thTa}{{t:\hTa}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\d}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\I}{\text{I}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \newcommand{\hA}{\hat{A}} \begin{aligned} \mot^{EM}(\d(.,.,.|.),\ha_\thTa) :&= \max_{\d(\ha_{\hTa+1:\hT})} \; \I_{\d}(\hA_{\hTa+1:\hT}:\hS_\hT|\ha_\thTa) \\ &=\max_{\d(\ha_{\hTa+1:\hT})} \; \sum_{\ha_{\hTa+1:\hT},\hs_\hT} \d(\ha_{\hTa+1:\hT}) \d(\hs_\hT|\ha_\thT) \log \frac{\d(\hs_\hT|\ha_\thT)}{\d(\hs_\hT|\ha_\thTa)}. \end{aligned}
MEM(q(.,.,.∣.),a^t:T^a):=max⁡q(a^T^a+1:T^)  Iq(A^T^a+1:T^:S^T^∣a^t:T^a)=max⁡q(a^T^a+1:T^)  ∑a^T^a+1:T^,s^T^q(a^T^a+1:T^)q(s^T^∣a^t:T^)log⁡q(s^T^∣a^t:T^)q(s^T^∣a^t:T^a).\newcommand{\hT}{{\hat{T}}} \newcommand{\thT}{{t:\hT}} \newcommand{\hTa}{{\hat{T}_a}} \newcommand{\thTa}{{t:\hTa}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\d}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\I}{\text{I}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \newcommand{\hA}{\hat{A}} \begin{aligned} \mot^{EM}(\d(.,.,.|.),\ha_\thTa) :&= \max_{\d(\ha_{\hTa+1:\hT})} \; \I_{\d}(\hA_{\hTa+1:\hT}:\hS_\hT|\ha_\thTa) \\ &=\max_{\d(\ha_{\hTa+1:\hT})} \; \sum_{\ha_{\hTa+1:\hT},\hs_\hT} \d(\ha_{\hTa+1:\hT}) \d(\hs_\hT|\ha_\thT) \log \frac{\d(\hs_\hT|\ha_\thT)}{\d(\hs_\hT|\ha_\thTa)}. \end{aligned}

3. Some intrinsic motivations

4. Empowerment maximization

Actions should lead to control over as many future experiences as possible:

  • Actions \(\hat{a}_{t:\hat{T}_a}\) are taken such that subsequent actions \(\hat{a}_{\hat{T}_a+1:\hat{T}}\) have control
  • Can get needed distributions from complete posterior.

3. Some intrinsic motivations

4. Empowerment maximization

  • avoids random noise sources because they cannot be controlled
  • will not get stuck in known "dark room traps"
    • from $$\text{H}_{\text{q}}(\hat{S}_{\hat{T}}|\hat{a}_{t:\hat{T}_a})=0\Rightarrow\text{I}_{\text{q}}(\hat{A}_{\hat{T}_a+1:\hat{T}}\hat{S}_{\hat{T}}:|\hat{a}_{t:\hat{T}_a})=0$$
  • similar to energy and money maximization but more general
  • possible long term behavior:
    • remains in (or maintains) the situation where it expects the most control over future experience
    • exploration behavior not fully understood
    • Belief empowerment may solve it...

3. Some intrinsic motivations

4. Empowerment

Guckelsberger et al. (2016)

3. Some intrinsic motivations

5. Curiosity

Actions should lead to surprising environment states (sensor embeddings).

\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \begin{aligned} \mot(\q(.,.,.|.,\xi),\ha_\thT) :&=+\HS_{\q}(\hE_\thT|\ha_\thT)\\ &= \sum_{\hs_\thT} \q(\hE_\thT|\ha_\thT)(- \log \q(\hE_\thT|\ha_\thT))\\ \end{aligned}
M(q(.,.,.∣.,ξ),a^t:T^):=+Hq(E^t:T^∣a^t:T^)=∑s^t:T^q(E^t:T^∣a^t:T^)(−log⁡q(E^t:T^∣a^t:T^))\newcommand{\hT}{\hat{T}} \newcommand{\thT}{{t:\hT}} \newcommand{\hs}{\hat{s}} \newcommand{\pt}{{\prec t}} \newcommand{\pet}{{\preceq t}} \newcommand{\set}{{\succeq t}} \newcommand{\ha}{\hat{a}} \newcommand{\he}{\hat{e}} \newcommand{\q}{\text{q}} \newcommand{\diff}{\text{d}} \newcommand{\ptau}{{\prec \tau}} \newcommand{\petau}{{\preceq \tau}} \newcommand{\stau}{{\succ \tau}} \newcommand{\setau}{{\succeq \tau}} \newcommand{\argmax}{\text{argmax}} \newcommand{\mot}{\mathfrak{M}} \newcommand{\HS}{\text{H}} \newcommand{\hS}{\hat{S}} \newcommand{\hE}{\hat{E}} \begin{aligned} \mot(\q(.,.,.|.,\xi),\ha_\thT) :&=+\HS_{\q}(\hE_\thT|\ha_\thT)\\ &= \sum_{\hs_\thT} \q(\hE_\thT|\ha_\thT)(- \log \q(\hE_\thT|\ha_\thT))\\ \end{aligned}
  • maximize expected surprise (=entropy)
  • Get density from the complete posterior.

3. Some intrinsic motivations

5. Curiosity

  • will not get stuck in known "dark room traps"
    • it directly pursues the opposite situation
  • will get stuck at random noise sources
  • in deterministic environments not a big problem

3. Some intrinsic motivations

5. Curiosity

Burda et al. (2018)

References:

Aslanides, J., Leike, J., and Hutter, M. (2017). Universal Reinforcement Learning Algorithms: Survey and Experiments. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 1403–1410.


Ay, N., Bertschinger, N., Der, R., Güttler, F., and Olbrich, E. (2008). Predictive Information and Explorative Behavior of Autonomous Robots. The European Physical Journal B-Condensed Matter and Complex Systems, 63(3):329–339.

 

Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. (2018). Large-Scale Study of Curiosity-Driven Learning.  arXiv:1808.04355 [cs, stat]. arXiv: 1808.04355.


Friston, K. J., Parr, T., and de Vries, B. (2017). The Graphical Brain: Belief Propagation and Active Inference. Network Neuroscience, 1(4):381–414.


Klyubin, A., Polani, D., and Nehaniv, C. (2005). Empowerment: A Universal Agent-Centric Measure of Control. In The 2005 IEEE Congress on Evolutionary Computation, 2005, volume 1, pages 128–135.


Orseau, L., Lattimore, T., and Hutter, M. (2013). Universal Knowledge-Seeking Agents for Stochastic Environments. In Jain, S., Munos, R., Stephan, F., and Zeugmann, T., editors, Algorithmic Learning Theory, number 8139 in Lecture Notes in Computer Science, pages 158–172. Springer Berlin Heidelberg.

 

Oudeyer, P.-Y. and Kaplan, F. (2008). How can we define intrinsic motivation? In Proceedings of the 8th International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, Lund University Cognitive Studies, Lund: LUCS, Brighton. Lund University Cognitive Studies, Lund: LUCS, Brighton.


Schmidhuber, J. (2010). Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230–247.


Storck, J., Hochreiter, S., and Schmidhuber, J. (1995). Reinforcement Driven Information Acquisition in Non-Deterministic Environments. In Proceedings of the International Conference on Artificial Neural Networks, volume 2, pages 159–164.

 

Guckelsberger, C., Salge, C., & Colton, S. (2016). Intrinsically Motivated General Companion NPCs via Coupled Empowerment Maximisation. 2016 IEEE Conf. Computational Intelligence in Games (CIG’16), 150–157

 

 

Made with Slides.com