Introduction to intrinsic motivations
Martin Biehl
- Background / introduction to intrinsic motivations
- Formal framework for intrinsic motivations
- Generative model
- Prediction / complete posterior
- Action selection
- Some intrinsic motivations:
- Free Energy Minimization
- Predictive information maximization
- Knowledge seeking
- Empowerment maximization
- Curiosity
Overview
Originally from psychology e.g. (Ryan and Deci, 2000):
activity for its inherent satisfaction rather than separable consequence
for the fun or challenge entailed rather than because of external products, pressures or reward
Examples (Oudeyer, 2008):
- infants grasping, throwing, biting new objects,
- adults playing crosswords, painting, gardening, reading novels...
Background on intrinsic motivations
But can always argue:
- these things possibly increase the probability of survival in some way
- "learned" by evolution
- cannot be sure they have no purpose
Background on intrinsic motivations
Motivation is something that generates behavior for an agent (robot, living organism)
- similar to reward function in reinforcement learning (RL)
Background on intrinsic motivations
Working definition compatible with Oudeyer (2008):
Motivation is intrinsic if it:
- ``rewiring agnostic''.
- embodiment independent,
- semantic free / information theoretic,
This includes the approach by Schmidhuber (2010):
Motivation is intrinsic if it
- rewards improvement of some model quality measure.
Background on intrinsic motivations
rewiring agnostic means
- if we rewire the sensors or rewire the actuators the intrinsic motivation will still lead to similar behaviour
this implies
- cannot assume that there is a special sensor that corresponds to reward
- so reward functions of MDPs, POMDPs, and standard RL are not available
Background on intrinsic motivations
Embodiment independent means it should work (without changes) for any form of agent:
Background on intrinsic motivations
and produce "worthwhile" behavior
Embodiment independent means it should work (without changes) for any form of agent:
Background on intrinsic motivations
- Should work for any number or kind of
- sensors
- actuators
Semantic free, information theoretic:
- relations between sensors, actuators, and internal variables count
- specific values don't
- information theory quantifies relations
- if f and g are bijective functions then I(X:Y)=I(f(X):g(Y))
- so the values of X or Y can play no role in mutual information.
Background on intrinsic motivations
Background on intrinsic motivations
Another important but not defining feature is usually known from evolution:
Background on intrinsic motivations
Another important but not defining feature is usually known from evolution:
open endedness
The motivation should not vanish until the capacities of the agent are exhausted.
Background on intrinsic motivations
Applications of intrinsic motivations:
- developmental robotics
- sparse reward reinforcement learning problems
- Human level and artificial general AI
Background on intrinsic motivations
Developmental robotics:
- study developmental processes of infants
- motor skill acquisition
- language acquisition
- implement similar processes in robots
Background on intrinsic motivations
Sparse reward reinforcement learning:
- Add additional term rewarding model improvement / curiosity / control
- when not obtaining reward this lets the agent find useful behaviour (hopefully)
Background on intrinsic motivations
AGI:
- drive open ended and continual learning with intrinsic motivation
- no limit?
Background on intrinsic motivations
Advantages of intrinsic motivations
- scalability :
- no need to design reward function for each environment
- environment kind and size does not change reward function
- agent complexity does not change reward function
Disadvantage:
- often (very) hard to compute
- too general, faster if available:
- specifically designed (dense) reward
- imitation learning
Examples:
- hunger is not an intrinsic motivation
- not embodiment (digestive system) independent
- eating more doesn't improve our model of the world
Background on intrinsic motivations
Examples:
- maximizing stored energy is closer to an intrinsic motivation
- real world agents need energy but not virtual ones
- doesn't directly improve the world model
- but maybe indirectly
- open ended?
Background on intrinsic motivations
Examples:
- maximizing money is also close to an intrinsic motivation
- but it only exists in some societies
- may also indirectly improve our model
- open ended?
Background on intrinsic motivations
Examples:
- minimizing prediction error of the model is an intrinsic motivation
- any agent that remembers its predictions can calculate the prediction error
- reducing it improves the model (at least locally)
Background on intrinsic motivations
Background on intrinsic motivations
dark room problem
Examples:
- minimizing prediction error of the model is an intrinsic motivation
- any agent that remembers its predictions can calculate the prediction error
- reducing it improves the model (at least locally)
- not open ended
Solution for dark room problem
-
maximizing the decrease of the prediction error (prediction progress) is an intrinsic motivation
- improves the predictions of the model in one area until more progress can be made in another
- may be open ended
Background on intrinsic motivations
- Background to intrinsic motivations
- Formal framework for intrinsic motivations
- Generative model
- Prediction / complete posterior
- Action selection
- Some intrinsic motivations:
- Free Energy Minimization
- Predictive information maximization
- Knowledge seeking
- Empowerment maximization
Overview
2. Formal framework for intrinsic motivations
Remarks:
- Intrinsic motivations quantify statistical relations between sensor values, actions, and beliefs.
- Taking actions according to such measures requires to predict them.
- Possible by using parameterized / generative model.
- Encodes beliefs and predictions in probability distributions over parameters and latent variables.
- Easy to rigorously express many intrinsic motivations.
- Naive computation is intractable.
- Making it tractable is not discussed.
2. Formal framework for intrinsic motivations
1. Generative model
- Internal to the agent
- For parameters write Θ=(Θ1,Θ2,Θ3)
- ξ=(ξ1,ξ2,ξ3) are fixed hyperparameters that encode priors over the parameters
2. Formal framework for intrinsic motivations
2. Generative model
Model split up into three parts:
- sensor dynamics model q(s^∣e^,θ)
- environment dynamics model q(e^′∣a^,e^,θ)
- initial environment distribution q(e^∣θ)
2. Formal framework for intrinsic motivations
2. Generative model
Model split up into three parts:
- sensor dynamics model q(s^∣e^,θ)
- environment dynamics model q(e^′∣a^,e^,θ)
- initial environment distribution q(e^∣θ)
2. Formal framework for intrinsic motivations
2. Generative model
Model split up into three parts:
- sensor dynamics model q(s^∣e^,θ)
- environment dynamics model q(e^′∣a^,e^,θ)
- initial environment distribution q(e^∣θ)
2. Formal framework for intrinsic motivations
2. Prediction
So at t agent can plug its experience sa≺t into model
- updates the probability distribution to a posterior
\newcommand{\tT}{{t:T}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\hT}{{\hat{T}}}
\newcommand{\diff}{\text{d}}
\q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT,sa_\pt,\xi)
q(s^t:T^,e^0:T^,θ∣a^t:T^,sa≺t,ξ)
predicts consequences of a^t:T^ for relations between:
\newcommand{\tT}{{t:T}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\hT}{{\hat{T}}}
\newcommand{\diff}{\text{d}}
\q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT,sa_\pt,\xi)
q(s^t:T^,e^0:T^,θ∣a^t:T^,sa≺t,ξ)
- parameters Θ
- latent variables E^0:T^
- future sensor values S^t:T^
2. Formal framework for intrinsic motivations
2. Prediction
Call q(s^t:T^,e^0:T^,θ∣a^t:T^,sa≺t,ξ) the complete posterior.
2. Formal framework for intrinsic motivations
2. Prediction
2. Formal framework for intrinsic motivations
3. Action selection
- define intrinsic motivations as functions M of this posterior and a given sequence a^t:T^ of future actions:
\newcommand{\tT}{{t:T}}
\newcommand{\hT}{\hat{T}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\set}{{\succeq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\diff}{\text{d}}
\newcommand{\ptau}{{\prec \tau}}
\newcommand{\petau}{{\preceq \tau}}
\newcommand{\stau}{{\succ \tau}}
\newcommand{\setau}{{\succeq \tau}}
\newcommand{\mot}{\mathfrak{M}}
\mot(\q(.,.,.|.,sa_\pt,\xi),\ha_\thT)
- Requirement is conditional probability q(s^t:T^,e^0:T^,θ∣a^t:T^) how this is obtained by agent does not matter.
- drop (sa≺t,ξ) in following
- To act find best sequence:
\newcommand{\hT}{\hat{T}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\set}{{\succeq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\diff}{\text{d}}
\newcommand{\ptau}{{\prec \tau}}
\newcommand{\petau}{{\preceq \tau}}
\newcommand{\stau}{{\succ \tau}}
\newcommand{\setau}{{\succeq \tau}}
\newcommand{\argmax}{\text{argmax}}
\newcommand{\mot}{\mathfrak{M}}
\ha^*_\thT(sa_\pt):=\argmax_{\ha_\thT} \mot(\q(.,.,.|.,sa_\pt,\xi),\ha_\thT)
3. Some intrinsic motivations
1. Free energy minimization
Actions should lead to environment states expected to have precise sensor values.
\newcommand{\hT}{\hat{T}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\set}{{\succeq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\diff}{\text{d}}
\newcommand{\ptau}{{\prec \tau}}
\newcommand{\petau}{{\preceq \tau}}
\newcommand{\stau}{{\succ \tau}}
\newcommand{\setau}{{\succeq \tau}}
\newcommand{\argmax}{\text{argmax}}
\newcommand{\mot}{\mathfrak{M}}
\newcommand{\HS}{\text{H}}
\newcommand{\hS}{\hat{S}}
\newcommand{\hE}{\hat{E}}
\q(\he_\thT|\ha_\thT)=
\int \sum_{\hs_\thT,\he_\pt} \q(\hs_\thT,\he_{0:\hT},\theta|\ha_\thT) \diff \theta
\newcommand{\hT}{\hat{T}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\set}{{\succeq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\diff}{\text{d}}
\newcommand{\ptau}{{\prec \tau}}
\newcommand{\petau}{{\preceq \tau}}
\newcommand{\stau}{{\succ \tau}}
\newcommand{\setau}{{\succeq \tau}}
\newcommand{\argmax}{\text{argmax}}
\newcommand{\mot}{\mathfrak{M}}
\newcommand{\HS}{\text{H}}
\newcommand{\hS}{\hat{S}}
\newcommand{\hE}{\hat{E}}
\begin{aligned}
\mot(\q(.,.,.|.,\xi),\ha_\thT) :&=-\HS_{\q}(\hS_\thT|\hE_\thT,\ha_\thT)\\
&= \sum_{\he_\thT} \q(\he_\thT|\ha_\thT) \sum_{\hs_\thT} \q(\hs_\thT|\he_\thT) \log \q(\hs_\thT|\he_\thT)\\
\end{aligned}
Get q(e^t:T^∣a^t:T^) frome the complete posterior:
3. Some intrinsic motivations
1. Free energy minimization
- random noise source are avoided
- will get stuck in known "dark room traps"
- we know Hq(S^t:T^∣a^t:T^)=0⇒Hq(S^t:T^∣E^t:t^,a^t:T^)=0
- such an optimal action sequence a^t:T^ exists e.g. if there is a "dark room" in the environment
- even if it cannot be escaped once entered
- solved by adding KL divergence to constructed desired sensory experience
- breaks purpose of intrinsic motivations (not scalable)
- Free energy is not suitable for AGI
\newcommand{\hT}{{\hat{T}}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hTa}{{\hat{T}_a}}
\newcommand{\thTa}{{t:\hTa}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\set}{{\succeq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\d}{\text{q}}
\newcommand{\diff}{\text{d}}
\newcommand{\ptau}{{\prec \tau}}
\newcommand{\petau}{{\preceq \tau}}
\newcommand{\stau}{{\succ \tau}}
\newcommand{\setau}{{\succeq \tau}}
\newcommand{\argmax}{\text{argmax}}
\newcommand{\mot}{\mathfrak{M}}
\newcommand{\HS}{\text{H}}
\newcommand{\I}{\text{I}}
\newcommand{\hS}{\hat{S}}
\newcommand{\hE}{\hat{E}}
\newcommand{\hA}{\hat{A}}
\begin{aligned}
\mot^{PI}(\d(.,.,.|.),\ha_\thT) :&= \I_{\q}(\hS_{t:t+k-1}:\hS_{t+k:t+2k-1}|\ha_\thT)\\
&=\sum_{\hs_{t:t+2k-1}} \q(\hs_{t:t+2k-1}|\ha_\thT) \log \frac{\q(\hs_{t+k:t+2k-1}|\hs_{t:t+k-1},\ha_\thT)}{\q(\hs_{t:t+k-1}|\ha_\thT)}\\
\end{aligned}
3. Some intrinsic motivations
2. Predictive information maximization
Actions should lead to the most complex sensor stream:
- Next k sensor values should have max mutual information with the subsequent k.
- Can get needed distributions from complete posterior.
3. Some intrinsic motivations
2. Predictive information maximization
- random noise source are avoided as they produce no mutual information
- will not get stuck in known "dark room traps"
- from Hq(S^t+k:t+2k−1∣a^t:T^)=0⇒Iq(S^t:t+k−1,S^t+k:t+2k−1∣a^t:T^)=0
- possible long term behavior:
- ergodic sensor process
- finds a subset of environment states that allows this ergodicity
3. Some intrinsic motivations
2. Predictive information maximization
Georg Martius, Ralf Der
\newcommand{\hT}{{\hat{T}}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hTa}{{\hat{T}_a}}
\newcommand{\thTa}{{t:\hTa}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\set}{{\succeq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\d}{\text{q}}
\newcommand{\diff}{\text{d}}
\newcommand{\ptau}{{\prec \tau}}
\newcommand{\petau}{{\preceq \tau}}
\newcommand{\stau}{{\succ \tau}}
\newcommand{\setau}{{\succeq \tau}}
\newcommand{\argmax}{\text{argmax}}
\newcommand{\mot}{\mathfrak{M}}
\newcommand{\HS}{\text{H}}
\newcommand{\I}{\text{I}}
\newcommand{\hS}{\hat{S}}
\newcommand{\hE}{\hat{E}}
\newcommand{\hA}{\hat{A}}
\begin{aligned}
\mot^{KSA}(\q(.,.,.|.),\ha_\thT) :&= \I_{\q}(\hS_\thT:\Theta|\ha_\thT)\\
&=\sum_{\hs_\thT} \int \q(\hs_\thT,\theta|\ha_\thT) \log \frac{\q(\theta|\hs_\thT,\ha_\thT)}{\q(\theta)} \diff \theta
\end{aligned}
3. Some intrinsic motivations
3. Knowledge seeking
Actions should lead to sensor values that tell the most about model parameters Θ:
- Also known as information gain maximization
- Can get needed distributions from complete posterior.
3. Some intrinsic motivations
3. Knowledge seeking
- avoids random noise sources once they are known
- similar to prediction progress
- can rewrite as Hq(Θ)−Hq(Θ∣S^t:T^,a^t:T^)
- will not get stuck in known "dark room traps"
- from Hq(S^t:T^∣a^t:T^)=0⇒Iq(S^t:T^,Θ∣a^t:T^)=0
- possible long term behavior:
- when model is known does nothing / random walk
3. Some intrinsic motivations
3. Knowledge seeking
Bellemare et al. (2016)
\newcommand{\hT}{{\hat{T}}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hTa}{{\hat{T}_a}}
\newcommand{\thTa}{{t:\hTa}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\set}{{\succeq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\d}{\text{q}}
\newcommand{\diff}{\text{d}}
\newcommand{\ptau}{{\prec \tau}}
\newcommand{\petau}{{\preceq \tau}}
\newcommand{\stau}{{\succ \tau}}
\newcommand{\setau}{{\succeq \tau}}
\newcommand{\argmax}{\text{argmax}}
\newcommand{\mot}{\mathfrak{M}}
\newcommand{\HS}{\text{H}}
\newcommand{\I}{\text{I}}
\newcommand{\hS}{\hat{S}}
\newcommand{\hE}{\hat{E}}
\newcommand{\hA}{\hat{A}}
\begin{aligned}
\mot^{EM}(\d(.,.,.|.),\ha_\thTa) :&= \max_{\d(\ha_{\hTa+1:\hT})} \; \I_{\d}(\hA_{\hTa+1:\hT}:\hS_\hT|\ha_\thTa) \\
&=\max_{\d(\ha_{\hTa+1:\hT})} \; \sum_{\ha_{\hTa+1:\hT},\hs_\hT} \d(\ha_{\hTa+1:\hT}) \d(\hs_\hT|\ha_\thT) \log \frac{\d(\hs_\hT|\ha_\thT)}{\d(\hs_\hT|\ha_\thTa)}.
\end{aligned}
3. Some intrinsic motivations
4. Empowerment maximization
Actions should lead to control over as many future experiences as possible:
- Actions a^t:T^a are taken such that subsequent actions a^T^a+1:T^ have control
- Can get needed distributions from complete posterior.
3. Some intrinsic motivations
4. Empowerment maximization
- avoids random noise sources because they cannot be controlled
- will not get stuck in known "dark room traps"
- from Hq(S^T^∣a^t:T^a)=0⇒Iq(A^T^a+1:T^S^T^:∣a^t:T^a)=0
- similar to energy and money maximization but more general
- possible long term behavior:
- remains in (or maintains) the situation where it expects the most control over future experience
- exploration behavior not fully understood
- Belief empowerment may solve it...
3. Some intrinsic motivations
4. Empowerment
Guckelsberger et al. (2016)
3. Some intrinsic motivations
5. Curiosity
Actions should lead to surprising environment states (sensor embeddings).
\newcommand{\hT}{\hat{T}}
\newcommand{\thT}{{t:\hT}}
\newcommand{\hs}{\hat{s}}
\newcommand{\pt}{{\prec t}}
\newcommand{\pet}{{\preceq t}}
\newcommand{\set}{{\succeq t}}
\newcommand{\ha}{\hat{a}}
\newcommand{\he}{\hat{e}}
\newcommand{\q}{\text{q}}
\newcommand{\diff}{\text{d}}
\newcommand{\ptau}{{\prec \tau}}
\newcommand{\petau}{{\preceq \tau}}
\newcommand{\stau}{{\succ \tau}}
\newcommand{\setau}{{\succeq \tau}}
\newcommand{\argmax}{\text{argmax}}
\newcommand{\mot}{\mathfrak{M}}
\newcommand{\HS}{\text{H}}
\newcommand{\hS}{\hat{S}}
\newcommand{\hE}{\hat{E}}
\begin{aligned}
\mot(\q(.,.,.|.,\xi),\ha_\thT) :&=+\HS_{\q}(\hE_\thT|\ha_\thT)\\
&= \sum_{\hs_\thT} \q(\hE_\thT|\ha_\thT)(- \log \q(\hE_\thT|\ha_\thT))\\
\end{aligned}
- maximize expected surprise (=entropy)
- Get density from the complete posterior.
3. Some intrinsic motivations
5. Curiosity
- will not get stuck in known "dark room traps"
- it directly pursues the opposite situation
- will get stuck at random noise sources
- in deterministic environments not a big problem
3. Some intrinsic motivations
5. Curiosity
Burda et al. (2018)
References:
Aslanides, J., Leike, J., and Hutter, M. (2017). Universal Reinforcement Learning Algorithms: Survey and Experiments. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 1403–1410.
Ay, N., Bertschinger, N., Der, R., Güttler, F., and Olbrich, E. (2008). Predictive Information and Explorative Behavior of Autonomous Robots. The European Physical Journal B-Condensed Matter and Complex Systems, 63(3):329–339.
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. (2018). Large-Scale Study of Curiosity-Driven Learning. arXiv:1808.04355 [cs, stat]. arXiv: 1808.04355.
Friston, K. J., Parr, T., and de Vries, B. (2017). The Graphical Brain: Belief Propagation and Active Inference. Network Neuroscience, 1(4):381–414.
Klyubin, A., Polani, D., and Nehaniv, C. (2005). Empowerment: A Universal Agent-Centric Measure of Control. In The 2005 IEEE Congress on Evolutionary Computation, 2005, volume 1, pages 128–135.
Orseau, L., Lattimore, T., and Hutter, M. (2013). Universal Knowledge-Seeking Agents for Stochastic Environments. In Jain, S., Munos, R., Stephan, F., and Zeugmann, T., editors, Algorithmic Learning Theory, number 8139 in Lecture Notes in Computer Science, pages 158–172. Springer Berlin Heidelberg.
Oudeyer, P.-Y. and Kaplan, F. (2008). How can we define intrinsic motivation? In Proceedings of the 8th International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, Lund University Cognitive Studies, Lund: LUCS, Brighton. Lund University Cognitive Studies, Lund: LUCS, Brighton.
Schmidhuber, J. (2010). Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230–247.
Storck, J., Hochreiter, S., and Schmidhuber, J. (1995). Reinforcement Driven Information Acquisition in Non-Deterministic Environments. In Proceedings of the International Conference on Artificial Neural Networks, volume 2, pages 159–164.
Guckelsberger, C., Salge, C., & Colton, S. (2016). Intrinsically Motivated General Companion NPCs via Coupled Empowerment Maximisation. 2016 IEEE Conf. Computational Intelligence in Games (CIG’16), 150–157
Introduction to intrinsic motivations Martin Biehl
Introduction to intrinsic motivations
By slides_martin
Introduction to intrinsic motivations
Presentation at Araya on 17 August 2018
- 1,176
