Intrinsic social motivation
via causal influence
in multi-agent RL
Natasha Jaques, Angeliki Lazaridou,
Edward Hughes, Çaglar Gulçehre et al.
Presented by
Breandan Considine
2005
Intrinsically motivated Reinforcement Learning
2005
2008
Intrinsic motivation
Novelty/Surprise
1. Barto, A. et al. 2013. Novelty or Surprise? Frontiers in Psychology.
2. Houthooft, R. et al. 2016. Information Maximizing Exploration. Advances in Neural Information Processing Systems.
3. Itti, L. and Baldi, P. Bayesian surprise attracts human attention. Vision Research, 49(10):1295–1306, 2009.
4. Conti, et al. Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents. CoRR, abs/1712.06560.
Empowerment
1. Klyubin, A.S., et al. 2005. All else being equal be empowered. In European Conference on Artificial Life (pp. 744–753).
2. Mnih, V., et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540), p.529.
3. Klyubin, A.S., et al. 2008. Keep your options open: An information-based driving principle for sensorimotor systems.
4. Jung, T., et al., 2011. Empowerment for continuous agent — environment systems. Adaptive Behavior, 19(1), pp.16–39.
2016
Unifying count-based exploration and intrinsic motivation
2017
Surprise-based intrinsic motivation
for deep reinforcement learning
2017
2017
2017
Sequential social dilemmas
Notation
<State, Transition, Action, reward>
The actions of all N agents are combined to form a joint action
Discounted future rewards
Extrinsic and Intrinsic rewards
Trajectories
How would B’s action change if
I had acted differently in this situation?
Averaging over multiple counterfactuals
Averaging over multiple counterfactuals
Mutual information and empowerment
Sequential social dilemmas
Model of other agents
