Intrinsic social motivation

via causal influence

in multi-agent RL

 

Natasha Jaques, Angeliki Lazaridou,

Edward Hughes, Çaglar Gulçehre et al.

Presented by

Breandan Considine

2005

http://www.cs.cornell.edu/~helou/IMRL.pdf

Intrinsically motivated Reinforcement Learning

2005

2008

Intrinsic motivation

Novelty/Surprise

1. Barto, A. et al. 2013. Novelty or Surprise? Frontiers in Psychology.

2. Houthooft, R. et al. 2016. Information Maximizing Exploration. Advances in Neural Information Processing Systems.

3. Itti, L. and  Baldi, P. Bayesian surprise attracts human attention. Vision Research, 49(10):1295–1306, 2009.

4. Conti, et al. Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents. CoRR, abs/1712.06560.

Empowerment

1. Klyubin, A.S., et al. 2005. All else being equal be empowered. In European Conference on Artificial Life (pp. 744–753).

2. Mnih, V., et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540), p.529.

3. Klyubin, A.S., et al. 2008. Keep your options open: An information-based driving principle for sensorimotor systems.

4. Jung, T., et al., 2011. Empowerment for continuous agent — environment systems. Adaptive Behavior, 19(1), pp.16–39.

2016

Unifying count-based exploration and intrinsic motivation

2017

Surprise-based intrinsic motivation

for deep reinforcement learning

2017

2017

2017

Sequential social dilemmas

Notation

 <State, Transition, Action, reward>

 

The actions of all N agents are combined to form a joint action

 

 

 

Discounted future rewards

 

Extrinsic and Intrinsic rewards

 

Trajectories

 

 

 

 

How would B’s action change if

I had acted differently in this situation?

Averaging over multiple counterfactuals

Averaging over multiple counterfactuals

Mutual information and empowerment

Sequential social dilemmas

Model of other agents

References