PlanGAN - Model-based planning for multi-goal,

sparse-reward environments

Henry Charlesworth and Giovanni Montana

Warwick Manufacturing Group, University of Warwick

NeurIPS 2020


  • Sparse rewards - usually easier to specify when a task is complete rather than defining a reward function
  • Want to train agents capable of carrying out multiple different tasks
  • Currently, best RL methods for these kinds of task are model-free (Hindsight Experience Replay)
  • But model-based RL can often be substantially more sample efficient - can we come up with a model-based approach that works for sparse-reward, multi-goal problems?


  • Build upon the same principle that underlies Hindsight Experience Replay - that trajectories that don't achieve the specified goal still contain useful information for how to achieve the goal(s) that actually were achieved


  • Aim: a goal-conditioned generative model (GAN) that can produce plausible trajectories that lead from current state towards a specified goal state
  • Train on gathered experience - relabel desired goal to be a goal achieved at some later point during the observed trajectory.
  • Gives us lots of example data to train on!


  • Gather data using a planner to plan actions using many trajectories produced by generative model


  • Massively outperforms normal model-based method
  • Significantly more sample efficient than model-free methods designed for these kinds of problems!
Made with