Second progress report

In nutshell:

1. Construct Bayesian network with
   1. fixed kernels (white)
   2. controllable (agent) kernels (grey)
   3. Goal node (G)
2. Find controllable kernels that maximize goal probability.

Planning as inference

To find unknown kernels \(p_A:=\{p_a: a \in A\}\)

• "just" maximize probability of achieving the goal:
  
  \[p_A^* = \text{arg} \max_{p_A} p(G=g).\]
• automates design of agent memory dynamics and action selection (but hard to solve)

Planning as inference

Planning as inference

Practical side of original framework:

• Represent
  • Planning problem structure by Bayesian networks
  • Goal and possible policies by sets of probability distributions
• Find policies that maximize probability of goal via geometric EM algorithm.

   

Bayesian network

goal

policies

Planning as inference

• Problem structures as Bayesian networks:
  • Let \(V=\{1,...,n\}\) index the nodes in a graph then Bayesian network of random variables \(X=(X_1,..,X_n)\) defines factorized joint probability distribution:
    \[\newcommand{\pa}{\text{pa}}p(x) = \prod_{v\in V} p(x_v | x_{\pa(v)}).\]
  • distinguish:
    • fixed nodes \(B\subset V\)
    • changeable nodes \(A\subset V\)
    • then:
      \[\newcommand{\pa}{\text{pa}}p(x) = \prod_{a\in A} p(x_a | x_{\pa(a)}) \, \prod_{b\in B} \bar p(x_b | x_{\pa(b)})\]

Planning as inference

• Problem structures as Bayesian networks:

Planning as inference

Bayesian network

goal

• Represent goals and policies by sets of probability distributions:
  • goal must be an event i.e. function \(G(x)\) with
    • \(G(x)=1\) if goal is achieved
    • \(G(x)=0\) else.
  • goal manifold is set of distributions where the goal event occurs with probability one:
    \[M_G:=\{P: p(G=1)=1\}\]

Planning as inference

• Represent goals and policies by sets of probability distributions:
  • policy is a choice of the changeable Markov kernels
    \[\newcommand{\pa}{\text{pa}}\{p(x_a | x_{\pa(a)}):a \in A\}\]
  • agent manifold/policy manifold is set of distributions that can be achieved by varying policy
    \[\newcommand{\pa}{\text{pa}}p(x) = \prod_{a\in A} p(x_a | x_{\pa(a)}) \, \prod_{b\in B} \bar p(x_b | x_{\pa(b)})\]

Bayesian network

goal

policies

Planning as inference

• Find policies that maximize probability of goal via geometric EM algorithm:
  • planning as inference finds policy / agent kernels such that:
    \[P^* = \text{arg} \max_{P \in M_A} p(G=1).\]
  • (compare to maximum likelihood inference)

Bayesian network

goal

policies

Planning as inference

• Find policies that maximize probability of goal via geometric EM algorithm:
  • Can prove that \(P^*\) is the distribution in agent manifold closest to goal manifold in terms of KL-divergence
  • Local minimizers of this KL-divergence can be found with the geometric EM algorithm

Bayesian network

goal

policies

Planning as inference

• Find policies that maximize probability of goal via geometric EM algorithm:

1. Start with an initial prior, \(P_0 \in M_A\) .
2. (e-projection)
   \[Q_t = \text{arg} \min_{Q\in M_G} D_{KL}(Q∥P_t )\]
3. (m-projection)
   \[P_{t+1} = \text{arg} \min_{P \in M_A} D_{KL} (Q_t ∥P )\]

Bayesian network

goal

policies

Planning as inference

• Find policies that maximize probability of goal via geometric EM algorithm:
• Equivalent algorithm using only marginalization and conditioning:

1. Initial agent kernels define prior, \(P_0 \in M_A\).
2. Get  \(Q_t\), from \(P_t\) by conditioning on the goal:  \[q_t(x) = p_t(x|G=1).\]
3. Get \(P_{t+1}\), by replacing agent kernels by conditional distributions in \(Q_t\):
   \[\newcommand{\pa}{\text{pa}} p_{t+1}(x) = \prod_{a\in A} q_t(x_a | x_{\pa(a)}) \, \prod_{b\in B} \bar p(x_b | x_{\pa(b)})\]
   \[\newcommand{\pa}{\text{pa}}  \;\;\;\;\;\;\;= \prod_{a\in A} p_t(x_a | x_{\pa(a)},g) \, \prod_{b\in B} \bar p(x_b | x_{\pa(b)})\]

Designing artificial agents

Use planning as inference for agent design:

Note: Bayesian network structure of

• fixed kernels expresses/defines structure of problem
• controllable kernels expresses/defines structure of solutions

Controllable kernel structure can be used to make internal structure of designed agent explicit!

Designing artificial agents

Consider designing an artificial agent for a POMDP i.e. you know

• dynamics of environment \(E\) the agent will face
• sensor values \(S\) available to it
• actions \(A\) it can take
• goal \(G\) (or reward) it should achieve

Then find

• kernels that produce actions \(A_t\)
• maximize goal probability

via planning as inference.

Designing artificial agents

Get three kernels:

• \(p(a_0|s_0)\)
• \(p(a_1|s_0,a_0,s_1)\)
• \(p(a_2|s_0,a_0,s_1,a_1,s_2)\)

But in practice (on a robot) time passes between \(S_0\) and \(A_2\):

• if \(A_2\) depends on \(S_0\)
  • have to store \(S_0\)
  • but where?

Text

Designing artificial agents

Action maybe produced by three kernels:

• \(p(a_0|s_0)\)
• \(p(a_1|s_0,a_0,s_1)\)
• \(p(a_2|s_0,a_0,s_1,a_1,s_2)\)

In practice time passes between \(S_0\) and \(A_2\):

• if \(A_2\) depends on \(S_0\)
  • have to store \(S_0\)

So also need to find

• dynamics of memory \(M\)

Also want constant memory and action kernels

• \(p(m_{t_1}|s_{t_1},m_{t_1-1})=p(m_{t_2}|s_{t_2},m_{t_2-1})\)

Designing artificial agents

In following "agent" usually means

• two Markov kernels
  • memory update \[p(m_t|s_t,m_{t-1})\]
  • action selection \[p(a_t|m_t)\]
• constant over time (if there is time)
  \[p(m_{t_1}|s_{t_1},m_{t_1-1})=p(m_{t_2}|s_{t_2},m_{t_2-1})\]
  \[p(a_{t_1}|m_{t_1})=p(a_{t_2}|m_{t_2})\]
• (formally: use shared parameters...)

Two situations:

• designer uncertain about environment dynamics
• agent supposed to deal with different environments

Explicitly reflect either by

• additional variable \(\phi\)
• prior distribution over \(\phi\)

Application: Designer uncertainty

Then

• resulting agent will find out what is necessary to achieve goal
• i.e. it will trade off exploration (try out different arms) and exploitation (winning)
• comparable to meta-learning

Application: Designer uncertainty

Example: 2-armed bandit

• Constant hidden "environment state" \(\phi=(\phi_1,\phi_2)\) storing win probabilities of two arms
• agent action is choice of arm \(a_t \in \{1,2\}\)
• sensor value is either win or lose sampled according to win probability of chosen arm \(s_t \in \{\text{win},\text{lose}\}\)
  \[p_{S_t}(s_t|a_{t-1},\phi)=\phi_{a_{t-1}}^{\delta_{\text{win}}(s_t)} (1-\phi_{a_{t-1}})^{\delta_{\text{lose}}(s_t)}\]
• goal is achieved if last arm choice results in win \(s_3=\text{win}\)
  \[p_G(G=1|s_3)=\delta_{\text{win}}(s_3)\]
• memory \(m_t \in \{1,2,3,4\}\) is enough to store all different results.

Alternatively:

• construct (part of) memory update function
• guarantee existence of notion of knowledge e.g. Bayesian beliefs

Dynamically changing goals

Interpreting systems a Bayesian reasoners

Recall Bayes rule (for any random variables \(A,B\)):

\[p(b\,|\,a) = \frac{p(a\,|\,b)}{p(a)} \;p(b)\]

• where
  \[p(a) = \sum_b p(a\,|\,b)\;p(b)\]
• if we know \(p(a\,|\,b)\) and \(p(b)\) we can compute \(p(b\,|\,a)\)
• nice.

Interpreting systems a Bayesian reasoners

Bayesian inference:

• parameterized model: \(p(x|\theta)\)
• prior belief over parameters \(p(\theta)\)
• observation / data: \(\bar x\)
• Then according to Bayes rule:
  
  \[p(\theta|x) = \frac{p(x|\theta)}{p(x)} p(\theta)\]

Interpreting systems a Bayesian reasoners

Bayesian belief updating:

• introduce time steps \(t\in \{1,2,3...\}\)
• initial belief \(p_1(\theta)\)
• sequence of observations \(x_1,x_2,...\)
• use Bayes rule to get \(p_{t+1}(\theta)\):
  
  \[p_{t+1}(\theta) = \frac{p(x_t|\theta)}{p_t(x_t)} p_t(\theta)\]

Interpreting systems a Bayesian reasoners

Bayesian belief updating with conjugate priors:

• now parameterize the belief
  using the "conjugate prior" for model \(p(x|\theta)\)
  \[p_1(\theta) \mapsto p(\theta|\alpha_1) \]
• \(\alpha\) are called hyperparameters
• plug this into Bayesian belief updating:
  
  \[p(\theta|\alpha_{t+1}) = \frac{p(x_t|\theta)}{p(x_t|\alpha_t)} p(\theta|\alpha_t)\]
  where \(p(x_t|\alpha_t) := \int_\theta p(x_t|\theta)p(\theta|\alpha_t) d\theta\)
• So what?

Interpreting systems a Bayesian reasoners

Bayesian belief updating with conjugate priors:

• well then: there exists a hyperparameter translation function \(f\) such that
  \[\alpha_{t+1} = f(x,\alpha_t)\]
• maps current hyperparameter \(\alpha_t\) and observation \(x\) to next hyperparameter \(\alpha_{t+1}\)
• but beliefs are determined by the hyperparamter!
• we can always recover them (if we want) but for computation we can forget all the probabilities!
  \[p(\theta|f(x_t,\alpha_t)) = \frac{p(x_t|\theta)}{p(x_t|\alpha_t)} p(\theta|\alpha_t)\]

Interpreting systems a Bayesian reasoners

Bayesian belief updating with conjugate priors:

• so if we have a conjugate prior we can implement Bayesian belief updating with a function
  \[f:X \times \mathcal{A} \to \mathcal{A}\]
  \[\alpha_{t+1}= f(x,\alpha_t)\]
• this is just a (discrete time) dynamical system with inputs
• Interpreting systems as Bayesian reasoners just turns this around!

Interpreting systems a Bayesian reasoners

• given discrete dynamical system with input defined by
  \[\mu: S \times M \to M\]
  \[m_{t+1}=\mu(s_t,m_t)\]
• a consistent Bayesian inference interpretation is:
  • interpretation map \(\psi(\theta|m)\)
  • model \(\phi(s|\theta)\)
  • such that the dynamical system acts like a hyperparameter translation function i.e. if:
    \[\psi(\theta|\mu(s_t,m_t)) = \frac{\phi(s_t|\theta)}{(\phi \circ \psi)(s_t|m_t)} \psi(\theta|m_t)\]

Note:

• can then also turn this information gain into a goal to create intrinsically motivated agent
• from report:

Dynamically changing goals

Designing artificial agents

Consider agent that solves a problem in uncertain environment.

• but if we construct (part of) the memory update function then we can guarantee the existence of a notion of knowledge e.g. Bayesian beliefs
• this gives us a way to turn expected information gain maximization into a goal.

Multiple, possibly competing goals

• want to harness power of collectives
• both collaboration and competition may be beneficial for collective
• to understand possibilities consider incompatible goals -> game theory
• problems can be expressed using multiple goal nodes
• solving them can be hard (we found out and it is known)
• when and why which algorithm works to be seen
• (may shed light on why GANs work)

Multi-agent setup

Example multi agent setups:

Two agents interacting with same environment

Two agents with same goal

Two agents with different goals

Multi-agent setup

• Note:
  • In cooperative setting:
    • can often combine multiple goals to single common goal via event intersection, union, complement (supplied by \(\sigma\)-algebra)
    • single goal manifold
    • in principle can use single agent PAI as before

Multi-agent setup

• Note:
  • In non-cooperative setting:
    • goal events have empty intersection
      • no common goal
      • multiple disjoint goal manifolds

Multi-agent setup

Example non-cooperative game: matching pennies

• Each player \(P_i \in \{1,2\}\) controls a kernel \(p(a_i)\) determining probabilities of heads and tails
• First player wins if both pennies are equal second player wins if they are different

Multi-agent setup

Example non-cooperative game: matching pennies

• Each player \(P_i \in \{1,2\}\) controls a kernel \(p(a_i)\) determining probabilities of heads and tails
• First player wins if both pennies are equal second player wins if they are different

joint pdists \(p(a_1,a_2)\)

disjoint goal manifolds

agent manifold

\(p(a_1,a_2)=p(a_1)p(a_2)\)

Non-cooperative games

• In non-cooperative setting
  • instead of maximizing goal probability:
    • find Nash equilibria
  • can we adapt PAI to do this?
    • established that using EM algorithm alternatingly does not converge to Nash equilibria
    • other adaptations may be possible...

Non-cooperative games

• Counterexample for multi-agent alternating EM convergence:
  • Two player game: matching pennies
    • Each player \(P_i \in \{1,2\}\) controls a kernel \(p(a_i)\) determining probabilities of heads and tails
    • First player wins if both pennies are equal second player wins if they are different

Non-cooperative games

• Counterexample for multi-agent alternating EM convergence:
  • Nash equilibrium is known to be both players playing uniform distribution
  • Using EM algorithm to fully optimize player kernels alternatingly does not converge
  • Taking only single EM steps alternatingly also does not converge

EM

EM

Non-cooperative games

• Counterexample for multi-agent alternating EM convergence
  • fix a strategy for player 2 e.g. \(p_0(A_2=H)=0.2\)

Non-cooperative games

Text

• run EM algorithm for P1
  • Ends up on edge from (tails,tails) to (tails,heads)
• result: \(p(A_1=T)=1\)
• then optimizing P2 leads to \(p(A_2=H)=1\)
• then optimizing P1 leads to \(p(A_1=H)=1\)
• and on and on...

Non-cooperative games

Text

• Taking one EM step for P1 and then one for P2 and so on...
• ...also leads to loop.

Cooperative games

• Concrete example of agent manifold reduction under switch from single agent to multi-agent to identical multi-agent setup
  • Like matching pennies but single goal: "different outcome"
    • solution: one action/player always plays heads and one always tails
  • agent manifold:
    • single-agent manifold would be whole simplex
    • multi-agent manifold is independence manifold
    • multi-agent manifold with shared kernel is submanifold of independence manifold

Cooperative games

2. Agent number can change from \(n\) to \(m\) depending on events at runtime

• Bayesian networks: network structure can't change due to events
• Need more general setting.        

Dynamic scalability of multi-agent systems

Can't be combined in one Bayesian network!

• E.g.: agent number can't depend on environment state:

Bayesian network limitations

Can't be combined in one Bayesian network!

• E.g.: copy direction can't be changed
  • if \(A=0\) let \(p(c|b)=\delta_b(c)\)
  • if \(A=1\) let \(p(b|c)=\delta_c(b)\)

Can't be combined in one Bayesian network!

Bayesian network limitations

• Rollouts in Bayesian networks instantiate all variables
  • Forks don't represent alternative consequences
  • Forks represent multiple "parallel" consequences!

?

?

?

Bayesian network limitations

• Rollouts in Bayesian networks instantiate all variables
  • Forks don't represent alternative consequences
  • Forks represent multiple "parallel" consequences!

?

Bayesian network limitations

• Rollouts in Bayesian networks instantiate all variables
  • Forks don't represent alternative consequences
  • Forks represent multiple "parallel" consequences!

Bayesian network limitations

Expected reward maximization

• PAI finds policy that maximizes probability of the goal event:
  \[P^* = \text{arg} \max_{P \in M_A} p(G=1).\]
• Often want to maximize expected reward of a policy:
  \[P^* = \text{arg} \max_{P \in M_A} \mathbb{E}_P[r]\]
• Can we solve the second problem via the first?
• Yes, at least if reward has a finite range \([r_{\text{min}},r_{\text{max}}]\):
  • add binary goal node \(G\) to Bayesian network and set:
    \[\newcommand{\ma}{{\text{max}}}\newcommand{\mi}{{\text{min}}}\newcommand{\bs}{\backslash}p(G=1|x):= \frac{r(x)-r_\mi}{r_\ma-r_\mi}.\]

Shared kernels

• Introduce "types" for agent kernels
  • let \(c(a)\) be the type of kernel \(a \in A\)
  • kernels of same type share
    • input spaces
    • output space
    • parameter
  • then \(p_{c(a)}(x_a|x_{\text{pa}(a)},\theta_c)\) is the kernel of all nodes with \(c(a)=c\).

Shared kernels

• Example agent manifold change under shared kernels