Interpreting systems as solving POMDPs: a step towards a formal understanding of agency

Martin Biehl (Cross Labs)

Nathaniel Virgo (Earth-Life Science Institute)

Made possible via funding by:

Background

agents?

What is the difference?

non-agents?

Background

Physics:

• laws of physics are the same everywhere
  • no matter whether there is a rock or a human / animal / agent
• no difference?

Background

Daniel Dennett:

• can take various stances towards any object
• stance with better prediction of the object behavior "wins"

Two stances (ignore design stance):

• physical stance: use physical constituents of object and laws of physics
• intentional stance: assume object
  • has beliefs 
  • has a goal
  • acts rationally (=optimally) to achieve goal under its beliefs

Background

Daniel Dennett:

• can take various stances towards any object
• stance with better prediction of the object behavior "wins"

Two stances (ignore design stance):

• physical stance: use physical constituents of object and laws of physics
• intentional stance: assume object
  • has beliefs
  • has a goal
  • acts rationally (=optimally) to achieve goal under its beliefs

Background

Our approach:

• does not apply to actual objects (rocks / animals)
  • only applies to well defined/abstract systems (Moore machines)
• does not compare prediction performance of stances
  • instead checks whether interpretation in terms of beliefs and goal is consistent with system dynamics

Moore machine definition

stochastic Moore machine:

• internal state space \(\mathcal{M}\)
• input space \(\mathcal{I}\)
• output space \(\mathcal O\)
• machine kernel \(\mu:\mathcal{I}\times \mathcal{M}\to P\mathcal{M}\)
• output function \(\omega: \mathcal M \to \mathcal O\)

notation:

• \(\mu(m'|i,m)\) probability of next state \(m'\) given input \(i\) and state \(m\)
• \(\omega(m)\) output in state \(m\)

Informal answer

Given stochastic Moore machine (without any environment):

Question: When is it justified to call it an agent?

Proposal: Justified to call it "agent" if find

• consistent interpretation in terms of
  • beliefs
  • goal
  • optimal/rational actions to achieve goal according to belief

Moore machine interpretation via POMDP

Formal answer:

• consistent interpretation as a solution to a POMDP

  • this consists of

    • a POMDP and

    • its solution (optimal policy on beliefs)

    • interpretation function

  • satisfying

    • a consistency equation (Bayesian updating)

    • optimal action equation

Standard solution of POMDP (well known) induces a Moore machine:

• states are beliefs (=probability distributions) \(b \in P \mathcal H\) over hidden states
• machine kernel \(\delta_{f^*}: \mathcal I \times P \mathcal H \to P\mathcal H\) update beliefs according to Bayes rule
• outputs are optimal policy:
  • function \(\pi^*:P\mathcal H \to \mathcal O\) from beliefs over hidden states to actions/outputs

Moore machine interpretation via POMDP

This machine has beliefs, a goal, and acts optimally \(\Rightarrow\) can call it an agent!

Standard solution of POMDP (well known) induces a Moore machine:

• states are beliefs (=probability distributions) \(b \in P \mathcal H\) over hidden states
• machine kernel \(\delta_{f^*}: \mathcal I \times P \mathcal H \to P\mathcal H\) update beliefs according to Bayes rule
• outputs are optimal policy:
  • function \(\pi^*:P\mathcal H \to \mathcal O\) from beliefs over hidden states to actions/outputs

Moore machine interpretation via POMDP

But most machines don't have states that are probability distributions \(\Rightarrow\) can they be agents too?

Interpretation function:

• \(\psi:\mathcal M \to P\mathcal H\)
• specifies belief \(\psi(m)\) for each internal state \(m\) of Moore machine
  
  
  
   
• also write \(\psi(h|m)\) for probability of hidden state \(h \in \mathcal H\) specified for machine state \(m \in \mathcal M\)

Moore machine interpretation via POMDP

Interpretation function:

Moore machine interpretation via POMDP

belief \(\psi(h|m)\) before update

belief \((\psi\circ \mu)(h|i,m)\) after update

Consistency equation:

• ensures that Bayes rule holds for specified beliefs (not internal states)

Optimal action equation:

• ensures that output equal to optimal action for specified belief \(\psi(m)\):

Moore machine interpretation via POMDP

Can show:

• if Moore machine and interpretation map obey these equations:
  • then Moore machine also maximizes expected reward

Reason:

• it updates beliefs specified by \(\psi\) in same way as standard solution
• for each specified belief \(\psi(m)\) it outputs the same action as the standard solution

Moore machine interpretation via POMDP

But:

• no assumption of environment
• no ergodicity assumptions
• only need Moore machine to decide about agency!

Relation to FEP

• Showed a way to justify that stochastic Moore machine is an agent
• used:
  • standard solutions to POMDPs are basically agents with beliefs, goals, and rationality
  • interpretation function extends this to machines whose state spaces aren't probability distributions directly
• Maybe similar approach for other systems exists?
• https://arxiv.org/pdf/2209.01619.pdf

Summary

End

• Agent is consistent pair of system and interpretation
  
• if we choose
  • Moore machines as systems
  • interpretation as solution to POMDP
• we gave formal equations to check consistency
• https://arxiv.org/pdf/2209.01619.pdf

Summary

One possible answer:

• consistent interpretation as a solution to a POMDP

  • this consists of

    • a POMDP i.e.:

      • hidden state space \(\mathcal{H}\)
      • transition kernel \(\kappa:\mathcal{H}\times \mathcal{O}\to P(\mathcal{H}\times\mathcal I)\)
      • reward function \(r: \mathcal H \times \mathcal A \to \mathbb R\)

    • interpretation function \(\psi: \mathcal M \to P\mathcal H\)

  • satisfying

    • a consistency equation

    • optimal action equation

Moore machine interpretation via POMDP

consistency equation

optimal action equation

Moore machine interpretation via POMDP

Moore machine

POMDP transition kernel

interpretation map

consistency equation

optimal action equation

Moore machine interpretation via POMDP

Agent definition

One possible answer:

• consistent interpretation as a solution to a POMDP

  • this consists of

    • a POMDP

    • interpretation function \(\psi: \mathcal M \to P\mathcal H\)

  • satisfying

    • a consistency equation
      
      
      
      
       

    • optimal action equation

Agent definition

One possible answer:

• consistent interpretation as a solution to a POMDP

  • this consists of

    • a POMDP

    • interpretation function \(\psi: \mathcal M \to P\mathcal H\)

  • satisfying

    • a consistency equation

    • optimal action equation

Agent definition

Lucky observation:

• exist Moore machines that incontrovertibly have such interpretation:
  • they solve to partially observable Markov decision processes (POMDPs)
  • should count as agents

Then:

• extend set of those Moore machines

Agent definition

Argumentation:

• POMDPs have optimal solutions where
  • assume belief (distribution) over hidden state
  • update belief via Bayes' rule
  • optimal action is function of this belief
• if we consider
  • beliefs as internal states 
  • Bayes rule as machine kernel
  • optimal action as output for given belief/internal state
• we get a (deterministic) Moore machine and a justified to say it
  • has beliefs
  • has a goal (rewards of the POMDP)
  • acts rationally according to belief to achieve goal

Agent definition

Argumentation:

• POMDPs have optimal solutions where
  • assume belief (distribution) over hidden state
  • update belief via Bayes' rule
  • optimal action is function of this belief
• if we consider
  • beliefs as internal states
  • Bayes rule as machine kernel
  • optimal action as output for given belief/internal state
• we get a (deterministic) Moore machine and a justified to say it
  • has beliefs
  • has a goal (rewards of the POMDP)
  • acts rationally according to belief to achieve goal

is an agent!

Agent definition

Argumentation:

• but can extend this:
  • internal state space need not be space of belief
    • only needs to specify/determine beliefs
• thus:
  • weaker condition on (stochastic) Moore machines exists that guarantees:
    • beliefs are specified by internal state
    • Bayes rule holds for specified beliefs not internal states directly
    • outputs are optimal actions for specified beliefs for given POMDP

also an agent!

POMDP solution

Given a POMDP with hidden state space \(\mathcal H\)

• introduce "belief distribution" \(b:\mathcal H \to [0,1]\) over \(\mathcal H\)
• write \(b(h)\) for probability/confidence that hidden state is \(h \in \mathcal H\)
• take action \(a \in \mathcal A\)
• observe sensor value \(s \in \mathcal S\)
• then using transition kernel \(\kappa(h',s|h,a)\) Bayes rule specifies how to update the belief distribution to \(b'(h')\):

POMDP solution

Given a POMDP with hidden state space \(\mathcal H\)

• introduce "belief distribution" \(b:\mathcal H \to [0,1]\) over \(\mathcal H\)
• write \(b(h)\) for probability/confidence that hidden state is \(h \in \mathcal H\)
• take action \(a \in \mathcal A\)
• observe sensor value \(s \in \mathcal S\)
• then using transition kernel \(\kappa(h',s|h,a)\) Bayes rule specifies how to update the belief distribution to \(b'(h')\):

POMDP solution

Given a POMDP with hidden state space \(\mathcal H\)

• introduce "belief distribution" \(b:\mathcal H \to [0,1]\) over \(\mathcal H\)
• write \(b(h)\) for probability/confidence that hidden state is \(h \in \mathcal H\)
• take action \(a \in \mathcal A\)
• observe sensor value \(s \in \mathcal S\)
• then using transition kernel \(\kappa(h',s|h,a)\) Bayes rule specifies how to update the belief distribution to \(b'(h')\):
  
  
   
• optimal policy is function of this belief, and can be obtained via belief-MDP and optimal value function \(V^*:P\mathcal H \to \mathbb R\) on beliefs

POMDP solution

Given a POMDP with hidden state space \(\mathcal H\)

• optimal value function solves Bellman equation:
  
  
  
  
   
• 
  
  
    

Moore machine that solves POMDP

Given a POMDP construct Moore machine

• choose
  • \(P\mathcal H\) as internal state space
  • \(\mathcal S\) as input space
  • \(\mathcal A\) as output space
  • \(\pi^*:P\mathcal H \to \mathcal A\) as output function
• given optimal policy \(\pi^*:P\mathcal H \to \mathcal A\)
  • choosing optimal action \(a=\pi^*(b)\) we can
  • turn belief update into Moore machine transition kernel:
    • \(f_{\pi^*}(s,b):=f(b,\pi^*(b),s)\)
    • \(\mu(b'|s,b):=\delta_{f_{\pi^*}(s,b)}(b')\)

Justified to call this a rational agent with beliefs and a goal!

Moore machine with interpretation

Yey! We got some agent! But we can find more! Some machines

• do everything necessary to specify the beliefs and actions that solve a POMDP
• but have state spaces that aren't spaces of probability distributions themselves (they are like sufficient statistics a bit)
• for this we need to specify an interpretation map \(\psi:\mathcal M \to P\mathcal H\) such that the machine kernel must obey a consistency equation with respect to the POMDP transition kernel:

Simplified notation:

• specify an interpretation map \(\psi:\mathcal M \to P\mathcal H\) such that the machine kernel \(\mu\) must obey a consistency equation with respect to the POMDP transition kernel \(\kappa\)
• define prior over next hidden state and input/sensor value:
  
  
   
• and only over next sensor value:
  
   
• then consistency means (if \(\psi_{\mathcal I}(i|m)>0\)):

Moore machine with interpretation

Informal image:

Moore machine with interpretation

Main statement:

Given a Moore machine a consistent interpretation as a solution to a POMDP is given by:

• a POMDP
• an interpretation map

such that

1. machine kernel \(\mu\) is consistent with interpretation map \(\psi\) and POMDP transition kernel \(\kappa\)
2. output function of Moore machine select optimal action:

Moore machine with interpretation

String diagram consistency

Interpreting systems a Bayesian reasoners

Recall Bayes rule (for any random variables \(A,B\)):

\[p(b\,|\,a) = \frac{p(a\,|\,b)}{p(a)} \;p(b)\]

• where
  \[p(a) = \sum_b p(a\,|\,b)\;p(b)\]
• if we know \(p(a\,|\,b)\) and \(p(b)\) we can compute \(p(b\,|\,a)\)
• nice.

Interpreting systems a Bayesian reasoners

Bayesian inference:

• parameterized model: \(p(x|\theta)\)
• prior belief over parameters \(p(\theta)\)
• observation / data: \(\bar x\)
• Then according to Bayes rule:
  
  \[p(\theta|x) = \frac{p(x|\theta)}{p(x)} p(\theta)\]

Interpreting systems a Bayesian reasoners

Bayesian belief updating:

• introduce time steps \(t\in \{1,2,3...\}\)
• initial belief \(p_1(\theta)\)
• sequence of observations \(x_1,x_2,...\)
• use Bayes rule to get \(p_{t+1}(\theta)\):
  
  \[p_{t+1}(\theta) = \frac{p(x_t|\theta)}{p_t(x_t)} p_t(\theta)\]

Interpreting systems a Bayesian reasoners

Bayesian belief updating with conjugate priors:

• now parameterize the belief
  using the "conjugate prior" for model \(p(x|\theta)\)
  \[p_1(\theta) \mapsto p(\theta|\alpha_1) \]
• \(\alpha\) are called hyperparameters
• plug this into Bayesian belief updating:
  
  \[p(\theta|\alpha_{t+1}) = \frac{p(x_t|\theta)}{p(x_t|\alpha_t)} p(\theta|\alpha_t)\]
  where \(p(x_t|\alpha_t) := \int_\theta p(x_t|\theta)p(\theta|\alpha_t) d\theta\)
• So what?

Interpreting systems a Bayesian reasoners

Bayesian belief updating with conjugate priors:

• well then: there exists a hyperparameter translation function \(f\) such that
  \[\alpha_{t+1} = f(x,\alpha_t)\]
• maps current hyperparameter \(\alpha_t\) and observation \(x\) to next hyperparameter \(\alpha_{t+1}\)
• but beliefs are determined by the hyperparamter!
• we can always recover them (if we want) but for computation we can forget all the probabilities!
  \[p(\theta|f(x_t,\alpha_t)) = \frac{p(x_t|\theta)}{p(x_t|\alpha_t)} p(\theta|\alpha_t)\]

Interpreting systems a Bayesian reasoners

Bayesian belief updating with conjugate priors:

• so if we have a conjugate prior we can implement Bayesian belief updating with a function
  \[f:X \times \mathcal{A} \to \mathcal{A}\]
  \[\alpha_{t+1}= f(x,\alpha_t)\]
• this is just a (discrete time) dynamical system with inputs
• Interpreting systems as Bayesian reasoners just turns this around!

Interpreting systems a Bayesian reasoners

• given discrete dynamical system with input defined by
  \[\mu: S \times M \to M\]
  \[m_{t+1}=\mu(s_t,m_t)\]
• interpret it as implementing belief updating if find:
  • interpretation map \(\psi(\theta|m)\)
  • model \(\phi(s|\theta)\)
  • such that the dynamical system acts like a hyperparameter translation function i.e. if:
    \[\psi(\theta|\mu(s_t,m_t)) = \frac{\phi(s_t|\theta)}{(\phi \circ \psi)(s_t|m_t)} \psi(\theta|m_t)\]

Motivation / big picture

• Can we formalize the process of designing artificial agents?

Then could

• think more precisely about design choices
• classify kinds of problems in machine learning
• identify issues that hold us back from designing more interesting agents (autonomous agents / open ended agents etc)

Overview

• Designing artificial agents using probabilistic inference
• Two applications of approach:
  • Designer uncertainty / problem uncertainty
  • Agent uncertainty
• Recap
• Issues

Applications of this approach

Two situations:

• designer uncertain about environment dynamics
• agent supposed to deal with different environments

Explicitly reflect either by

• additional variable \(\phi\)
• prior distribution over \(\phi\)

Designer uncertainty

Then

• resulting agent will find out what is necessary to achieve goal
• i.e. it will trade off exploration (try out different arms) and exploitation (winning)
• comparable to meta-learning

Designer uncertainty

Then can make uncertainty explicit:

• choose model of causes of sensor values for agent
  
• fix memory dynamics \(p_M(m_t|s_t,m_{t-1})\) such that it has consistent Bayesian interpretation w.r.t chosen model
• only action kernels \(p_A(a_t|m_t)\) unknown

Agent uncertainty

Then by construction

• know a consistent Bayesian interpretation
• i.e.\ an interpretation map \(\psi:M \to PH\)
• well defined agent uncertainty in state \(m_t\) via Shannon entropy \[H_{\text{Shannon}}(m_t):=\sum_{h} \psi(h||m_t) \log \psi(h||m_t)\]
• information gain from \(m_0\) to \(m_t\) by KL divergence \[D_{\text{KL}}[m_t||m_0]:=\sum_h \psi(h||m_t) \log \frac{\psi(h||m_t)}{\psi(h||m_0)}\]

Agent uncertainty

Note:

• can then also turn this information gain into a goal to create intrinsically motivated agent
• (some trick is needed...)

Agent uncertainty

• Considered design of artificial agents as planning task
• can (sometimes) turn this into inference problem in Bayesian network
• can explicitly reflect designer's uncertainty / knowledge
  • resulting agents automatically trade of exploration and exploitation
• constructing interpretable agent memory allows
  • well defined (subjective) agent uncertainty and information gain
  • design of agents maximizing info gain

Recap

• Why don't machine learning paper all start with a Bayesian network that shows what kind of agent they are designing?
  • Can't always use Bayesian networks:
    • if dependencies of variables depend on values of variables
    • e.g. number of timesteps to finish task depends on actions
  • Finding correct Bayesian network can be subtle

• Hope to find better formalism that makes something like this possible...

Issues

Perspective: "design as inference"

Underlying perspective:

• view design of artificial agents as planning task:
  • usually planning means find (own) actions that achieve a goal
  • when designing an artificial agent also create a kind of "plan" but
    • it will be executed by something else (e.g. computer)
    • must explicitly plan "internal actions" like what to remember (memory update)

Perspective: "design as inference"

Underlying perspective:

• design of artificial agents similar to planning task:
  • usually planning means find (own) actions that achieve a goal
  • when designing an artificial agent to achieve a goal:
    • also create a kind of "plan" but
    • plan will be executed by something else (e.g. computer)
    • plan must include "internal actions" like plan of what to remember (memory update)

Underlying perspective:

• known: planning can be done by probabilistic inference

\(\Rightarrow\) if we formulate agent design problems as planning problems they become inference problems

Perspective: "design as inference"

Perspective: design as planning

Underlying perspective:

• view design of artificial agents as planning task:
• use formal way to represent this planning task (planning as inference)
• gives us formal way to design agents

design as planning \(\to\) planning as inference

\(\Rightarrow\) design as inference?

General setting

Assume

• want to achieve some goal
• know:
  • the environment you want to achieve it in
  • actions you can take
  • observations you will be able to make

Then

• planning is the process of finding actions that lead to goal

POMDPs

Formalize as POMDP:

• Known:
  • goal
  • environment/world state \(W\) with dynamics: \(p(w_{t+1}|w_t,a_t)\)
  • observations / sensor values \(S\): \(p(s_t|w_t)\)
•  

Planning as inference

Terminology:

• Planning:
  • finding policy parameters that maximize probability of achieving a goal
• Maximum likelihood inference:
  • finding model parameters that maximize probability of data

Planning as inference

What is it good for?

Automatically find a probabilistic policy to achieve a goal.

What do you need to use it?

• probabilistically specified problem!
  • dynamics of environment (including influence of actions)
  • inputs / sensor values / observation available to policy
  • goal
  •  

Planning as inference

Combination:

• Planning as inference (PAI):
  • Consider the achievement of the goal as the only available data
  • ensure policy parameters are the only model parameters
  • then maximizing likelihood of data maximizes likelihood of achieving the goal

\(\Rightarrow\) can use max. likelihood to solve planning!

Planning as inference

Note that for maximum lik

1. Set of observations can be arbitrarily simple
2. Set of probability distributions can be arbitrarily complicated

So we can use :

1. Binary "goal-achieved-variable" with \(G=\{g,\neg g\}\) for observations
2. Bayesian networks (with hidden variables) for specifying the sets of probability distributions

Planning as inference

• Problem structures as Bayesian networks:

Planning as inference

Bayesian network

goal

• Represent goals and policies by sets of probability distributions:
  • goal must be an event i.e. function \(G(x)\) with
    • \(G(x)=1\) if goal is achieved
    • \(G(x)=0\) else.
  • goal manifold is set of distributions where the goal event occurs with probability one:
    \[M_G:=\{P: p(G=1)=1\}\]

Planning as inference

• Find policies that maximize probability of goal via geometric EM algorithm:
  • planning as inference finds policy / agent kernels such that:
    \[P^* = \text{arg} \max_{P \in M_A} p(G=1).\]
  • (compare to maximum likelihood inference)

Bayesian network

goal

policies

Planning as inference

Practical side of original framework:

• Represent
  • Planning problem structure by Bayesian networks
  • Goal and possible policies by sets of probability distributions
• Find policies that maximize probability of goal via geometric EM algorithm.

   

Bayesian network

goal

policies

Expected reward maximization

• Example application: Markov decision process
  • reward only depending on states \(S_0,...,S_3\): \[r(x):=r(s_0,s_1,s_2,s_3)\]
  • reward is sum over reward at each step:
    \[r(s_0,s_1,s_2,s_3):= \sum_i r_i(s_i)\]

Planning as inference

• Represent goals and policies by sets of probability distributions:
  • policy is a choice of the changeable Markov kernels
    \[\newcommand{\pa}{\text{pa}}\{p(x_a | x_{\pa(a)}):a \in A\}\]
  • agent manifold/policy manifold is set of distributions that can be achieved by varying policy
    \[\newcommand{\pa}{\text{pa}}p(x) = \prod_{a\in A} p(x_a | x_{\pa(a)}) \, \prod_{b\in B} \bar p(x_b | x_{\pa(b)})\]

Bayesian network

goal

policies

Planning as inference

• Find policies that maximize probability of goal via geometric EM algorithm:
  • Can prove that \(P^*\) is the distribution in agent manifold closest to goal manifold in terms of KL-divergence
  • Local minimizers of this KL-divergence can be found with the geometric EM algorithm

Bayesian network

goal

policies

Planning as inference

• Find policies that maximize probability of goal via geometric EM algorithm:

1. Start with an initial prior, \(P_0 \in M_A\) .
2. (e-projection)
   \[Q_t = \text{arg} \min_{Q\in M_G} D_{KL}(Q∥P_t )\]
3. (m-projection)
   \[P_{t+1} = \text{arg} \min_{P \in M_A} D_{KL} (Q_t ∥P )\]

Bayesian network

goal

policies

Planning as inference

• Find policies that maximize probability of goal via geometric EM algorithm:
• Equivalent algorithm using only marginalization and conditioning:

1. Initial agent kernels define prior, \(P_0 \in M_A\).
2. Get  \(Q_t\), from \(P_t\) by conditioning on the goal:  \[q_t(x) = p_t(x|G=1).\]
3. Get \(P_{t+1}\), by replacing agent kernels by conditional distributions in \(Q_t\):
   \[\newcommand{\pa}{\text{pa}} p_{t+1}(x) = \prod_{a\in A} q_t(x_a | x_{\pa(a)}) \, \prod_{b\in B} \bar p(x_b | x_{\pa(b)})\]
   \[\newcommand{\pa}{\text{pa}}  \;\;\;\;\;\;\;= \prod_{a\in A} p_t(x_a | x_{\pa(a)},g) \, \prod_{b\in B} \bar p(x_b | x_{\pa(b)})\]

Uncertain (PO)MDP

• Assume
  • (as usual) transition kernel of environment is constant over time, but
  • we are uncertain about what is the transition kernel
• How can we reflect this in our setup / PAI?
• Can we find agent kernels that solve problem in a way that is robust against variation of those transition kernels?

Uncertain (PO)MDP

• Extend original (PO)MDP Bayesian network with two steps:
  • parametrize environment transition kernels by shared parameter \(\phi\):
    \[\bar{p}(x_e|x_{\text{pa}(e)}) \to \bar{p}_\phi(x_e|x_{\text{pa}(e)},\phi)\]
  • introduce prior distribution \(\bar{p}(\phi)\) over environment parameter

Uncertain (PO)MDP

• Same structure can be hidden in original network but in this way becomes a requirement/constraint
• If increasing goal probability involves actions that resolve uncertainty about the environment then PAI finds those actions!
• PAI results in curious agent kernels/policy.

Uncertain (PO)MDP

• Same structure can be hidden in original network but in this way becomes a requirement/constraint
• If increasing goal probability involves actions that resolve uncertainty about the environment then PAI finds those actions!
• PAI results in curious agent kernels/policy.

Uncertain (PO)MDP

•  Relevance for project:
  • agents that can achieve goals in unknown / uncertain environments are important for AGI
  • related to meta-learning
  • understanding knowledge and uncertainty representation is important for agent design in general

Related project funded

• John Templeton Foundation has funded related project titled: Bayesian agents in a physical world
• Goal:
  • What does it mean that a physical system (dynamical system) contains a (Bayesian) agent?

Related project funded

• Starting point:
  • given system with inputs defined by
    \[f:C \times S \to C\]
  • define a consistent Bayesian interpretation as:
    1. model / Markov kernel \(q: H \to PS\)
    2. interpretation function \(g:C \to PH\)
  such that \[g(c_{t+1})(h)=g(f(c_t,s_t))(h)=\frac{q(s_t|h) g(c_t)(h)}{\sum_{\bar h} q(s_t|\bar h) g(c_t)(\bar h)} \]

Related project funded

• more suggestive notation:
  \[g(h|c_{t+1})=g(h|f(c_t,s_t))=\frac{q(s_t|h) g(h|c_t)}{\sum_{\bar h} q(s_t|\bar h) g(\bar h|c_t)} \]
• but note: \(PH_i\) are deterministic random variables and need no extra sample space
• \(H\) isn't even a deterministic random variable (what???)

Related project funded

• Take away message :
  • Formal condition for when a dynamical system with inputs can be interpreted as consistently updating probabilistic beliefs about the causes of its inputs (e.g. environment)
  • Extensions to include stochastic systems, actions, goals, etc. ongoing...

Related project funded

• Relevance for project
  • deeper understanding of relation between physical systems and agents will also help in thinking about more applied aspects  
  • a lot of physical agents are made of smaller agents and grow / change their composition, understanding this is also part of the funded project and is also directly relevant for the point "dynamical scalability of multi-agent systems" in the proposal

Two kinds of uncertainty

• Designer uncertainty:
  • model our own uncertainty about environment when designing the agent to make it more robust / general
• Agent uncertainty:
  • constructing an agent that uses specific probabilistic belief update method
    • exact Bayesian belief updating (exponential families and conjugate priors)
    • approximate belief updating (VAE world models?)

Two kinds of uncertainty

• Designer uncertainty:
  • introduce hidden parameter \(\phi\) with prior \(\bar p(\phi)\) among fixed kernels
  • planning as inference finds agent kernels / policy that deal with this uncertainty

Two kinds of uncertainty

• Agent uncertainty:
  • In perception-action loop:
    • construct agent's memory transition kernels that consistently update probabilistic beliefs about their environment
    • these beliefs come with uncertainty
    • can turn uncertainty reduction itself into a goal!

Two kinds of uncertainty

• Agent uncertainty:
  • E.g: Define goal event via information gain :
    \[G=1 \Leftrightarrow D_{KL}[PH_2(x)||PH_0(x)] > d\]
  • PAI solves for policy that employs agent memory to gain information / reduce its uncertainty by \(d\) bits

Two kinds of uncertainty

•  Relevance for project:
  • taking decisions based on agent's knowledge is part of the project

Progress

• Successfully extended framework by features necessary for tackling goals of our project.
• These are discussed next:
  • Expected reward maximization
  • Parametrized kernels
  • Shared kernels
  • Multi-agent setup and games
  • Uncertain (PO)MDP
  • Related project funded
  • Two uncertainties: designer and agent uncertainty
• Relevance for project will be highlighted

Expected reward maximization

• Relevance for project:
  • reward based problems more common than goal event problems ((PO)MDPs, RL, losses...)
  • extends applicability of framework

Parametrized kernels

• Often we don't want to choose the agent kernels completely freely e.g.:
  • choose parametrized agent kernels
    \[p(x_a|x_{\text{pa}(a)}) \to p(x_a|x_{\text{pa}(a)},\theta_a)\]
• What do we have to adapt in this case?
  • Step 3 of EM algorithm has to be adapted

Parametrized kernels

• Algorithm for parametrized kernels (not only conditioning and marginalizing anymore):
  1. Initial parameters \(\theta(0)\) define prior, \(P_0 \in M_A\).
  2. Get  \(Q_t\), from \(P_t\) by conditioning on the goal:  \[q_t(x) = p_t(x|G=1).\]
  3. Get \(P_{t+1}\), by replacing parameter \(\theta_a\) of each agent kernel with result of:
      

Parametrized kernels

• Relevance for project:
  • needed for shared kernels
  • needed for continuous random variables
  • neural networks are parametrized kernels
  • scalability

Shared kernels

• We also often want to impose the condition that multiple agent kernels are identical
• E.g. the three agent kernels in this MDP:

Shared kernels

• Algorithm then becomes
  1. Initial parameters \(\theta(0)\) define prior, \(P_0 \in M_A\).
  2. Get  \(Q_t\), from \(P_t\) by conditioning on the goal:  \[q_t(x) = p_t(x|G=1).\]
  3. Get \(P_{t+1}\), by replacing parameter \(\theta_c\) of all agent kernels of type \(c\) with result of:
      

Proposal

• Exploit planning as inference setup to answer questions about:

  • Multiple, possibly competing goals

  • Coordination and communication from an information theoretic perspective

  • Dynamic scalability of multi-agent systems

  • Dynamically changing goals that depend on knowledge acquired through observations

     

Shared kernels

• Relevance for project:
  • scalability (less parameters to optimize)
  • make it possible to have
    • multiple agents with same policy
    • constant policy over time

Multi-agent setup

Example multi agent setups:

Two agents interacting with same environment

Two agents with same goal

Two agents with different goals

Multi-agent setup

• Note:
  • In cooperative setting:
    • can often combine multiple goals to single common goal via event intersection, union, complement (supplied by \(\sigma\)-algebra)
    • single goal manifold
    • in principle can use single agent PAI as before

Multi-agents and games

• Relevance to project:
  • dealing with multi-agent and multiple, possibly competing goals is a main goal of the project
  • basis for studying communication and interaction
  • basis for scaling up number of agents
  • basis for understanding advantages of multi-agent setups

Preliminary work

1. Implementation of PAI in state of the art software (e.g. using Pyro)
2. Planning to learn / uncertain MDP, bandit example.
3. Do some agents have no interpretation e.g. the "Absent minded driver"? Collaboration with Simon McGregor.
4. Design independence: some problems can be solved even if kernels are chosen independently others require coordinated choice of kernels, some are in between.
5. Bayesian networks cannot change their structure dependent on the states of the contained random variables.

Preliminary work

1. Implementation of PAI in state of the art software (e.g. using Pyro)

• Proofs of concept coded up in Pyro
  • uses stochastic variational inference (SVI) for PAI instead of geometric EM
  • may be useful to connect to work with neural networks since based on PyTorch
• For simple cases and visualizations also have Mathematica code

Preliminary work

2. Planning to learn / uncertain MDP, bandit example.

• currently investigating PAI for one armed bandit
• goal event is \(G=S_3=1\)
• actions choose one of two bandits that have different win probabilities determined by \(\phi\)
• agent kernels can use memory \(C_t\) to learn about \(\phi\)

Preliminary work

3. Do some agents have no interpretation e.g. the "Absent minded driver"? Collaboration with Simon McGregor.

• Driver has to take third exit
• all agent kernels share parameter \(\theta = \)probability of exiting
• optimal is \(\theta= 1/3\)
• Is this an agent even though it may have no consistent intepretation?

Preliminary work

4. Design independence:
   • some problems can be solved even if kernels are chosen independently
   • others require coordinated choice of kernels,
   • some are in between.

• Two player penny game
• goal is to get different outcomes
• one has to play heads with high probability the other has to play tails
• can't choose two kernels independently

Preliminary work

5. Bayesian networks cannot change their structure dependent on the states of the contained random variables.

• Once we fix the (causal) Bayesian network it stays like that ...

if x=1

Preliminary work

5. Bayesian networks cannot change their structure dependent on the states of the contained random variables.

• Once we fix the (causal) Bayesian network it stays like that ...

if x=1

But for adding and removing agents probably needed

Preliminary work

5. Bayesian networks cannot change their structure dependent on the states of the contained random variables.

• Once we fix the (causal) Bayesian network it stays like that ...
• We are learning about modern ways to deal with such changes dynamically -- polynomial functors.

Thank you for your attention!

Uncertain MDP / RL

• In RL and RL for POMDPs the transition kernels of the environment are considered unknown / uncertain

Two kinds of uncertainty

• Saw before that we can derive policies that deal with uncertainty
• This uncertainty can be seen as the "designer's uncertainty"
• But we can also design agents that have models and come with their own well defined uncertainty
• For those we can turn uncertainty reduction itself into a goal!

Two kinds of uncertainty

• Agent uncertainty:
  • e.g. for agent memory implement stochastic world model that updates in response to sensor values
    • then by construction each internal state has a well defined associated belief distribution \(ph_t=f(c_t)\) over hidden variables
    •  
  • turn uncertainty reduction itself into a goal!