Active inference in multi-armed bandits

Active inference and semi-markov decision processes

(Part I) Active inference in multi-armed bandits
- Comparison with UCB and Thompson sampling.
- Comparison of state and outcome based preferences.

(Part II) Active inference and semi-Markov processes
- Semi-Markov models and a representation of state duration.
- Learning the hidden temporal structure of state transitions.
- Application: Reversal learning task.

(Part III) Active inference and semi-Markov decision processes
- Extending policies with action (policy) duration.
- Decision about when actions should be taken and for how long.
- Applications: Temporal attention, intertemporal choices.

Motivation

Multi-armed bandits generalise resource allocation problems
Often used in experimental cognitive neuroscience:
- Decision making in dynamic environments
- Spatio-temporal attentional processing
Wide range of industrial applications.
Good benchmark problem for comparing active inference based solutions with the state-of-the-art alternatives.

Multi-armed Bandits

Stationary bandits
Dynamic bandits
Adversarial bandits
Risk-aware bandits
Contextual bandits
non-Markovian bandits

Stationary (classical) bandits

On any trial an agents makes a choice between K arms.
Choice outcomes are binary variables $\rightarrow$ Bernoulli bandits
Reward probabilities are fixed with $p_{max}=\frac{1}{2} + \epsilon$ and $p_{\neg max} = \frac{1}{2}$ .
Task difficulty:
- The best arm advantage -> $\epsilon$
- The number of arms -> $K$

Non-stationary (Dynamic) bandits

On any trial an agents makes a choice between K arms.
Choice outcomes are binary variables $\rightarrow$ Bernoulli bandits.
Reward probabilities associated with each arm change over time:
- changes happen at the same time on all arms (e.g. switching bandits)
- changes happen independently on each arm (e.g. restless bandits)
The additional task difficulty:
- The rate of change/change probability

Choice optimality as regret minimization

At trial $t$ an agent chooses arm $a_t$ .
We define the regret as

$R(t) = p_{max}(t) - p_{a_t}(t)$

The cumulative regret after $T$ trials is obtained as

$\hat{R}(T) = \sum_{t=1}^T R(t)$

Regret rate is defined as

$r(T) = \frac{1}{T} \hat{R}(T)$

Bayesian Inference

Bernoulli bandits $\rightarrow$ Bernoulli likelihoods.
Beliefs about reward probabilities as Beta distributions.
Bayesian belief updating for all action selection algorithms.

Generative model

p(o_t|\theta^k_t) = \left(\theta^k_t\right)^{o_{t}}\left( 1- \theta^k_t \right)^{1-o_{t}}

p(o_t|\theta^k_t) = \left(\theta^k_t\right)^{o_{t}}\left( 1- \theta^k_t \right)^{1-o_{t}}

A K-armed bandit.
The $k$ th arm is associated with a reward probability $\theta^k_t$ at trial t.
Choice outcomes denoted with $o_t$ .

p(\theta^k_t|\theta^{k}_{t-1}, j_t) = \left\{ \begin{array}{cc} \mathcal{Be}(\alpha_0, \beta_0) & \textrm{for } j_{t-1} = 1 \\ \delta(\theta^k_{t} - \theta^k_{t-1}) & \textrm{for } j_{t-1} = 0 \end{array} \right.

p(\theta^k_t|\theta^{k}_{t-1}, j_t) = \left\{ \begin{array}{cc} \mathcal{Be}(\alpha_0, \beta_0) & \textrm{for } j_{t-1} = 1 \\ \delta(\theta^k_{t} - \theta^k_{t-1}) & \textrm{for } j_{t-1} = 0 \end{array} \right.

p(j_{t}=j|j_{t-1}, \rho) = \left\{ \begin{array}{cc} \delta_{j, 0} & \textrm{for } j_{t-1} = 1 \\ \rho^{j}(1-\rho)^{1-j} & \textrm{for } j_{t-1} = 0 \end{array} \right.

p(j_{t}=j|j_{t-1}, \rho) = \left\{ \begin{array}{cc} \delta_{j, 0} & \textrm{for } j_{t-1} = 1 \\ \rho^{j}(1-\rho)^{1-j} & \textrm{for } j_{t-1} = 0 \end{array} \right.

Belief updating

Given some choice $a_t$ on trial $t$ and $\vec{\theta}_t = (\theta_1, \ldots, \theta_K)$ belief updating corresponds to

p(\vec{\theta}_t, j_t|o_{t:1}, a_{t:1}) \propto p(o_t|\vec{\theta}_t, a_t) p(\vec{\theta}_t|j_t, o_{t-1:1}) p(j_t|o_{t-1:1})

p(\vec{\theta}_t, j_t|o_{t:1}, a_{t:1}) \propto p(o_t|\vec{\theta}_t, a_t) p(\vec{\theta}_t|j_t, o_{t-1:1}) p(j_t|o_{t-1:1})

Q(\theta_k) = \mathcal{Be}(\alpha^k_t, \beta^k_t), \quad Q(j_t=1) = \omega_t

Q(\theta_k) = \mathcal{Be}(\alpha^k_t, \beta^k_t), \quad Q(j_t=1) = \omega_t

p(\vec{\theta}_t, j_t|o_{t:1}, a_{t:1}) \approx \prod_{k=1}^K Q(\theta_k) Q(j_t)

p(\vec{\theta}_t, j_t|o_{t:1}, a_{t:1}) \approx \prod_{k=1}^K Q(\theta_k) Q(j_t)

Belief updating

Given some choice $a_t$ on trial $t$ and $\vec{\theta}_t = (\theta_1, \ldots, \theta_K)$ belief updating corresponds to

\omega_t = \frac{\frac{1}{2}\rho(1 - \omega_{t-1})}{ \frac{1}{2}\rho(1 - \omega_{t-1}) + (\mu_t^{a_t})^{o_t}(1 - \mu_t^{a_t})^{1-o_t}(1 - \rho(1-\omega_{t-1}))}

\omega_t = \frac{\frac{1}{2}\rho(1 - \omega_{t-1})}{ \frac{1}{2}\rho(1 - \omega_{t-1}) + (\mu_t^{a_t})^{o_t}(1 - \mu_t^{a_t})^{1-o_t}(1 - \rho(1-\omega_{t-1}))}

\alpha^k_t = (1 - \omega_t) \alpha^k_{t-1} + \omega_t \alpha_0 + \delta_{a_t, k} o_t

\alpha^k_t = (1 - \omega_t) \alpha^k_{t-1} + \omega_t \alpha_0 + \delta_{a_t, k} o_t

\beta^k_t = (1 - \omega_t) \beta^k_{t-1} + \omega_t \beta_0 + \delta_{a_t, k} (1 - o_t)

\beta^k_t = (1 - \omega_t) \beta^k_{t-1} + \omega_t \beta_0 + \delta_{a_t, k} (1 - o_t)

Belief updating

Given some choice $a_t$ on trial $t$ and $\vec{\theta}_t = (\theta_1, \ldots, \theta_K)$ belief updating corresponds to

\omega_t = \frac{\frac{1}{2}\rho(1 - \omega_{t-1})}{ \frac{1}{2}\rho(1 - \omega_{t-1}) + (\mu_t^{a_t})^{o_t}(1 - \mu_t^{a_t})^{1-o_t}(1 - \rho(1-\omega_{t-1}))}

\omega_t = \frac{\frac{1}{2}\rho(1 - \omega_{t-1})}{ \frac{1}{2}\rho(1 - \omega_{t-1}) + (\mu_t^{a_t})^{o_t}(1 - \mu_t^{a_t})^{1-o_t}(1 - \rho(1-\omega_{t-1}))}

\alpha^k_t = (1 - \omega_t) \alpha^k_{t-1} + \omega_t \alpha_0 + \delta_{a_t, k} o_t

\alpha^k_t = (1 - \omega_t) \alpha^k_{t-1} + \omega_t \alpha_0 + \delta_{a_t, k} o_t

\beta^k_t = (1 - \omega_t) \beta^k_{t-1} + \omega_t \beta_0 + \delta_{a_t, k} (1 - o_t)

\beta^k_t = (1 - \omega_t) \beta^k_{t-1} + \omega_t \beta_0 + \delta_{a_t, k} (1 - o_t)

Stationary case $\rightarrow$ $\rho = 0$

\omega_t = 0

\omega_t = 0

\alpha^k_t = \alpha^k_{t-1} + \delta_{a_t, k} o_t

\alpha^k_t = \alpha^k_{t-1} + \delta_{a_t, k} o_t

\beta^k_t = \beta^k_{t-1} + \delta_{a_t, k} (1 - o_t)

\beta^k_t = \beta^k_{t-1} + \delta_{a_t, k} (1 - o_t)

Action Selection Algorithms

Optimistic Thompson sampling (O-TS)
Bayesian upper confidence bound (B-UCB)
Active inference

Thompson sampling

Raj, Vishnu, and Sheetal Kalyani. "Taming non-stationary bandits: A Bayesian approach."

arXiv preprint arXiv:1707.09727 (2017).

Classical algorithm for Bayesian bandits

$a_t = \arg\max_k \theta^*_k, \qquad \theta^*_k \sim p\left(\theta^k_t|o_{t-1:1}, a_{t-1:1}\right)$

Optimistic Thompson sampling (O-TS) defined as

$a_t = \arg\max_k \left[ \max(\theta^*_k, \langle \theta^k_t \rangle) \right]$

Upper Confidence bound (UCB)

One of the oldest algorithms defined as

a_t = \left\{ \begin{array}{cc} \arg\max_k \left(\mu^k_{t-1} + \sqrt{\frac{2 \ln t}{n^{k}_{t-1}}}\right) & \textrm{for } t>K \\ t & \textrm{otherwise} \end{array} \right.

a_t = \left\{ \begin{array}{cc} \arg\max_k \left(\mu^k_{t-1} + \sqrt{\frac{2 \ln t}{n^{k}_{t-1}}}\right) & \textrm{for } t>K \\ t & \textrm{otherwise} \end{array} \right.

$\mu^{k}_t$ denotes the expected reward probability of the $k$ th arm.
$n^k_{t}$ denotes the number of times $k$ th arm was selected.

Bayesian Upper Confidence bound (B-UCB)

a_t = \arg\max_k CDF^{-1}\left( 1 - \frac{1}{t}, \bar{\alpha}_{t,k}, \bar{\beta}_{t,k} \right)

a_t = \arg\max_k CDF^{-1}\left( 1 - \frac{1}{t}, \bar{\alpha}_{t,k}, \bar{\beta}_{t,k} \right)

Kaufmann, Emilie, Olivier Cappé, and Aurélien Garivier. "On Bayesian upper confidence bounds for bandit problems." Artificial intelligence and statistics, 2012.

\int_{0}^x \mathcal{Be} \left(\theta; \bar{\alpha}_t^k, \bar{\beta}_t^k \right) \textrm{d}\theta = 1 - \frac{1}{t}

\int_{0}^x \mathcal{Be} \left(\theta; \bar{\alpha}_t^k, \bar{\beta}_t^k \right) \textrm{d}\theta = 1 - \frac{1}{t}

Inverse regularised incomplete beta function

\bar{\alpha}_t^k = \left(1 - \rho(1-\omega_{t-1})\right) \alpha_{t-1}^k + \frac{1}{2}\rho(1-\omega_{t-1})

\bar{\alpha}_t^k = \left(1 - \rho(1-\omega_{t-1})\right) \alpha_{t-1}^k + \frac{1}{2}\rho(1-\omega_{t-1})

\bar{\beta}_t^k = \left(1 - \rho(1-\omega_{t-1})\right) \beta_{t-1}^k + \frac{1}{2}\rho(1-\omega_{t-1})

\bar{\beta}_t^k = \left(1 - \rho(1-\omega_{t-1})\right) \beta_{t-1}^k + \frac{1}{2}\rho(1-\omega_{t-1})

Active inference

expected free energy

S(a_t) = D_{KL}\left(Q(o_t |a_t)||P(o_t)\right) + E_{Q(\vec{\theta}|a_t)}\left[H[o_t|\vec{\theta}, a_t] \right]

S(a_t) = D_{KL}\left(Q(o_t |a_t)||P(o_t)\right) + E_{Q(\vec{\theta}|a_t)}\left[H[o_t|\vec{\theta}, a_t] \right]

rolling behavioural policies $\rightarrow$ independent from the past choices.
a behavioural policy corresponds to a single choice $\rightarrow$ $\pi = a_t, \quad a_t \in \{1,\ldots,K\}$ .

G(a_t) = D_{KL}\left(Q(\vec{\theta}|a_t)||P(\vec{\theta}|a_t)\right) + E_{Q(\vec{\theta}|a_t)}\left[H[o_t|\vec{\theta}, a_t] \right]

G(a_t) = D_{KL}\left(Q(\vec{\theta}|a_t)||P(\vec{\theta}|a_t)\right) + E_{Q(\vec{\theta}|a_t)}\left[H[o_t|\vec{\theta}, a_t] \right]

expected surprisal

Parr, Thomas, and Karl J. Friston. "Generalised free energy and active inference." Biological cybernetics 113.5-6 (2019): 495-513.

Active inference

expected free energy

S(a_t) = D_{KL}\left(Q(o_t |a_t)||P(o_t)\right) + E_{Q(\vec{\theta}|a_t)}\left[H[o_t|\vec{\theta}, a_t] \right]

S(a_t) = D_{KL}\left(Q(o_t |a_t)||P(o_t)\right) + E_{Q(\vec{\theta}|a_t)}\left[H[o_t|\vec{\theta}, a_t] \right]

rolling behavioural policies $\rightarrow$ independent from the past choices.
a behavioural policy corresponds to a single choice $\rightarrow$ $\pi = a_t, \quad a_t \in \{1,\ldots,K\}$ .

G(a_t) = D_{KL}\left(Q(\vec{\theta}|a_t)||P(\vec{\theta}|a_t)\right) + E_{Q(\vec{\theta}|a_t)}\left[H[o_t|\vec{\theta}, a_t] \right]

G(a_t) = D_{KL}\left(Q(\vec{\theta}|a_t)||P(\vec{\theta}|a_t)\right) + E_{Q(\vec{\theta}|a_t)}\left[H[o_t|\vec{\theta}, a_t] \right]

expected surprisal

$G(a_t) \geq S(a_t)$

$\arg\min_a G(a) \neq \arg\min_a S(a)$

$P(\vec{\theta}_t| a_t=k) \propto \left( \theta_t^k\right)^{\left( e^{2\lambda}-1 \right)}$

$P(o_t) \propto e^{o_t \lambda} e^{-(1-o_t) \lambda}$

$P(o_t) = \int d \vec{\theta} p(o_t|\vec{\theta}_t, a_t) P(\vec{\theta}_t|a_t)$

Parr, Thomas, and Karl J. Friston. "Generalised free energy and active inference." Biological cybernetics 113.5-6 (2019): 495-513.

Active inference - choice selection

Posterior over policies

$Q(\pi) \propto e^{\gamma G(\pi) + F(\pi)}$

For rolling policies $F(\pi) = const.$

$Q(\pi)\propto e^{\gamma G(\pi)}$

Choice selection based on expected free energy (F-AI)

$a_t \sim p(a_t) \propto e^{-\gamma G(a_t, \lambda)}$

Choice selection based on expected surprisal (S-AI)

$a_t \sim p(a_t) \propto e^{-\gamma S(a_t, \lambda)}$

approximate Expected surprisal (A-Ai)

Da Costa, Lancelot, et al. "Active inference on discrete state-spaces: a synthesis."

arXiv preprint arXiv:2001.07203 (2020).

S(a_t) \approx - \left(1-\rho(1-\omega_{t-1})\right)\cdot\left[\lambda (2\mu_{t-1}^{a_t} - 1) + \frac{1}{2\nu_{t-1}^{a_t}} \right]

S(a_t) \approx - \left(1-\rho(1-\omega_{t-1})\right)\cdot\left[\lambda (2\mu_{t-1}^{a_t} - 1) + \frac{1}{2\nu_{t-1}^{a_t}} \right]

Next step:

compare F-AI, S-AI, and A-AI in stationary bandits

Between AI comparison

Parametrised regret rate for A-AI algorithm

comparison:

Optimistic Thompson sampling (O-TS)
Bayesian upper confidence bound (B-UCB)
Approximate active inference (A-Ai)

comparison in Stationary bandits

A-AI:

$\lambda = 0.8$
$\gamma = 20$

regret rate over time

A-AI:

$\lambda = 0.8$
$\gamma = 20$

$\epsilon = 0.25$

$\epsilon = 0.10$

$\epsilon = 0.40$

Switching bandits

Choice outcomes are binary variables $\rightarrow$ Bernoulli bandits
One arm is associate with maximal reward probability $p_{max}=\frac{1}{2} + \epsilon$
All other arms are fixed to $p_{\neg max} = \frac{1}{2}$ .
The optimal arm changes with probability $\rho$ .
Task difficulty:
- Advantage of the best arm $\rightarrow$ $\epsilon$
- Number of arms $\rightarrow$ $K$
- Change probability $\rightarrow$ $\rho$

comparison in switching bandits

$\epsilon=0.25$

A-AI:

$\lambda = 0.8$
$\gamma = 20$

conclusion

When comparing active inference on stationary bandits, we get mixed results.

When comparing active inference on stationary bandits, we get mixed results.

In non-stationary bandits, active inference based agents show lower performance if changes occur often enough.

A tentative TODO list:
- Determine the optimality relation for AI algorithms, e.g. $\gamma^* = f(\lambda, K, \epsilon, \rho)$ .
- Introduce learning of $\lambda$ .
- Would adaptive $\lambda$ lead to values that minimize regret rate?

Thanks to:

Hrvoje Stojić
Sarah Schwöbel
Stefan Kiebel
Thomas Parr
Karl Friston
Ryan Smith
Lancelot de Costa

https://slides.com/revealme/

https://github.com/dimarkov/aibandits

Active Inference in multi-armed bandits