Anticipating changes

Anticipating changes:

Decision making with temporal expectations

Dimitrije Marković

Bernstein Conference 2020

Satellite Workshop: "Dynamic probabilistic inference in the brain"

Dynamic probabilistic inference

How is uncertainty represented and updated?
How is approximate inference implemented?
How is spatio-temporal structure of our natural environment represented?

Introduce a computational model that represents temporal structure of a dynamic environment.

Infer learned temporal structure from human behaviour.

Anticipating changes

Recent empirical evidence of neuronal circuitry supporting anticipatory behaviour:

A Vilà-Balló, et al. Journal of Neuroscience (2017).
VD Costa, et al. Journal of Neuroscience (2015).

Accurate temporal representation \(\rightarrow\) anticipating events.

Anticipating changes

Marković, et al. PLoS computational biology (2019).

Temporal decision-making

Interval timing\(^1\)
Temporal attention\(^2\)
Delay discounting\(^3\)

Temporal expectations and their impact on behaviour:

M Jazayeri, and MN Shadlen, Nature neuroscience (2010).
AC Nobre, and F Van Ede, Nature Reviews Neuroscience (2018).
JT McGuire, and JW Kable, Cognition (2012).

Outline

Probabilistic reversal learning task
Behavioural model
Model-based data analysis
Learning the hidden temporal structure
Conclusion

Probabilistic reversal learning

Probabilistic reversal learning task
Behavioural model
Model-based data analysis
Learning the hidden temporal structure
Conclusion

Representing duration statistics

two hidden states

\( s_t \in \{A, B\}\)

Hidden semi-Markov model

Transition probability

\[ p(s_{t+1}|s_t, f_t) = \left\{ \begin{array}{ll} I_2, & \text{ for } f_t < n+1 \\ J_2 - I_2, & \text{ for } f_t = n + 1 \end{array} \right. \]

Duration probability

\[ p(f_{t+1}|f_t) \rightarrow p(d) \]

Discrete phase-type distribution

Phase transitions

\[p(f_t|f_{t-1})\]

\pi_i = {n \choose i - 1} (1 - \delta)^{n - i -1}\delta^{i-1}, \:\: \pi_0 = 1 - \sum_{i=1}^n \pi_i

M Varmazyar, et al., Journal of Industrial Engineering International (2019).

Discrete phase-type distribution

Phase transitions

\[p(f_t|f_{t-1})\]

Duration distribution

\[p(d) = {d + n - 2 \choose d-1}(1-\delta)^{d-1}\delta^n\]

\begin{aligned} E[d]&=\frac{n (1-\delta)}{\delta}+1 \\ &= \mu + 1 \end{aligned}

Var[d] = \mu + \frac{\mu^2}{n}

\pi_i = {n \choose i - 1} (1 - \delta)^{n - i -1}\delta^{i-1}, \:\: \pi_0 = 1 - \sum_{i=1}^n \pi_i

M Varmazyar, et al., Journal of Industrial Engineering International (2019).

Negative binomial prior

\[p(d) = NB(\mu, n)\]

\[\delta_\tau = p(s_\tau = B| s_0=A)\]

Behavioural model

K Friston, et al., Neural computation (2017).

history of past outcomes and choices \( H_{t-1} = (o_{t-1:1}, a_{t-1:1}) \)

belief updating (Bayes rule)
\[ p\left(s_{t}, f_{t}| H_{t} \right) = \frac{p\left(o_{t}| s_{t}, a_{t}\right)p\left(s_{t}, f_{t}| H_{t-1} \right)}{p\left(o_{t}| a_{t}, H_{t-1} \right)} \]

p(a_{t} = a|H_{t-1}) = \frac{e^{-\beta \left(V_{t}^a + \alpha IG^a_{t} \right)}}{\sum_u e^{-\beta \left( V_{t}^{u} + \alpha IG^u_{t} \right) }}

b_t

o_t

a_t

s'_t

b_{t+1}

o_{t+1}

a_{t+1}

s'_{t+1}

b_{t-1}

o_{t-1}

a_{t-1}

s'_{t-1}

Generative process

Action selection

b_{t+1} \equiv \sum_{s_t, f_t} p(s_{t+1}, f_{t+1}|s_t, f_t) p(s_t, f_t|H_t)

Parameter inference and sampling

A Gelman, et al., Statistica sinica (1996).

Observed participant's responses

\( A_T = (a^*_1, \ldots, a^*_T) \)

Posterior predictive sampling

\[\vec{\theta}_i, n_i \sim p(\vec{\theta}, n| A_T)\]

\[ \tilde{a}^i_t \sim p(a_t|H_{t-1}, \vec{\theta}_i, n_i) \]

p(\vec{\theta}, n| A_T) \propto p(\vec{\theta}) p(n) \prod_{t=1}^Tp(a^*_{t}|H_{t-1}, \vec{\theta}, n)

Posterior estimate over model paramters

Action selection

p(a_{t} = a|H_{t-1}) = \frac{e^{-\beta \left(V_{t}^a + \alpha IG^a_{t} \right)}}{\sum_u e^{-\beta \left( V_{t}^{u} + \alpha IG^u_{t} \right) }}

expected choice value

V_{t}^a = \begin{cases} 2 p(o_t|a, H_{t-1}) - 1 &\text{if } a \in \{L, R\} \\ 2\lambda - 1 &\text{if } a = M \end{cases}

expected information gain

IG_{t}^{a} = \sum_{o_{t}} p\left(o_{t}| a_{t}=a\right) D_{KL}\left[ p(s_{t}, f_{t}|o_{t}, a, H_{t-1}) || p(s_{t}, f_{t}|H_{t-1})\right]

Friston, Karl, et al. Neural computation (2017).

Probabilistic reversal learning task
Behavioural model
Model-based data analysis
Learning the hidden temporal structure
Conclusion

Behavioural data

50 healthy volunteers (20-30 years old):
- 27 subjects in the condition with regular reversals
- 23 subjects in the condition with irregular reversal
40 trials long training with a single reversal