Anticipating changes

Anticipating changes:

Decision making with temporal expectations

Dimitrije Marković

Bernstein Conference 2020

Satellite Workshop: "Dynamic probabilistic inference in the brain"

Dynamic probabilistic inference

How is uncertainty represented and updated?
How is approximate inference implemented?
How is spatio-temporal structure of our natural environment represented?

Introduce a computational model that represents temporal structure of a dynamic environment.

Infer learned temporal structure from human behaviour.

Anticipating changes

Recent empirical evidence of neuronal circuitry supporting anticipatory behaviour:

A Vilà-Balló, et al. Journal of Neuroscience (2017).
VD Costa, et al. Journal of Neuroscience (2015).

Accurate temporal representation \(\rightarrow\) anticipating events.

Anticipating changes

Marković, et al. PLoS computational biology (2019).

Outline

Probabilistic reversal learning task
Behavioural model
Model-based data analysis
Learning the hidden temporal structure
Conclusion

Probabilistic reversal learning

Probabilistic reversal learning task
Behavioural model
Model-based data analysis
Learning the hidden temporal structure
Conclusion

Representing duration statistics

two hidden states

\( s_t \in \{A, B\}\)

Hidden semi-Markov model

Transition probability

\[ p(s_{t+1}|s_t, f_t) = \left\{ \begin{array}{ll} I_2, & \text{ for } f_t < n+1 \\ J_2 - I_2, & \text{ for } f_t = n + 1 \end{array} \right. \]

Duration probability

\[ p(f_{t+1}|f_t) \rightarrow p(d) \]

Discrete phase-type distribution

Phase transitions

\[p(f_t|f_{t-1})\]

\pi_i = {n \choose i - 1} (1 - \delta)^{n - i -1}\delta^{i-1}, \:\: \pi_0 = 1 - \sum_{i=1}^n \pi_i

M Varmazyar, et al., Journal of Industrial Engineering International (2019).

Discrete phase-type distribution

Phase transitions

\[p(f_t|f_{t-1})\]

Duration distribution

\[p(d) = {d + n - 2 \choose d-1}(1-\delta)^{d-1}\delta^n\]

\begin{aligned} E[d]&=\frac{n (1-\delta)}{\delta}+1 \\ &= \mu + 1 \end{aligned}

Var[d] = \mu + \frac{\mu^2}{n}

\pi_i = {n \choose i - 1} (1 - \delta)^{n - i -1}\delta^{i-1}, \:\: \pi_0 = 1 - \sum_{i=1}^n \pi_i

M Varmazyar, et al., Journal of Industrial Engineering International (2019).

Behavioural model

K Friston, et al., Neural computation (2017).

history of past outcomes and choices \( H_{t-1} = (o_{t-1:1}, a_{t-1:1}) \)

belief updating (Bayes rule)
\[ p\left(s_{t}, f_{t}| H_{t} \right) = \frac{p\left(o_{t}| s_{t}, a_{t}\right)p\left(s_{t}, f_{t}| H_{t-1} \right)}{p\left(o_{t}| a_{t}, H_{t-1} \right)} \]

p(a_{t} = a|H_{t-1}) = \frac{e^{-\beta \left(V_{t}^a + \alpha IG^a_{t} \right)}}{\sum_u e^{-\beta \left( V_{t}^{u} + \alpha IG^u_{t} \right) }}

b_t

o_t

a_t

s'_t

b_{t+1}

o_{t+1}

a_{t+1}

s'_{t+1}

b_{t-1}

o_{t-1}

a_{t-1}

s'_{t-1}

Generative process

Action selection

b_{t+1} \equiv \sum_{s_t, f_t} p(s_{t+1}, f_{t+1}|s_t, f_t) p(s_t, f_t|H_t)

Probabilistic reversal learning task
Behavioural model
Model-based data analysis
Learning the hidden temporal structure
Conclusion

Behavioural data

50 healthy volunteers (20-30 years old):
- 27 subjects in the condition with regular reversals
- 23 subjects in the condition with irregular reversal
40 trials long training with a single reversal