Anticipating changes:
Decision making with temporal expectations
Dimitrije Marković
Bernstein Conference 2020
Satellite Workshop: "Dynamic probabilistic inference in the brain"
Dynamic probabilistic inference
- How is uncertainty represented and updated?
- How is approximate inference implemented?
- How is spatio-temporal structure of our natural environment represented?
Introduce a computational model that represents temporal structure of a dynamic environment.
Infer learned temporal structure from human behaviour.
Anticipating changes
Recent empirical evidence of neuronal circuitry supporting anticipatory behaviour:
- A Vilà-Balló, et al. Journal of Neuroscience (2017).
- VD Costa, et al. Journal of Neuroscience (2015).
Accurate temporal representation \(\rightarrow\) anticipating events.
Anticipating changes
Marković, et al. PLoS computational biology (2019).
Temporal decision-making
- Interval timing\(^1\)
- Temporal attention\(^2\)
- Delay discounting\(^3\)
Temporal expectations and their impact on behaviour:
- M Jazayeri, and MN Shadlen, Nature neuroscience (2010).
- AC Nobre, and F Van Ede, Nature Reviews Neuroscience (2018).
- JT McGuire, and JW Kable, Cognition (2012).
Outline
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Probabilistic reversal learning
Probabilistic reversal learning
Probabilistic reversal learning
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Representing duration statistics
two hidden states
\( s_t \in \{A, B\}\)
Hidden semi-Markov model
Transition probability
\[ p(s_{t+1}|s_t, f_t) = \left\{ \begin{array}{ll} I_2, & \text{ for } f_t < n+1 \\ J_2 - I_2, & \text{ for } f_t = n + 1 \end{array} \right. \]
Duration probability
\[ p(f_{t+1}|f_t) \rightarrow p(d) \]
Discrete phase-type distribution
Phase transitions
\[p(f_t|f_{t-1})\]
\pi_i = {n \choose i - 1} (1 - \delta)^{n - i -1}\delta^{i-1}, \:\: \pi_0 = 1 - \sum_{i=1}^n \pi_i
M Varmazyar, et al., Journal of Industrial Engineering International (2019).
Discrete phase-type distribution
Phase transitions
\[p(f_t|f_{t-1})\]
Duration distribution
\[p(d) = {d + n - 2 \choose d-1}(1-\delta)^{d-1}\delta^n\]
\begin{aligned}
E[d]&=\frac{n (1-\delta)}{\delta}+1 \\
&= \mu + 1
\end{aligned}
Var[d] = \mu + \frac{\mu^2}{n}
\pi_i = {n \choose i - 1} (1 - \delta)^{n - i -1}\delta^{i-1}, \:\: \pi_0 = 1 - \sum_{i=1}^n \pi_i
M Varmazyar, et al., Journal of Industrial Engineering International (2019).
Negative binomial prior
\[p(d) = NB(\mu, n)\]
\[\delta_\tau = p(s_\tau = B| s_0=A)\]
Behavioural model
K Friston, et al., Neural computation (2017).
history of past outcomes and choices \( H_{t-1} = (o_{t-1:1}, a_{t-1:1}) \)
belief updating (Bayes rule)
\[ p\left(s_{t}, f_{t}| H_{t} \right) = \frac{p\left(o_{t}| s_{t}, a_{t}\right)p\left(s_{t}, f_{t}| H_{t-1} \right)}{p\left(o_{t}| a_{t}, H_{t-1} \right)} \]
p(a_{t} = a|H_{t-1}) = \frac{e^{-\beta \left(V_{t}^a + \alpha IG^a_{t} \right)}}{\sum_u e^{-\beta \left( V_{t}^{u} + \alpha IG^u_{t} \right) }}
b_t
o_t
a_t
s'_t
b_{t+1}
o_{t+1}
a_{t+1}
s'_{t+1}
b_{t-1}
o_{t-1}
a_{t-1}
s'_{t-1}
Generative process
Action selection
b_{t+1} \equiv \sum_{s_t, f_t} p(s_{t+1}, f_{t+1}|s_t, f_t) p(s_t, f_t|H_t)
Parameter inference and sampling
A Gelman, et al., Statistica sinica (1996).
Observed participant's responses
\( A_T = (a^*_1, \ldots, a^*_T) \)
Posterior predictive sampling
\[\vec{\theta}_i, n_i \sim p(\vec{\theta}, n| A_T)\]
\[ \tilde{a}^i_t \sim p(a_t|H_{t-1}, \vec{\theta}_i, n_i) \]
p(\vec{\theta}, n| A_T) \propto p(\vec{\theta}) p(n) \prod_{t=1}^Tp(a^*_{t}|H_{t-1}, \vec{\theta}, n)
Posterior estimate over model paramters
Action selection
p(a_{t} = a|H_{t-1}) = \frac{e^{-\beta \left(V_{t}^a + \alpha IG^a_{t} \right)}}{\sum_u e^{-\beta \left( V_{t}^{u} + \alpha IG^u_{t} \right) }}
expected choice value
V_{t}^a = \begin{cases}
2 p(o_t|a, H_{t-1}) - 1 &\text{if } a \in \{L, R\} \\
2\lambda - 1 &\text{if } a = M
\end{cases}
expected information gain
IG_{t}^{a} = \sum_{o_{t}} p\left(o_{t}| a_{t}=a\right) D_{KL}\left[ p(s_{t}, f_{t}|o_{t}, a, H_{t-1}) || p(s_{t}, f_{t}|H_{t-1})\right]
Friston, Karl, et al. Neural computation (2017).
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Behavioural data
- 50 healthy volunteers (20-30 years old):
- 27 subjects in the condition with regular reversals
- 23 subjects in the condition with irregular reversal
- 40 trials long training with a single reversal
learning phase
model fitting
model testing
Model selection
Inferred duration distribution
Trials until correct (TUC)
Perfomance
Group level trajectories
Posterior samples vs data
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Learning temporal structure
Condition with regular reversals
Condition with irregular reversals
duration [d]
duration [d]
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Conclusion
- Modelling and assessing influence of temporal-expectations on decision-making in dynamic environments.
- How people learn temporal expectations could also be addressed with this approach, but some challenges remain.
- Linking the underlying representation of the temporal structure to behaviour provides a novel method for computational cognitive phenotyping.
Thanks to:
- Stefan Kiebel
- Andrea Reiter
- Thomas Parr
- Karl Friston
- Sebastian Bitzer
Anticipating changes
By dimarkov
