Anticipating changes:
Decision making with temporal expectations
Dimitrije Marković
Bernstein Conference 2020
Satellite Workshop: "Dynamic probabilistic inference in the brain"
Dynamic probabilistic inference
- How is uncertainty represented and updated?
- How is approximate inference implemented?
- How is spatio-temporal structure of our natural environment represented?
Introduce a computational model that represents temporal structure of a dynamic environment.
Infer learned temporal structure from human behaviour.
Anticipating changes
Recent empirical evidence of neuronal circuitry supporting anticipatory behaviour:
- A Vilà-Balló, et al. Journal of Neuroscience (2017).
- VD Costa, et al. Journal of Neuroscience (2015).
Accurate temporal representation \(\rightarrow\) anticipating events.
Anticipating changes
Marković, et al. PLoS computational biology (2019).
Outline
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Probabilistic reversal learning
Probabilistic reversal learning
Probabilistic reversal learning
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Representing duration statistics
two hidden states
\( s_t \in \{A, B\}\)
Hidden semi-Markov model
Transition probability
\[ p(s_{t+1}|s_t, f_t) = \left\{ \begin{array}{ll} I_2, & \text{ for } f_t < n+1 \\ J_2 - I_2, & \text{ for } f_t = n + 1 \end{array} \right. \]
Duration probability
\[ p(f_{t+1}|f_t) \rightarrow p(d) \]
Discrete phase-type distribution
Phase transitions
\[p(f_t|f_{t-1})\]
\pi_i = {n \choose i - 1} (1 - \delta)^{n - i -1}\delta^{i-1}, \:\: \pi_0 = 1 - \sum_{i=1}^n \pi_i
M Varmazyar, et al., Journal of Industrial Engineering International (2019).
Discrete phase-type distribution
Phase transitions
\[p(f_t|f_{t-1})\]
Duration distribution
\[p(d) = {d + n - 2 \choose d-1}(1-\delta)^{d-1}\delta^n\]
\begin{aligned}
E[d]&=\frac{n (1-\delta)}{\delta}+1 \\
&= \mu + 1
\end{aligned}
Var[d] = \mu + \frac{\mu^2}{n}
\pi_i = {n \choose i - 1} (1 - \delta)^{n - i -1}\delta^{i-1}, \:\: \pi_0 = 1 - \sum_{i=1}^n \pi_i
M Varmazyar, et al., Journal of Industrial Engineering International (2019).
Behavioural model
K Friston, et al., Neural computation (2017).
history of past outcomes and choices \( H_{t-1} = (o_{t-1:1}, a_{t-1:1}) \)
belief updating (Bayes rule)
\[ p\left(s_{t}, f_{t}| H_{t} \right) = \frac{p\left(o_{t}| s_{t}, a_{t}\right)p\left(s_{t}, f_{t}| H_{t-1} \right)}{p\left(o_{t}| a_{t}, H_{t-1} \right)} \]
p(a_{t} = a|H_{t-1}) = \frac{e^{-\beta \left(V_{t}^a + \alpha IG^a_{t} \right)}}{\sum_u e^{-\beta \left( V_{t}^{u} + \alpha IG^u_{t} \right) }}
b_t
o_t
a_t
s'_t
b_{t+1}
o_{t+1}
a_{t+1}
s'_{t+1}
b_{t-1}
o_{t-1}
a_{t-1}
s'_{t-1}
Generative process
Action selection
b_{t+1} \equiv \sum_{s_t, f_t} p(s_{t+1}, f_{t+1}|s_t, f_t) p(s_t, f_t|H_t)
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Behavioural data
- 50 healthy volunteers (20-30 years old):
- 27 subjects in the condition with regular reversals
- 23 subjects in the condition with irregular reversal
- 40 trials long training with a single reversal
learning phase
model fitting
model testing
Model selection
Inferred duration distribution
Trials until correct (TUC)
Perfomance
Group level trajectories
Posterior samples vs data
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Learning temporal structure
Condition with regular reversals
Condition with irregular reversals
duration [d]
duration [d]
- Probabilistic reversal learning task
- Behavioural model
- Model-based data analysis
- Learning the hidden temporal structure
- Conclusion
Conclusion
- Modelling and assessing influence of temporal-expectations on decision-making in dynamic environments.
- How people learn temporal expectations could also be addressed with this approach, but some challenges remain.
- Linking the underlying representation of the temporal structure to behaviour provides a novel method for computational cognitive phenotyping.
Thanks to:
- Stefan Kiebel
- Andrea Reiter
- Thomas Parr
- Karl Friston
- Sebastian Bitzer
Anticipating changes
By dimarkov
