Active Inference with Semi-markov models
Dimitrije Marković
Theoretical Neurobiology Meeting
01.03.2020
Active inference and semi-markov decision processes
-
(Part I) Active inference in multi-armed bandits
- Empirical comparison with UCB and Thompson sampling.
- https://slides.com/dimarkov/ai-mbs
-
(Part II) Active inference and semi-Markov processes
- Hidden semi-Markov models and a representation of state duration.
- Learning the hidden temporal structure of state transitions.
- Application: Reversal learning task.
-
(Part III) Active inference and semi-Markov decision processes
- Extending policies with action (policy) duration.
- Decision about when actions should be taken and for how long.
- Applications: Temporal attention, intertemporal choices.
Active inference and semi-markov decision processes
-
(Part I) Active inference in multi-armed bandits
- Empirical comparison with UCB and Thompson sampling.
- https://slides.com/dimarkov/ai-mbs
-
(Part II) Active inference and semi-Markov processes
- Hidden semi-Markov models and a representation of state duration.
- Learning the hidden temporal structure of state transitions.
- Application: Reversal learning task.
-
(Part III) Active inference and semi-Markov decision processes
- Extending policies with action (policy) duration.
- Decision about when actions should be taken and for how long.
- Applications: Temporal attention, intertemporal choices.
-
(Part II) Active inference and semi-Markov processes
- Hidden semi-Markov models and a representation of state duration.
- Learning the hidden temporal structure of state transitions.
- Application: Reversal learning task.
Semi-Markov processes
https://en.wikipedia.org/wiki/Markov_renewal_process#Relation_to_other_stochastic_processes
- State space S
- Jump times Tn and states Xn
- Inter-arrival time τn=Tn−Tn−1
- The sequence [(X0,T0),…,(Xn,Tn),…] is called a Markov renewal process if:
Pr(τn≤t,Xn=j∣(X0,T0),…,(Xn−1=i,Tn))=Pr(τn≤t,Xn=j∣Xn−1=i)
Pr\left(\tau_n \leq t, X_{n}=j| (X_0, T_0), \ldots, (X_{n-1}=i, T_n) \right) = Pr\left(\tau_n \leq t, X_{n}=j| X_{n-1}=i \right)
If Yt≡Xn for t∈[Tn,Tn+1) then the process Yt is called a semi-Markov process
Semi-Markov processes
State space S
X0
X_0
X1
X_1
X2
X_2
Xn−1
X_{n-1}
Xn
X_n
Xn+1
X_{n+1}
…
…
T0
T_0
T1
T_1
T2
T_2
Tn−1
T_{n-1}
Tn
T_{n}
Tn+1
T_{n+1}
Time
τ1
\tau_1
τ2
\tau_2
τn
\tau_n
τn+1
\tau_{n+1}
Special cases
For exponentially distributed iid waiting times we have a continuous time Markov chain
Pr(τn≤t,Xn=j∣Xn−1=i)=Pr(Xn=j∣Xn−1=i)(1−eλit)
Pr\left(\tau_n \leq t, X_{n}=j| X_{n-1}=i \right) = Pr\left(X_{n}=j| X_{n-1}=i \right) \left( 1 - e^{\lambda_i t} \right)
A discrete time Markov chain has geometrically distributed waiting times
Pr(Xn=i∣Xn−1=i)=pi
Pr\left(X_{n}=i| X_{n-1}=i \right) = p_i
τn∼p(τ∣pi)=piτ−1(1−pi),τ∈{1,2,3,…}
\tau_n \sim p(\tau|p_i) = p_i^{\tau - 1}(1 - p_i), \:\: \tau \in \{1, 2, 3, \ldots\}
Hidden Semi-Markov models
Shun-Cheng Yu, "Hidden semi-Markov Models: Theory, Algorithms and Applications", Elsevir 2016.
A graphical representation of HSMM
Latent variables
outcomes
ft
f_t
st
s_t
ot
o_t
ft−1
f_{t-1}
ft+1
f_{t+1}
st−1
s_{t-1}
st+1
s_{t+1}
ot−1
o_{t-1}
ot+1
o_{t+1}
Example
f∈{1,2,3}
τ=3
\tau = 3
time step
1
1
2
2
3
3
A
A
B
B
s∈{A,B}
τ=5
\tau = 5
τ=2
\tau = 2
p(ft∣ft−1,st−1)≡τ∼p(τ∣s)
p(f_t|f_{t-1}, s_{t-1}) \equiv \tau \sim p(\tau|s)
Example
f∈{1,2,3}
τ=3
\tau = 3
time step
1
1
2
2
3
3
A
A
B
B
s∈{A,B}
τ=5
\tau = 5
τ=2
\tau = 2
p(ft∣ft−1)≡τ∼p(τ)
p(f_t|f_{t-1}) \equiv \tau \sim p(\tau)
Phase transitions
p(ft∣ft−1)
M Varmazyar, et al., Journal of Industrial Engineering International (2019).
Discrete phase-type distribution
ωi=(i−1n)(1−δ)n−i+1δi−1,ω0=1−∑i=1nωi
\omega_i = {n \choose i-1} (1 - \delta)^{n - i + 1}\delta^{i-1}, \:\: \omega_0 = 1 - \sum_{i=1}^{n} \omega_i
1−δ
1-\delta
1−δ
1-\delta
1−δ
1-\delta
δ
\delta
δ
\delta
n
n
n+1
n+1
…
ω1
\omega_1
ω2
\omega_2
ωn
\omega_{n}
ω0
\omega_{0}
1
1
2
2
Phase transitions
p(ft∣ft−1)
M Varmazyar, et al., Journal of Industrial Engineering International (2019).
Discrete phase-type distribution
Duration distribution
Negative binomial
p(τ)=(τ−1τ+n−2)(1−δ)τ−1δn
E[τ]=δn(1−δ)+1=μ+1
E[\tau]=\frac{n (1-\delta)}{\delta}+1 = \mu + 1
Var[τ]=μ+nμ2
Var[\tau] = \mu + \frac{\mu^2}{n}
ωi=(i−1n)(1−δ)n−i+1δi−1,ω0=1−∑i=1nωi
\omega_i = {n \choose i-1} (1 - \delta)^{n - i + 1}\delta^{i-1}, \:\: \omega_0 = 1 - \sum_{i=1}^{n} \omega_i
1−δ
1-\delta
1−δ
1-\delta
1−δ
1-\delta
δ
\delta
δ
\delta
n
n
n+1
n+1
…
ω1
\omega_1
ω2
\omega_2
ωn
\omega_{n}
ω0
\omega_{0}
1
1
2
2
Phase transitions
p(ft∣ft−1)
State transitions
State transitions
p(st∣st−1,ft−1)
A
B
A
B
1−δ
1-\delta
1−δ
1-\delta
1−δ
1-\delta
δ
\delta
δ
\delta
1
1
n
n
n+1
n+1
2
2
…
ω1
\omega_1
ω2
\omega_2
ωn
\omega_{n}
ω0
\omega_{0}
negative binomial distribution
δd=p(st+d=B∣st=A)
\delta_d = p(s_{t+d}=B|s_t=A)
Active Inference
p(otl∣st1,…,stK)=Cat(Al)
p\left(o_t^l| s_t^1, \ldots, s_t^K\right) = Cat\left( A^l \right)
p(Al)=Dir(al)
p\left(A^l\right) = Dir(a^l)
p(stk∣st−1k,π,ft)=Cat(Bfk,π)
p\left(s_t^k| s_{t-1}^k, \pi, f_t \right) = Cat\left( B^{k, \pi}_f \right)
p(π)=smax(−γG),P(otl)=smax(Cl)
p(\pi) = s_{max}\left( -\gamma G \right), \quad P(o^l_t) = s_{max} \left( C^l \right)
p(ft∣ft−1,m)=Cat(Lm)
p\left(f_t| f_{t-1}, m \right) = Cat\left( L^{m} \right)
p(m)=Cat(M),m=(μ,n)
p(m) = Cat(M), \quad m = (\mu, n)
μ∈{μmin,…,μmax},n∈{1,…,nmax}
\mu \in \{ \mu_{min}, \ldots, \mu_{max} \}, \: n \in \{1, \ldots, n_{max}\}
m
m
st−11
s^1_{t-1}
st1
s^1_{t}
Al
A^l
ot
o_t
Bk
B^k
π
\pi
ft−1
f_{t-1}
ft
f_{t}
Belief updating and learning
History of past outcomes Ot=(o1,…,ot)
Q(m)∝p(ot∣m,Ot−1)p(m∣Ot−1)
Q(m) \propto p(o_t|m, O_{t-1}) p(m|O_{t-1})
Q(ft∣m)∝p(ot∣ft,Ot−1)p(ft∣m,Ot−1)
Q(f_t|m) \propto p(o_t|f_t, O_{t-1}) p(f_t|m, O_{t-1})
Q(st∣ft)∝p(ot∣st)e∑πQ(π)lnp(st∣ft,π,Ot−1)
Q(s_t|f_t) \propto p(o_t|s_t) e^{\sum_{\pi} Q(\pi) \ln p\left(s_t|f_t, \pi, O_{t-1} \right)}
Q(π)∝p(ot∣π,Ot−1)p(π∣Ot−1)
Q(\pi) \propto p(o_t|\pi, O_{t-1}) p(\pi|O_{t-1})
Q(A)∝e∑stQ(st)lnp(ot∣st,A)P(A∣Ot−1)
Q(A) \propto e^{\sum_{s_t} Q(s_t) \ln p(o_t|s_t, A)} P(A|O_{t-1})
p(ot∣x,Ot−1)=∑yp(ot∣y)p(y∣x,Ot−1)
p(o_t| x, O_{t-1}) = \sum_y p(o_t|y) p(y|x, O_{t-1})
p(xt∣Ot−1)=∑xt−1p(xt∣xt−1)Q(xt−1)
p(x_t|O_{t-1}) = \sum_{x_{t-1}} p(x_t|x_{t-1}) Q(x_{t-1})
Marginal likelihood
Predictive prior
Action selection
ut∼p(π∣Ot−1)∝e−γG(π)
u_t \sim p\left(\pi|O_{t-1}\right) \propto e^{-\gamma G(\pi)}
When simulating behaviour γ→∞
For data analysis γ is a free parameter
G(π)=EQ(ot∣π)[lnP(ot)Q(ot∣π)]−EQ(ot,st,A∣π)[lnp(ot∣st,A)]
G(\pi) = E_{Q(o_t|\pi)}\left[ \ln \frac{Q(o_t|\pi)}{P(o_t)} \right] - E_{Q(o_t, s_t, A|\pi)} \left[ \ln p(o_t|s_t, A) \right]
Probabilistic reversal learning
Probabilistic reversal learning
Probabilistic reversal learning
Model parameters
f∈{1,…,nmax}
⊗
⊗
loss
gain
cue A
cue B
P(ot1)=[31,31,31]
P(ot2)=[ρ1,ρ2,2ρ,2ρ]
Performance
1−MA(t)MA(t)
\frac{MA(t)}{1 - MA(t)}
Trials Until correct (TUC)
In SILICO
Is the experimental setup useful?
- Can the temporal structure be learned?
- Will different temporal beliefs reflect different behaviour?
- How well can we differentiate between agents with different temporal beliefs?
In SILICO
Process:
- Fix priors, and action precision γ
- Simulate behaviour in both conditions with different nmax
- Illustrate, performance, TUC, and the learning of the latent temporal structure (μ,n)
- Model inversion, and confusion matrix for nmax
In SILICO - Behavioural metrics
In SILICO - Learning latent temporal structure
nmax=10
n_{max} = 10
In SILICO - Confusion matrix
In SILICO - Confusion matrix
Behavioural data
- 50 healthy volunteers (20-30 years old):
- 27 subjects in the condition with regular reversals
- 23 subjects in the condition with irregular reversals
- 40 trials long training with a single reversal
Performance-tuc trajectories
Model comparison
Labeled trajectories
conclusion
- Modelling and assessing influence of temporal-expectations on decision-making in dynamic environments.
- How people learn temporal expectations could also be addressed with this approach, but some challenges remain.
- Linking the underlying representation of the temporal structure to behaviour provides a novel method for computational cognitive phenotyping.
Thanks to:
- Andrea Reiter
- Stefan Kiebel
- Thomas Parr
- Karl Friston
https://slides.com/dimarkov/active-inference-semi-markov
https://github.com/dimarkov/pybefit
https://journals.plos.org/ploscompbiol/article?rev=2&id=10.1371/journal.pcbi.1006707
Active Inference with Semi-markov models Dimitrije Marković Theoretical Neurobiology Meeting 01.03.2020