cvanelteren
Computational scientist
What node is most important?
Casper van Elteren
Dynamic importance of nodes is poorly predicted by static topological features
Complex systems are ubiquitous
- Structure
- Dynamics
- Emergent behavior
Most approaches are not applicable to complex systems:
- Use simplified dynamics
- Use structure as dynamic importance
- Use overwhelming interventions
What is the most important node?
> What node drives the system?
x˙i=M0(xi)+∑iNAijM1(xi)M2(xj)
\dot x_i = M_0(x_i) + \sum_i^N A_{ij} M_1(x_i) M_2(x_j)
- Stationarity of system dynamics
- Local linearity of state transitions
- Memory-less dynamics
- ....
Wang et al. (2016)
However we have a many-to-one mapping
1. Simplified dynamics
"Well-connected nodes are dynamically important"
2. Which feature to select?
N.B. implicit dynamics assumption!
Harush et al. (2017)
Fi∝input×outputw−1
F_i \propto input \times output^{w-1}
2. Dynamic importance interacts with structure
Genetic
Epidemic
Biochemical
Ecological
3. The size of intervention matters
- Interventions are crucial for the scientific method
- Many are overwhelming:
- Gene knockout
- Replacing signal
Pearl (2000)
Mechanism driving behavior are different under overwhelming interventions!
Summary
We have seen:
- Most methods simplify the complex system
- ..use structure for identifying 'important' parts
- ..use overwhelming interventions
Possible solution: information theory
Information theory and complex systems
Traditional approaches are domain specific but all ask similar questions, e.g.:
- What node is most important?
- Does it exhibit criticality?
- How robust is the system to removal of signals?
- ....
Quax et al. (2016)
How to achieve universal approach to study various complex behavior?
There is a need for a universal language that decouples syntax from semantics
Quax et al. (2016)
Domain specific
+
x˙i=M0(xi)+∑iNAijM1(xi)M2(xj)
\dot x_i = M_0(x_i) + \sum_i^N A_{ij} M_1(x_i) M_2(x_j)
S={s1,s2,…,sn}
S = \{s_1, s_2, \dots, s_n\}
Quantify in terms of "information"
I(si:S)
I(s_i : S)
Traditional approach
Information viewpoint
Up
Down
...
P(System)
Up
Down
...
P(Bird)
Shannon (1948)
Information Entropy: "Amount of uncertainty"
Mutual information: "Shared information"
H(X)=−∑x∈XP(x)logP(x)
H(X) = - \sum_{x \in X} P(x) \log P(x)
I(X:Y)=x∈Xy∈Y∑P(x,y)logP(x)P(y)P(x,y)=H(X)−H(X∣Y)=H(Y)−H(Y∣X)
\begin{aligned}
I(X : Y) &= \sum_{x \in X y \in Y} P(x, y) \log \frac{P(x,y)}{P(x)P(y)}\\
&= H(X) - H(X | Y)\\
&= H(Y) - H(Y | X)\\
\end{aligned}
N.B. No assumption on what generates P
H(X)=−21log21−21log21=1
H(X) = - \frac{1}{2} \log \frac{1}{2} - \frac{1}{2} \log \frac{1}{2} = 1
P(Heads)=21P(Tails)=21
\begin{aligned}
P(Heads) = \frac{1}{2} \\
P(Tails) = \frac{1}{2} \\
\end{aligned}
H(X)=0
H(X) = 0
P(Heads)=0P(Tails)=1
\begin{aligned}
P(Heads) = 0 \\
P(Tails) = 1 \\
\end{aligned}
Information in complex systems
Given ergodic system S
Information will always decrease as function of time
Driver-node will share the most information with the system over time
Diminishing role of hubs
Quax & Sloot (2013)
- Infinitely sized networks
- Locally tree-like
- No-self loops
Degree
Numerical
Analytical
d(si)={t:I(sit0+t:St0)=21H(si)}
d(s_i) = \{t : I(s_i^{t_0 + t} : S^{t_0}) = \frac{1}{2} H(s_i) \}
Goals
Answer:
- Can information theory tools be used on real-world systems?
- Does well-connectedness translate to dynamical importance in real-world systems?
- Does intervention size matter in real-world systems?
Prior results:
- Assume dynamics
- Dynamics interact with structure
- Overwhelming interventions
- Theoretical
Goal: identify driver-node in real-world systems
Application domain
- Mildly depressed patients
- Center for epidemiologic studies depression scale (CES-D)
- Changing lives for older couples (CLOC), N = 241
Fried et al. (2015)
Node dynamics
P(sit∣Lit−1)∝exp(−TE(x))
P(s_i ^ t | L_i ^ {t-1}) \propto \exp( -\frac{E(x)}{T})
Ising spin dynamics
si∈{−1,1}
s_i \in \{-1, 1\}
Glauber (1963)
Used to model variety of behavior
- Neural dynamics
- Voting behavior
- ....
Causal influence
Causal influence forms the ground truth
- Underwhelming :
- Overwhelming:
E=0.1
E = 0.1
E=∞
E = \infty
Causal impact
γi:=∑t=0∞∑jND(Pi(sj)′∣∣Pi(sj))Δt
\gamma_i := \sum_{t=0}^\infty \sum_{j}^N D(P_i(s_j)' || P_i(s_j)) \Delta t
Advantages of KullBack-Leibler divergence:
- Non Negative
- Zero iff P' == P
- No assumption on P
- Optimality in coding setting
- Embodies extra bits needed to code samples from P' given code P
Information impact
- Node with largest causal influence has highest information impact
- Observations only!
- No perturbations required
μi:=∑t=0∞I(sit0+t:St0)Δt
\mu_i := \sum_{t=0}^\infty I(s_i^{t_0 + t} : S^{t_0}) \Delta t
Structural metrics
| Name | What does it measure? |
|---|---|
| Betweenness | Shortest path |
| Degree | Local influence |
| Current flow | Least resistance |
| Eigenvector | Infinite walks |
Statistical procedure
| Ind. var max(x) | Dep. var |
|---|---|
| - Degree centrality - Betweenness centrality - Current flow centrality - Eigenvector centrality - Information impact |
Causal impact - Underwhelming - Overwhelming |
Classification with random forest:
Random forest classifier
RNF classifier with high prediction accuracy
- Underwhelming does not match overwhelming causal impact
- Information impact matches underwhelming causal impact
Information impact captures driver-node change
Information impact varies linearly with low causal impact
- No structural metric showed a linear relation with low causal impact
- Information impact was highly linear with low causal impact
Summary
- Information impact predict driver-node for unperturbed dynamics
- Intervention size matters
- Structural metrics don't identify driver-node well
Take-home message:
Structural connectedness != dynamic importance
Future direction
- Does information impact generalize well to other graphs?
- Does it generalize well to other dynamics?
- Can it be used to detect transient structures?
- Asymmetry in time effects
- ....
Acknowledgement
- A big thanks to dr. Rick Quax for his supervision
Models
Information
toolbox
Plotting toolbox
IO toolbox
- Fast
- Extendable
- User-friendly
Information toolbox
Reference
- Glauber, R. J. Time-dependent statistics of the Ising model. Journal of Mathematical Physics 4, 294–307 (1963).
- Quax, R., Apolloni, A. & Sloot, P. M. a. The di-
minishing role of hubs in dynamical processes
on complex networks. Journal of the Royal Soci-
ety, Interface / the Royal Society 10, 20130568. arXiv:
1111.5483 (2013). - Quax, R., Har-Shemesh, O., Thurner, S., & Sloot, P. (2016). Stripping syntax from complexity: An information-theoretical perspective on complex systems. arXiv preprint arXiv:1603.03552.
- Harush, U. & Barzel, B. Dynamic patterns of in-
formation flow in complex networks. Nature Com-
munications 8, 1–11 (2017). - Harush, U. & Barzel, B. Dynamic patterns of in-
formation flow in complex networks. Nature Com-
munications 8, 1–11 (2017). - Fried, E. I. et al. From loss to loneliness: The re-
lationship between bereavement and depressive
symptoms. Journal of Abnormal Psychology 124, 256–
265 (2015).
Information impact
Betweenness
Degree
Current flow
Eigenvector
Low causal impact
High causal impact
Shannon (1948)
A
B
P(A)
0
1
P(B | A = a)
0
1
I(A:A)
I(A : A)
I(A:B)
I(A : B)
Entropy
Mutual information
Numerical methods
- Monte-Carlo methods
- Compare information impact and centrality metrics
- Intervention:
- Underwhelming
- Overwhelming
P(St0)
P(S^{t_0})
P(sit0+t∣St0)
P(s_i^{t_0 + t} | S^{t_0})
E=0.1
E = 0.1
E=∞
E = \infty
Statistical procedure
- Quantify accuracy with random forest classifier
- Validation with Leave-One-Out cross-validation
DYn∈NXn∈N={(X1,Y1),…,(XN,YN)}={ym1,…,ymN}={xm1,…,kmN}
\begin{aligned}
D &= \{(X_1, Y_1), \dots, (X_N, Y_N)\}\\
Y_{n \in N} &= \{y_{m1}, \dots, y_{mN}\}\\
X_{n \in N} &= \{x_{m1}, \dots, k_{mN} \}\\
\end{aligned}
m = amount of regressors
N = number of samples
What node is most important? Casper van Elteren Dynamic importance of nodes is poorly predicted by static topological features arxiv.org/abs/1904.06654
IAS&TU_delft
By cvanelteren
IAS&TU_delft
My talk at the TU delft and IAS. Preprint can be found at https://arxiv.org/abs/1904.06654
- 1,474
