### cvanelteren

Computational scientist

Casper van Elteren

Dynamic importance of nodes is poorly predicted by static topological features

Complex systems are ubiquitous

- Structure
- Dynamics
- Emergent behavior

Most approaches are not applicable to complex systems:

- Use simplified dynamics
- Use structure as dynamic importance
- Use overwhelming interventions

What is the most important node?

> What node drives the system?

\dot x_i = M_0(x_i) + \sum_i^N A_{ij} M_1(x_i) M_2(x_j)

- Stationarity of system dynamics
- Local linearity of state transitions
- Memory-less dynamics
- ....

Wang et al. (2016)

However we have a many-to-one mapping

1. Simplified dynamics

"Well-connected nodes are dynamically important"

2. Which feature to select?

N.B. implicit dynamics assumption!

Harush et al. (2017)

F_i \propto input \times output^{w-1}

2. Dynamic importance interacts with structure

Genetic

Epidemic

Biochemical

Ecological

3. The size of intervention matters

- Interventions are crucial for the scientific method
- Many are overwhelming:
- Gene knockout
- Replacing signal

Pearl (2000)

Mechanism driving behavior are different under overwhelming interventions!

We have seen:

- Most methods simplify the complex system
- ..use structure for identifying 'important' parts
- ..use overwhelming interventions

Possible solution: **information theory**

Information theory and complex systems

Traditional approaches are domain specific but all ask similar questions, e.g.:

- What node is most important?
- Does it exhibit criticality?
- How robust is the system to removal of signals?
- ....

Quax et al. (2016)

How to achieve universal approach to study various complex behavior?

*There is a need for a universal language that decouples** syntax **from** semantics*

Quax et al. (2016)

Domain specific

+

\dot x_i = M_0(x_i) + \sum_i^N A_{ij} M_1(x_i) M_2(x_j)

S = \{s_1, s_2, \dots, s_n\}

Quantify in terms of "information"

I(s_i : S)

Traditional approach

Information viewpoint

Up

Down

...

P(System)

Up

Down

...

P(Bird)

Shannon (1948)

Information Entropy: "Amount of uncertainty"

Mutual information: "Shared information"

H(X) = - \sum_{x \in X} P(x) \log P(x)

\begin{aligned}
I(X : Y) &= \sum_{x \in X y \in Y} P(x, y) \log \frac{P(x,y)}{P(x)P(y)}\\
&= H(X) - H(X | Y)\\
&= H(Y) - H(Y | X)\\
\end{aligned}

N.B. No assumption on what generates P

H(X) = - \frac{1}{2} \log \frac{1}{2} - \frac{1}{2} \log \frac{1}{2} = 1

\begin{aligned}
P(Heads) = \frac{1}{2} \\
P(Tails) = \frac{1}{2} \\
\end{aligned}

H(X) = 0

\begin{aligned}
P(Heads) = 0 \\
P(Tails) = 1 \\
\end{aligned}

Information in complex systems

Given ergodic system S

Information will always decrease as function of time

Driver-node will share the most information with the system over time

Diminishing role of hubs

Quax & Sloot (2013)

- Infinitely sized networks
- Locally tree-like
- No-self loops

Degree

Numerical

Analytical

d(s_i) = \{t : I(s_i^{t_0 + t} : S^{t_0}) = \frac{1}{2} H(s_i) \}

Answer:

- Can information theory tools be used on real-world systems?
- Does well-connectedness translate to dynamical importance in real-world systems?
- Does intervention size matter in real-world systems?

Prior results:

- Assume dynamics
- Dynamics interact with structure
- Overwhelming interventions
- Theoretical

Goal: identify driver-node in real-world systems

- Mildly depressed patients
- Center for epidemiologic studies depression scale (CES-D)
- Changing lives for older couples (CLOC), N = 241

Fried et al. (2015)

Node dynamics

P(s_i ^ t | L_i ^ {t-1}) \propto \exp( -\frac{E(x)}{T})

Ising spin dynamics

s_i \in \{-1, 1\}

Glauber (1963)

Used to model variety of behavior

- Neural dynamics
- Voting behavior
- ....

Causal influence forms the ground truth

- Underwhelming :
- Overwhelming:

E = 0.1

E = \infty

\gamma_i := \sum_{t=0}^\infty \sum_{j}^N D(P_i(s_j)' || P_i(s_j)) \Delta t

Advantages of KullBack-Leibler divergence:

- Non Negative
- Zero iff P' == P
- No assumption on P
- Optimality in coding setting
- Embodies extra bits needed to code samples from P' given code P

- Node with largest causal influence has highest information impact
- Observations only!
- No perturbations required

\mu_i := \sum_{t=0}^\infty I(s_i^{t_0 + t} : S^{t_0}) \Delta t

Name | What does it measure? |
---|---|

Betweenness | Shortest path |

Degree | Local influence |

Current flow | Least resistance |

Eigenvector | Infinite walks |

Statistical procedure

Ind. var max(x) | Dep. var |
---|---|

- Degree centrality - Betweenness centrality - Current flow centrality - Eigenvector centrality - Information impact |
Causal impact - Underwhelming - Overwhelming |

Classification with random forest:

RNF classifier with high prediction accuracy

- Underwhelming does not match overwhelming causal impact
- Information impact matches underwhelming causal impact

Information impact captures driver-node change

Information impact varies linearly with low causal impact

- No structural metric showed a linear relation with low causal impact
- Information impact was highly linear with low causal impact

Summary

- Information impact predict driver-node for unperturbed dynamics
- Intervention size matters
- Structural metrics don't identify driver-node well

Take-home message:

**Structural connectedness != dynamic importance**

Future direction

- Does information impact generalize well to other graphs?
- Does it generalize well to other dynamics?
- Can it be used to detect transient structures?
- Asymmetry in time effects
- ....

Acknowledgement

- A big thanks to dr. Rick Quax for his supervision

Models

Information

toolbox

Plotting toolbox

IO toolbox

- Fast

- Extendable

- User-friendly

Information toolbox

Reference

- Glauber, R. J. Time-dependent statistics of the Ising model. Journal of Mathematical Physics 4, 294–307 (1963).
- Quax, R., Apolloni, A. & Sloot, P. M. a. The di-

minishing role of hubs in dynamical processes

on complex networks. Journal of the Royal Soci-

ety, Interface / the Royal Society 10, 20130568. arXiv:

1111.5483 (2013). -
Quax, R., Har-Shemesh, O., Thurner, S., & Sloot, P. (2016). Stripping syntax from complexity: An information-theoretical perspective on complex systems.
*arXiv preprint arXiv:1603.03552*. - Harush, U. & Barzel, B. Dynamic patterns of in-

formation flow in complex networks. Nature Com-

munications 8, 1–11 (2017). - Harush, U. & Barzel, B. Dynamic patterns of in-

formation flow in complex networks. Nature Com-

munications 8, 1–11 (2017). - Fried, E. I. et al. From loss to loneliness: The re-

lationship between bereavement and depressive

symptoms. Journal of Abnormal Psychology 124, 256–

265 (2015).

Information impact

Betweenness

Degree

Current flow

Eigenvector

Low causal impact

High causal impact

Shannon (1948)

A

B

P(A)

0

1

P(B | A = a)

0

1

I(A : A)

I(A : B)

Entropy

Mutual information

- Monte-Carlo methods
- Compare information impact and centrality metrics
- Intervention:
- Underwhelming
- Overwhelming

P(S^{t_0})

P(s_i^{t_0 + t} | S^{t_0})

E = 0.1

E = \infty

Statistical procedure

- Quantify accuracy with random forest classifier
- Validation with Leave-One-Out cross-validation

\begin{aligned}
D &= \{(X_1, Y_1), \dots, (X_N, Y_N)\}\\
Y_{n \in N} &= \{y_{m1}, \dots, y_{mN}\}\\
X_{n \in N} &= \{x_{m1}, \dots, k_{mN} \}\\
\end{aligned}

m = amount of regressors

N = number of samples

By cvanelteren

My talk at the TU delft and IAS. Preprint can be found at https://arxiv.org/abs/1904.06654

- 1,204