Modulated NODEs NeurIPS 2023

Modulated NODEs

Key Idea:

does setting apart dynamic states from

underlying static factors of variation improve existing model performance?

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

\mathbf{z}_0

\mathbf{z}_0

Latent Space

Encoding

Network

Decoding

Network

Forward

Simulation

(ODE)

\mathbf{d}

\mathbf{d}

Dynamics Modulator

Modulator Prediction

Network

\mathbf{s}

\mathbf{s}

Modulator Prediction

Network

Static Modulator

Latent NODE

(Chen et al., 2018)

ours

MoNODE (2023)

Parameters of an ODE

Style of a digit

influence dynamics

influence reconstruction

Modulated NODEs

\mathbf{d} \sim p(\mathbf{d}) ~\quad \textit{// dynamics modulator} \\

\mathbf{d} \sim p(\mathbf{d}) ~\quad \textit{// dynamics modulator} \\

\mathbf{s} \sim p(\mathbf{s}) ~~\quad \textit{// static modulator} \\

\mathbf{s} \sim p(\mathbf{s}) ~~\quad \textit{// static modulator} \\

\mathbf{z}_0 \sim p(\mathbf{z}_0) \quad \textit{// latent ODE state} \\

\mathbf{z}_0 \sim p(\mathbf{z}_0) \quad \textit{// latent ODE state} \\

Generative Model

\mathbf{x}_i \sim p_{\phi}(\mathbf{x}_i \mid \mathbf{z}_i ~;~ \mathbf{s}).

\mathbf{x}_i \sim p_{\phi}(\mathbf{x}_i \mid \mathbf{z}_i ~;~ \mathbf{s}).

\mathbf{z}_i = \mathbf{z}_0 + \int_{t_0}^{t_i} \mathbf{f}_{\mathbf{\theta}}(\mathbf{z}(\tau);\mathbf{d})~d \tau \\

\mathbf{z}_i = \mathbf{z}_0 + \int_{t_0}^{t_i} \mathbf{f}_{\mathbf{\theta}}(\mathbf{z}(\tau);\mathbf{d})~d \tau \\

implicit, point-estimates

ELBO (Chen, 2018)

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

Modulated NODEs

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

A general framework

Latent NODE

(Chen et al., 2018)

Second Order NODE

(Norcliffe et al., 2020)

Latent Second Order NODE

(Yildiz et al., 2019)

Heavy Ball NODE

(Xia et al., 2021)

can be applied to most x-NODE

Modulated NODEs

A general framework

that improves forecasting and generalization

Sinusoidal Data

Latent NODE

(Chen et al., 2018)

ours Mo-xNODE

Heavy Ball NODE

(Xia et al., 2021)

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

Modulated NODEs

A general framework

that improves forecasting and generalization

Sinusoidal Data

Latent NODE

(Chen et al., 2018)

ours Mo-xNODE

Heavy Ball NODE

(Xia et al., 2021)

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

Modulated NODEs

A general framework

that is easier to train

Sinusoidal Data

ours: Mo-xNODE

NODE (Chen et al., 2018)

SONODE (Norcliffe et al., 2020)

HBNODE (Xia et al., 2021)

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

Modulated NODEs

A general framework

that disentangles underlying factors

Table 2. R^2 scores to predict the unknown Factors of Variation from inferred latents. Higher is better.

d

d

~

\frac{dx}{dt} = \alpha x - \beta xy

\frac{dx}{dt} = \alpha x - \beta xy

\frac{dy}{dt} = \delta xy - \gamma y

\frac{dy}{dt} = \delta xy - \gamma y

dynamics

modulator

PP parameters

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

Modulated NODEs

that improves performance

on real world data

Table 3. Test MSE and standard deviation. Lower is better.

MoNODE

NODE

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

Thank you for your attention

I.A. Auzina, Ç. Yıldız, S. Magliacane, M. Bethge and E. Gavves

Modulated NODEs

All experiments and models publicly available at:

https://github.com/IlzeAmandaA/MoNODE

Modulated

Neural ODEs