DataDistance

Contributions

1. We ask: Is it possible to bypass the difficult problem of statistical uncertainty quantification and directly use distance-to-data?

2. Amortization of this computation using diffusion score-matching (we make this computation batch-trainable)

3. Propose a class of algorithms that reinterpret and incorporate existing algorithms into a more coherent framework for MBRL (by framing them as KL-regularized planning between occupation measure and perturbed data distribution)

Review of Method

\begin{aligned} \min_{x_t,u_t} \quad & \sum_t c(x_t,u_t) - \beta \sum_t \log p (x_t, u_t) \\ \text{s.t.} \quad & x_{t+1} = f(x_t, u_t) \quad \forall t \end{aligned}

Single Shooting

\begin{aligned} \min_{x_t,u_t} \quad & \sum_t c(x_t,u_t) - \beta \sum_t \log p_\theta (x_t, u_t) \\ \text{s.t.} \quad & x_{t+1} = f_\alpha(x_t, u_t) \quad \forall t \\ \alpha^* = & \text{argmin}_\alpha \mathbb{E}_{x,u,x'} \|f_\alpha(x,u) - x'\|^2 \end{aligned}

Collocation

\begin{aligned} \min_{x_t,u_t} \quad & \sum_t c(x_t,u_t) - \beta \sum_t \log p_\theta (x_t, u_t, x_{t+1}) \\ \end{aligned}

Review of Method

Previously....

Uncertainty Metric

Type of Gradient

Choice of Transcription

Ensembles

Data Distance

GPs

Dropouts

First

Zeroth

Shooting

Transcription

Why Ensembles vs. Data Distance?

1. Data distance is stable to data as an uncertainty metric

2. Data distance never underestimates uncertainty if scaled appropriately.

* Side question: Data distance? Data energy?

What's really wrong with ensembles?

MPPI with Ensemble Variance Penalty

Gradient-based with Ensemble Variance Penalty

No Data Region

What's really wrong with ensembles?

- Ensembles do okayish in situations for detecting out-of-support kind of situations

Observed

- But there are interpolative regimes that are still "out-of-distribution", but is within the support of data.

- If function values in this region changes drastically, this kills ensembles

- Especially when the underlying model class has inductive bias towards certain wrong solutions.

Why Data Distance?

- Data distance completely bypasses this pathology by working in the domain of data. So it doesn't care about inductive bias.

- But this is a double-edged sword, as it will unfairly penalize regions without data IF we have the right inductive bias (e.g. if function is periodic and we chose sinusodial basis)

Preliminary Results: Corridor Example

Without data distance penalty

With data distance penalty

Preliminary Results: Pendulum Swingup

Without data distance penalty

With data distance penalty

Data region

Too large of a regularization hurts task performance

Preliminary Results: Pendulum Swingup

From keypoints: able to preserve implicit constraints

Pendulum with keypoints lie on a circular manifold.

We're able to impose data-based constraints very successfully.

Keypoints to the Future example

Example of Data stability: We're able to stabilize back to the data manifold

Training Distribution

Optimization against Data Distance Uncertainty

Pixel-space Single Integrator

Recall carrots case where things disappeared and appeared out of nowhere.

This might have a chance of resolving some of those issues with learned dynamics.

D4RL Mujoco Benchmarks

With regularization

Without regularization

D4RL Mujoco Benchmarks

Strangely, collocation does not work as well for Mujoco benchmarks,

so we went with the single-shooting variant

(We think it's because it's not able to obey dynamic constraints very well)

D4RL Mujoco Benchmarks

31.3

12.3

15.6

46.6

60.6

36.2

61.6

109.2

103.0

Ours

Lessons from D4RL

"A Divergence Minimizing Perspective on Imitation Learning" - CoRL 2019

Lessons from D4RL

We can use this method to IMPROVE existing demonstrations in a situation where we have access to rewards.

- Escapes pathological examples for behavior cloning by doing KL on occupation measure, NOT action distribution

- Improve over demonstration data by incorporating the ability to optimize rewards.

Questions & Future Directions for CoRL

Single shooting vs. Collocation - doesn't seem necessary to argue, since we're really trying to unify things into a bigger framework?

For example, diffuser is solving collocation

(with a very confusing notation & motivation)

\begin{aligned} \min_{x_t,u_t} \quad & \sum_t c(x_t,u_t) - \beta \sum_t \log p_\theta (x_t, u_t, x_{t+1}) \\ \end{aligned}

But there is no explicit interpretation that this is direct collocation with data distance penalization

Would it be okay to swap single shooting & collocation where it seems convenient, and say we're not selling an exact algorithm, but an entire framework?

(That way we can say we're not competing with diffusion planning, but say that their success already shows the effectiveness of our proposed framework)

Questions & Future Directions for CoRL

What would be an interesting hardware experiment?

1. Replicating Keypoints to the Future with Mocap Markers

2. Carrots for Pixel-based Planning

Russ Meeting