Deep Implicits

with Differential Rendering

Daniel Yukimura

scene

parameters

2D image

camera pose
geometry
materials
lighting
...

rendering

differentiable

Feedback

(Learning)

Differentiable Volumetric Rendering

Niemeyer, Mescheder, Oechsle, Geiger: Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision

Can we infer implicit 3D representations without 3D supervision?

Differentiable Volumetric Rendering

Niemeyer, Mescheder, Oechsle, Geiger: Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision

single-view 3D reconstruction

Architecture

f_\theta: \mathbb{R}^3 \times \mathcal{Z} \rightarrow [0,1]

t_\theta: \mathbb{R}^3 \times \mathcal{Z} \rightarrow \mathbb{R}^3

Forward Pass - Rendering:

\hat{p} = \text{``first intersection with } \{p\in \mathbb{R}^3 | f_\theta(p) = \tau \} \text{"}

Backward Pass - Gradients:

\mathcal{L}(\hat{I}, I) = \sum\limits_u \|\hat{I}_u - I_u \|

Loss function

Differentiable Rendering

\frac{\partial \mathcal{L}}{\partial\theta} = \sum\limits_u \frac{\partial \mathcal{L}}{\partial \hat{I}_u} \frac{\partial \hat{I}_u}{\partial \theta} \\

= \sum\limits_u \frac{\partial \mathcal{L}}{\partial \hat{I}_u} \left( \frac{d t_\theta (\hat{p})}{d \theta} + \frac{\partial t_\theta (\hat{p})}{\partial \hat p} \frac{\partial \hat{p}}{\partial\theta}\right)

KNOWN

Depth Gradients:

r(d) = r_0 + d w

\exists\hspace{1mm} \hat{d} \text{ s.t. } \hspace{2mm} \hat{p} = r(\hat{d})