Deep Implicits
with Differential Rendering
Daniel Yukimura
scene
parameters
2D image
- camera pose
- geometry
- materials
- lighting
- ...
rendering
differentiable
Feedback
(Learning)
Differentiable Volumetric Rendering
Can we infer implicit 3D representations without 3D supervision?
Differentiable Volumetric Rendering
single-view 3D reconstruction
Architecture
fθ:R3×Z→[0,1]
f_\theta: \mathbb{R}^3 \times \mathcal{Z} \rightarrow [0,1]
tθ:R3×Z→R3
t_\theta: \mathbb{R}^3 \times \mathcal{Z} \rightarrow \mathbb{R}^3
Forward Pass - Rendering:
p^=“first intersection with {p∈R3∣fθ(p)=τ}"
\hat{p} = \text{``first intersection with } \{p\in \mathbb{R}^3 | f_\theta(p) = \tau \} \text{"}
Backward Pass - Gradients:
L(I^,I)=u∑∥I^u−Iu∥
\mathcal{L}(\hat{I}, I) = \sum\limits_u \|\hat{I}_u - I_u \|
Loss function
Differentiable Rendering
∂θ∂L=u∑∂I^u∂L∂θ∂I^u
\frac{\partial \mathcal{L}}{\partial\theta} = \sum\limits_u \frac{\partial \mathcal{L}}{\partial \hat{I}_u} \frac{\partial \hat{I}_u}{\partial \theta} \\
=u∑∂I^u∂L(dθdtθ(p^)+∂p^∂tθ(p^)∂θ∂p^)
= \sum\limits_u \frac{\partial \mathcal{L}}{\partial \hat{I}_u} \left( \frac{d t_\theta (\hat{p})}{d \theta} + \frac{\partial t_\theta (\hat{p})}{\partial \hat p} \frac{\partial \hat{p}}{\partial\theta}\right)
KNOWN
??
Depth Gradients:
r(d)=r0+dw
r(d) = r_0 + d w
∃d^ s.t. p^=r(d^)
\exists\hspace{1mm} \hat{d} \text{ s.t. } \hspace{2mm} \hat{p} = r(\hat{d})
Single-View Reconstruction:
- Multi-View Supervision
- Single-View Supervision
Experiments:
Multi-View Reconstruction:
Experiments:
Implicit Differentiable Rendering
Multiview 3D Surface Reconstruction
Input: Collection of 2D images (masked)
with rough or noisy camera info.
Targets:
- Geometry
- Appearance (BRDF, lighting conditions)
- Cameras
Method:
Geometry:
Sθ={x∈R3∣f(x;θ)=0}
\mathcal{S}_\theta = \{ x\in \mathbb{R}^3 | f(x;\theta) = 0 \}
signed distance function (SDF) +
implicit geometric regularization (IGR)
θ
\theta
- geometry parameters
IDR - Forward pass
Rp(τ)={cp+tvp∣t≥0}
R_p(\tau) = \{ c_p + t v_p | t \geq 0 \}
x^p=x^p(θ,τ)=Rp(τ)∩Sθ
\hat{x}_p = \hat{x}_p(\theta, \tau) = R_p(\tau) \cap \mathcal{S}_\theta
τ
\tau
- camera parameters
Ray cast:
(first intersection)
IDR - Forward pass
Lp(θ,γ,τ)=M(x^p,n^p,z^p,vp;γ)
L_p(\theta, \gamma, \tau) = M(\hat{x}_p, \hat{n}_p, \hat{z}_p, v_p; \gamma)
γ
\gamma
- appearance parameters
Output (Light Field):
Surface normal
n^p(θ)
\hat{n}_p(\theta)
Global gometry feature vector
z^p(x^p;θ)
\hat{z}_p(\hat{x}_p; \theta)
Differentiable intersections
θ0,τ0−current parameters
\theta_0, \tau_0 - \text{current parameters}
x^(θ,τ)=c+t0v−∇xf(x0;θ0)⋅v0vf(c+t0v;θ)
\hat{x}(\theta, \tau) = c + t_0 v - \frac{v}{\nabla_x f(x_0; \theta_0) \cdot v_0} f(c + t_0 v; \theta)
Lemma:
Experiments:
Multi-View Reconstruction:
Experiments:
Disentangling Geometry and Appearance:
Deep Implicits with Differential Rendering Daniel Yukimura