3DOR-2024

M

A

R

F

Towards Multi-View Consistency
in Neural Ray Fields
using Parametric Medial Surfaces

P

Parametric

Peder Bergebakken Sundt

Theoharis Theoharis

M

A

R

F

P

Parametric

3DOR-2024

Peder Bergebakken Sundt

Theoharis Theoharis

Towards Multi-View Consistency
in Neural Ray Fields
using Parametric Medial Surfaces

M

A

R

F

P

Medial

Atom

Ray

Field

Parametric

3DOR-2024

Peder Bergebakken Sundt

Theoharis Theoharis

Towards Multi-View Consistency
in Neural Ray Fields
using Parametric Medial Surfaces

A novel shape representation

Motivation

Fields

  • Continuous
  • Differentiable
  • Different fields/signals are amenable to different novel applications
    • Signed distance \(\to\) quick to traverse, watertight
    • Generalized winding number  \(\to\) self intersections, cloth
    • Learned descriptor fields  \(\to\) robotics
    • Language embeddings \(\to\) semantic queries
\mathcal X\to\mathcal Y
\mathbb R ^3\to[0,1]

Why make novel Neural Fields?

  • Effective in Inverse Rendering and other problems in Visual Computing
    like analysis-by-synthesis

    • High compression ability

    • Able to learn mappings
      previously considered ill-posed
      (lottery ticket hypothesis)

  • New mappings enable new applications

Why ray fields?

Neural fields are great at solving inverse rendering, but are slow to render

Directly predict

ray integral

Why ray fields?

Neural fields are great at solving inverse rendering, but are slow to render

Offline Tabulation

Precompute samples,

discretization artifacts (aliasing)

Hybrid methods

(Many small networks)

Cheaper samples, trade compute for memory, loss of global shape prior

Gaussian splatting

100x, View-dependent, no notion of surface or prior, hard to extract geometry

Learn heuristic field

(SDF)

10x reduction in samples, no transparent objects

Directly predict

ray integral

Neural fields are great at solving inverse rendering, but are slow to render

Why ray fields?

100x, View-
dependent, can represent both color and geometry, less studied

Offline Tabulation

Precompute samples,

discretization artifacts (aliasing)

Hybrid methods

(Many small networks)

Cheaper samples, trade compute for memory, loss of global shape prior

Gaussian splatting

100x, View-dependent, no notion of surface or prior, hard to extract geometry

Learn heuristic field

(SDF)

10x reduction in samples, no transparent objects

Cartesian Field functions (\(\mathbb R^3\to\mathcal X\))

require multiple sample along ray

While Ray Field functions (\(\mathcal R\to\mathcal X\))

map directly to the result

We build on the Medial Atom Ray Field (MARF)

Why build on MARF?

  • Deals with ray field discontinuities
    (Lipschitz bound)
  • Inherent multi-view consistency in reconstruction domain
  • Emit cheap surface normals

Why build on MARF?

\bigg\}
  • Deals with ray field discontinuities
    (Lipschitz bound)
  • Inherent multi-view consistency in reconstruction domain
  • Emit cheap surface normals

Background

"Medial axis"

Set of maximally inscribed spheres

"Medial axis"

The set of maximally inscribed spheres, or "atoms"

Ridges of the distance transform

"Medial axis"

Ridges of the distance transform

The set of maximally inscribed spheres, or atoms

Set of points with two nearest neighbors

"Medial axis"

Set of points with two nearest neighbors

Ridges of the distance transform

Set of points with two nearest neighbors

"Medial axis"

\ell
\ell
\ell
\{
\mathbf x_i, r_i
\}_{i=1}^k
\rightarrow
\ell
\mathbf x_1, r_1

MARF

\mathbf x_2, r_2
\mathbf x_4, r_4
\mathbf x_3, r_3
\ell
\{
\mathbf x_i, r_i
\}_{i=1}^k
\rightarrow
\ell
\mathbf x_1, r_1

MARF

\mathbf x_2, r_2
\mathbf x_4, r_4
\mathbf x_3, r_3
\mathbf x_2, r_2
\mathbf x_2, r_2
\ell
\mathbf x_i, r_i
\mapsto

?

\mathbf x_2, r_2
\ell
\mathbf x_i, r_i
\mapsto
\mathbb R^3\times \mathbb R^+
\rightarrow
S^2\times \mathbb R ^2

(4 DoF)

\rightarrow
\mathbb R^2

(4 DoF)

?

View direction + xy in view plane

Atom/sphere

MARF

P

\mathbf x_2, r_2
\ell
\mathbf x_i, r_i
\mapsto
\mathbb R^3\times \mathbb R^+
\rightarrow
S^2\times \mathbb R ^2

(4 DoF)

\rightarrow
\mathbb R^2

(4 DoF)

2 DoF!

Parametric surface

MARF

P

f_i:\begin{cases} \ell &\to \mathbb R ^3\times\mathbb R^+ \\ (\mathbf x, \hat{\mathbf q}) & \mapsto \end{cases}
(u_i, v_i)
(x_i, y_i, z_i, r_i)
g_i:\begin{cases} \mathbb R ^2 &\to \mathbb R ^3\times\mathbb R^+ \\ (u_i, v_i) & \mapsto \end{cases}

Zero Thickness by construction

More accurate medial surface  

\(\Rightarrow\) More stable atoms

f_i:\begin{cases} \ell &\to \mathbb R ^3\times\mathbb R^+ \\ (\mathbf x, \hat{\mathbf q}) & \mapsto \end{cases}
(u_i, v_i)
(x_i, y_i, z_i, r_i)

Architecture

MARF

P

Prior work

Ours

MARF

P

Prior work

Ours

Multiple MLP Heads

Block-diagonal matrices

Batched matrix multiplication

MARF

P

Prior work

Ours

Multiple MLP Heads

Block-diagonal matrices

Batched matrix multiplication

MARF

P

Prior work

Ours

Multiple MLP Heads

Block-diagonal matrices

Batched matrix multiplication

Fewer weights

Block diagonal

Fewer weights

Fully connected

Fewer weights

Block diagonal

Fully connected

Batched Matrix

Fewer weights

Block diagonal

Fully connected

Batched Matrix

We evaluate 4 different activations/nonlinearities

Evaluation

[1] Ba JL, Kiros JR, Hinton GE. Layer Normalization 2016.

[3] Ramasinghe S, Lucey S. Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs. In: Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors. Computer Vision – ECCV 2022, Cham: Springer Nature Switzerland; 2022, p. 142–58.

[2] Sitzmann V, Martel JNP, Bergman AW, Lindell DB, Wetzstein G. Implicit Neural Representations with Periodic Activation Functions. Proc. NeurIPS, 2020

[4] Saragadam V, LeJeune D, Tan J, Balakrishnan G, Veeraraghavan A, Baraniuk RG. WIRE: Wavelet Implicit Neural Representations, 2023, p. 18507–16

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

We evaluate 4 different activations/nonlinearities

Evaluation

First we try each on the prior architecture

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

\times
\times
\times
\times

ReLU [1]

ReLU [1]

ReLU [1]

ReLU [1]

Ray decoder

Surface decoder

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

\times
\times
\times
\times

ReLU [1]

ReLU [1]

ReLU [1]

ReLU [1]

Ray decoder

WIRE [4]

Surface decoder

ReLU [1]

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

ReLU [1]

\times

ReLU [1]

ReLU [1]

ReLU [1]

\times
\times
\times

SIREN [2]

SIREN [2]

SIREN [2]

SIREN [2]

Ray decoder

WIRE [4]

Surface decoder

ReLU [1]

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

ReLU [1]

\times

ReLU [1]

ReLU [1]

ReLU [1]

\times
\times
\times

SIREN [2]

SIREN [2]

SIREN [2]

SIREN [2]

ReLU [1]

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

ReLU [1]

\times

ReLU [1]

ReLU [1]

ReLU [1]

\times
\times
\times

SIREN [2]

SIREN [2]

SIREN [2]

SIREN [2]

Gauss [3]

Gauss [3]

Gauss [3]

Gauss [3]

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

\times
\times
\times
\times

SIREN [2]

SIREN [2]

SIREN [2]

SIREN [2]

Gauss [3]

Gauss [3]

Gauss [3]

Gauss [3]

WIRE [4]

WIRE [4]

WIRE [4]

WIRE [4]

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

\times
\times
\times
\times

Gauss [3]

Gauss [3]

Gauss [3]

Gauss [3]

WIRE [4]

WIRE [4]

WIRE [4]

WIRE [4]

ReLU [1]

SIREN [2]

Gauss [3]

WIRE [4]

Ground

Truth

\times
\times
\times
\times

Gauss [3]

WIRE [4]

WIRE [4]

WIRE [4]

WIRE [4]

\bigg\}

MARF

arch is

unstable

\bigg\}
\bigg\}

MARF

arch is

unstable

\bigg\}
\bigg\}

MARF

arch is

unstable

\bigg\}
>>> # (BATCH, N_BLOCKS, INTER-BLOCK WIDTH)
>>> input = torch.empty(128, 8, 8) 
>>> weight = torch.empty(8, 8, 8)
>>> output = torch.einsum("...ij,ijk->...ik", input, weight)
>>> output.shape
torch.Size([128, 8, 8])
>>> output.stride()
(8, 1024, 1)
>>> torch.empty(128, 8, 8).stride()
(64, 8, 1)

>>> # (BATCH, N_BLOCKS, INTER-BLOCK WIDTH)
>>> input = torch.empty(128, 8, 8) 
>>> weight = torch.empty(8, 8, 8)
>>> output = torch.einsum("...ij,ijk->...ik", input, weight)
>>> output.shape
torch.Size([128, 8, 8])
>>> output.stride()
(8, 1024, 1)
>>> torch.empty(128, 8, 8).stride()
(64, 8, 1)

>>> # (BATCH, N_BLOCKS, INTER-BLOCK WIDTH)
>>> input = torch.empty(128, 8, 8) 
>>> weight = torch.empty(8, 8, 8)
>>> output = torch.einsum("...ij,ijk->...ik", input, weight)
>>> output.shape
torch.Size([128, 8, 8])
>>> output.stride()
(8, 1024, 1)
>>> torch.empty(128, 8, 8).stride()
(64, 8, 1)

>>> torch.tensor( 1 + 2j ).to(torch.float16)
UserWarning: Casting complex values to real discards the imaginary part

tensor(1., dtype=torch.float16)

>>> # (BATCH, N_BLOCKS, INTER-BLOCK WIDTH)
>>> input = torch.empty(128, 8, 8) 
>>> weight = torch.empty(8, 8, 8)
>>> output = torch.einsum("...ij,ijk->...ik", input, weight)
>>> output.shape
torch.Size([128, 8, 8])
>>> output.stride()
(8, 1024, 1)
>>> torch.empty(128, 8, 8).stride()
(64, 8, 1)

>>> torch.tensor( 1 + 2j ).to(torch.float16)
UserWarning: Casting complex values to real discards the imaginary part

tensor(1., dtype=torch.float16)

Initialization

Surface points

Surface clouds are simple to compute from depth scans or SfM landmarks

Surface points

Agnostic (MARF)

K-Means Clustering (Ours)

Average over 80 runs, for each shape activation function

MARFs

PMARFs

Non-ReLU

too unstable

to get a good read

Strict improvement

Exploring various unsupervised clustering algorithms

Surface points

Interior Volume points

(Requires full view, and is too expensive to compute in a real-time scenario)

Sparse-view Scenarios

35 training views, 15 validation views

~4000 test views

Sparse-view Scenarios

PMARFs

MARF

Sparse-view Scenarios

Conclusion

We present PMARF a Neural Ray Field with

  • Improved multi-view consistency
  • Improved reconstruction fidelity
  • A lower parameter count, and that
  • Performs well with more advanced nonlinearities

Thank you

Questions?