M

A

R

F

A novel neural ray field for
representing 3D shapes

Peder Bergebakken Sundt

Theoharis Theoharis

s.ntnu.no/marf

M

A

R

F

M

A

R

F

Medial

Atom

Ray

Fields

M

A

R

F

Medial

Atom

Ray

Fields

Lambertian

Single network evaluation per pixel.
(Including the normal)

Major speedup for inverse-rendering, and analysis-by-synthesis (Renders in realtime!)

MARFs improve multi-view consistency.
(Not guaranteed in ray fields)

M

A

R

F

Medial

Atom

Ray

Fields

we also get for essentially free

With our novel shape representation

M

A

R

F

Medial

Atom

Ray

Fields

Local Thickness

Unsupervised
Part Segmentation

"Approximate" Normals

We also get for essentially free:

Topological Skeleton

With our novel shape representation

M

A

R

F

Medial

Atom

Ray

Fields

Local Thickness

Subsurface Scattering

\to

M

A

R

F

Medial

Atom

Ray

Fields

With a backward pass we show
that one can also compute

(i.e. not realtime)

M

A

R

F

Medial

Atom

Ray

Fields

True Normals

Curvature

With a backward pass we show
that one can also compute:

Despite the network

having no second derivative

\to
\nabla

(i.e. not realtime)

"Approximate" Normals

M

A

R

F

Medial

Atom

Ray

Fields

Curvature

With a backward pass we show
that one can also compute:

Despite the network

having no second derivative

"Approximate" Normals

True Normals

\to
\nabla

(i.e. not realtime)

M

A

R

F

Medial

Atom

Ray

Fields

Curvature

Anisotropic shading

\to

(Using principal curvatures)

M

A

R

F

Medial

Atom

Ray

Fields

MARFs can represent multiple shapes
in a shared latent space

M

A

R

F

Medial

Atom

Ray

Fields

Latent space traversal

Learned from only 20 shapes

M

A

R

F

Medial

Atom

Ray

Fields

Medial atoms?

Ray Fields?

Neural Fields?


M

A

R

Medial

Atom

Ray

Fields

Medial atoms

Ray Fields

Neural Fields

MARFs

Why Neural Fields?

discrete 3D shape representations

Let's look at the shortcomings of

No topology,
lossy surface reconstruction.

Scales poorly

Fixed topology
or self-intersecting

... map poorly to neural networks!

Discrete 3D shape representations

Points

Voxels

Meshes

O(n ^3)

Functions mapping spatiotemporal coordinates to some quantity

 

Fields

f
:\mathcal X\to\mathcal Y

Neural

... represented with a neural network

 

Fields

:\mathcal X\to\mathcal Y

Functions mapping spatiotemporal coordinates to some quantity

 

f

Neural

Fields

  • Resolution Agnostic
  • Compact and scales well
  • Supports any topology
  • "implicit" decoder

Neural Radiance Fields - NeRF

\mapsto
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z

Neural Radiance Fields - NeRF

\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z
\mapsto

Neural Radiance Fields - NeRF

\mapsto
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z

Neural Radiance Fields - NeRF

\mapsto
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z

Neural Radiance Fields - NeRF

\int_{0}^{\infty }{L(\mathbf{x} (t),\hat\mathbf{n} (t),\hat\mathbf{v} )\sigma (\mathbf{x} (t))\exp (-\int_{0}^{t}{\sigma (\mathbf{x} (s))ds} )dt}

\(\Rightarrow\) Determine the volume rendering ray integral

Differentiable Ray-Marching

3D Cartesian radiance field

\mapsto
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z

\(\Rightarrow\)

Neural Radiance Fields - NeRF

  • Monte-Carlo approximation
  • Requires ~100 network evaluations per ray/pixel (slow)
  • Has great 3D inductive bias
  • By construction
    multi-view consistent

Differentiable Ray-Marching

Going faster

4 main strategies

Going faster

4 main strategies

(Offline) Tabulation

Discretization artifacts (aliasing)

Many small networks

Trade compute for memory,

loss of global shape prior

Directly predict

the ray integral

100x speedup,

Learn a distance field

instead of density

10x speedup,
no transparency

O(n^3)

No ray marching!

but difficult

Going faster

Directly predict

the ray integral

100x speedup,

No ray marching!

but difficult

(x, y, z)
\mapsto \mathcal Y
\text{ray}
\cancel{(x, y, z)}
\text{ray}
\ell
  • Not multi-view consistent
    by construction (4 DoF)
  • No 3D inductive bias
\mapsto \mathcal Y
\text{ray}
\ell
  • Not multi-view consistent
    by construction (4 DoF)
  • No 3D inductive bias
\mapsto \mathcal Y
\ell
(
  • Not multi-view consistent
    by construction (4 DoF)
  • No 3D inductive bias

Plücker Coordinate

\text{direction}, \text{moment}
)

No singularities
or discontinuities

\mapsto \mathcal Y
\hat\mathbf q, \mathbf m

No singularities
or discontinuities

Plücker Coordinate

(
)
\mapsto \mathcal Y

Two prior works use this ray encoding

Plücker Coordinate

\hat\mathbf q, \mathbf m
(
)
\ell
\mapsto
\ell
\mathbf p
\in

Intersection Fields
(PRIF)

Networks

\mathbb R ^3
(r,g,b)
\mapsto

Light Field

Primary Ray

Both struggle with discontinuities!

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

B.Y. Feng, Y. Zhang, D. Tang, R. Du, A. Varshney, PRIF: Primary Ray-Based Implicit Function, in: S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022: pp. 138–155. https://doi.org/10.1007/978-3-031-20062-5_9.

\ell
\mapsto
\ell
\mathbf p
\in

Intersection Fields
(PRIF)

\mathbb R ^3

Lipschitz bound!

(r,g,b)
\mapsto

Fuzzy edges

Requires filtering

Primary Ray

Both struggle with discontinuities!

Networks

Light Field

\ell
\mapsto
{\ell\mapsto(r,g,b)}
\mathbf p
\in

Intersection Fields
(PRIF)

Networks

\mathbb R ^3

Lipschitz bound

Light Field

Primary Ray

\ell
\mapsto
{\ell\mapsto(r,g,b)}
\mathbf p
\in

Intersection Fields
(PRIF)

Networks

\mathbb R ^3

Lipschitz bound

\bigg\}

Light Field

Primary Ray

PRIF

Ours

We need to represent 3D shapes

\mathcal O
\subset\mathbb R^3

Figures adapted from

D. Rebain, K. Li, V. Sitzmann, S. Yazdani, K.M. Yi, A. Tagliasacchi, Deep Medial Fields, ArXiv:2106.03804 [Cs]. (2021).

\text{sphere}_{i}
\mathcal O
=
\bigcup\limits_{i=1}^{\infty}
\subset\mathbb R^3

We need to represent 3D shapes

as a sum of spheres

Figures adapted from

D. Rebain, K. Li, V. Sitzmann, S. Yazdani, K.M. Yi, A. Tagliasacchi, Deep Medial Fields, ArXiv:2106.03804 [Cs]. (2021).

The Medial Axis Transform

\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)

A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, A. Telea, 3D Skeletons: A State-of-the-Art Report, Computer Graphics Forum. 35 (2016) 573–597. https://doi.org/10.1111/cgf.12865.

The Medial Axis Transform

\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)

Set of maximally inscribed spheres

Ridges in distance transform

Points with two+ nearest neighbors

Local axis of reflectional symmetry

Defining

A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, A. Telea, 3D Skeletons: A State-of-the-Art Report, Computer Graphics Forum. 35 (2016) 573–597. https://doi.org/10.1111/cgf.12865.

The Medial Axis Transform

\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)

Set of maximally inscribed spheres

Ridges in distance transform

Points with two+ nearest neighbors

Local axis of reflectional symmetry

Defining

A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, A. Telea, 3D Skeletons: A State-of-the-Art Report, Computer Graphics Forum. 35 (2016) 573–597. https://doi.org/10.1111/cgf.12865.

The Medial Axis Transform

\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)

Set of maximally inscribed spheres

Ridges in distance transform

Points with two+ nearest neighbors

Local axis of reflectional symmetry

D. Rebain, B. Angles, J. Valentin, N. Vining, J. Peethambaran, S. Izadi, A. Tagliasacchi, LSMAT Least Squares Medial Axis Transform, Computer Graphics Forum. 38 (2019) 5–18. https://doi.org/10.1111/cgf.13599.

Defining

A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, A. Telea, 3D Skeletons: A State-of-the-Art Report, Computer Graphics Forum. 35 (2016) 573–597. https://doi.org/10.1111/cgf.12865.

The Medial Axis Transform

\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)

Set of maximally inscribed spheres

Ridges in distance transform

Points with two+ nearest neighbors

Local axis of reflectional symmetry

D. Rebain, B. Angles, J. Valentin, N. Vining, J. Peethambaran, S. Izadi, A. Tagliasacchi, LSMAT Least Squares Medial Axis Transform, Computer Graphics Forum. 38 (2019) 5–18. https://doi.org/10.1111/cgf.13599.

Defining

Fit for iteration

MARF

\mapsto
\ell
(\mathbf x, r)

Medial Atom

Ray Field

(maximally inscribed sphere)

We propose

The Medial Atom Ray Field

Text

\mapsto
\ell
(\mathbf x, r)
\in\mathbb R ^3\times \mathbb R_{\ge0}

Medial Atom

Ray Field

MARF

We propose

The Medial Atom Ray Field

\mapsto
\ell

Medial Atom

Ray Field

\{(\mathbf x_i, r_i)\}_{i=1}^n
\in(\mathbb R ^3\times \mathbb R_{\ge0}) ^n

s

n

MARF

We propose

The Medial Atom Ray Field

\mapsto
\ell
(\mathbf x, r)

Medial Atom

Ray Field

\{(\mathbf x_i, r_i)\}_{i=1}^n
\in(\mathbb R ^3\times \mathbb R_{\ge0}) ^n
\in

MARF

We propose

The Medial Atom Ray Field

\mapsto
\ell

Medial Atom

Ray Field

\argmin
d(\ell, \mathbf x, r)
(\mathbf x, r)
\{(\mathbf x_i, r_i)\}_{i=1}^n
\in

MARF

We propose

The Medial Atom Ray Field

MARF

\bigg\}
\bigg\}
<
d(\ell, \mathbf x, r)
\ell
\mathbf x_1, r_1
\mathbf x_2, r_2

MARF

d(\ell, \mathbf x, r)
\mathbf x_2, r_2
\mathbf x_1, r_1
\ell

MARF

d(\ell, \mathbf x, r)
\mathbf x_1, r_1
\mathbf x_2, r_2
\ell

Supervising

MARF

\mathbf x_1, r_1
\ell

Supervising

\mathcal L_{\mathbf p}

Training Data

\mathbf x_2, r_2

MARF

\mathbf x_1, r_1
\ell

Supervising

\mathcal L_{\mathbf n}

We show the normal can be computed by

differentiating the network w.r.t. the ray,

\bigg\}

Training Data

MARF

\mathbf x_1, r_1
\ell

Supervising

\mathcal L_{\mathbf n}

We show the normal can be computed by

differentiating the network w.r.t. the ray,

but its much cheaper to draw a line between the atom center and the intersection point!

\bigg\}
-\hat{q}_{1} \left(\frac{\partial \mathbf{p} }{\partial x_{2} } \times \frac{\partial \mathbf{p} }{\partial x_{3} } \right)
-\hat{q}_{2} \left(\frac{\partial \mathbf{p} }{\partial x_{3} } \times \frac{\partial \mathbf{p} }{\partial x_{1} } \right)
-\hat{q}_{3} \left(\frac{\partial \mathbf{p} }{\partial x_{1} } \times \frac{\partial \mathbf{p} }{\partial x_{2} } \right)

MARF

\mathbf x_1, r_1
\ell

Supervising

\bigg\}
\mathcal L_{\mathbf s}
  • \(=0\) if the ray should hit
  • \(\gt0\) if the ray should miss
  • We supervise against
    precomputed ground truth

Training Data

MARF

\ell

Supervising

\mathbf x_1, r_1

MARF

\ell

Supervising

\mathbf x_1, r_1
\mathcal L_{\mathbf p}

MARF

\ell

Supervising

\mathbf x_1, r_1
\mathcal L_r

Constant positive
pressure on radius

MARF

\ell

Supervising

\mathbf x_1, r_1
\mathcal L_r
\ell_2

MARF

\ell

Supervising

\mathbf x_1, r_1
\ell_2

MARF

\ell

Supervising

\ell_2
\mathbf x_1, r_1
\mathcal L_{ih}
+\mathcal L_{im}

MARF

\ell

Supervising

\mathbf x_1, r_1
\ell _2
\mathcal L_\text{mv}?

Multi-view consistency

Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle

Supervising

Multi-view consistency

Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle

Supervising

Multi-view consistency

Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle

PRIF

Multi-view consistency

Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle

\bigg\}

PRIF

Multi-view consistency

Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle

\bigg\}

PRIF

MARF

Multi-view consistency

Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle

PRIF

MARF

Multi-view consistency

Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle

MARF

Multi-view consistency

Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle

\ell
\mapsto
{\ell\mapsto(r,g,b)}

PRIF

Light Fields

MARF

t
\mathbf p
:\ell(t)=
\ell
\mapsto
(\mathbf x, r)
t

(Diffuse only)

Supervising

Multi-view consistency

\ell
\mapsto
\mathbf p
(\mathbf x, r)

Supervising

Multi-view consistency

\ell
\mapsto
\mathbf p
(\mathbf x, r)

Supervising

Multi-view consistency

(t)
\ell
\mapsto
\mathbf p

Supervising

Multi-view consistency

=
\mathbf x_0
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p

Supervising

Multi-view consistency

=
\mathbf x_0
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p

Supervising

Multi-view consistency

=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
\mathbf x_0

Supervising

Multi-view consistency

=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
\text{GT}

Supervising

Multi-view consistency

\nabla_{\hat {\mathbf q}}
\|
\|
\mathbf p
=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
\text{GT}

Supervising

Multi-view consistency

\nabla_{\hat {\mathbf q}}
\|
\|
=
0
\mathbf p
=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
\text{GT}
\ell

Supervising

\mapsto
\mathbf p
\ell
=
(t)
+ \hat{\mathbf q}t
\ell
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\text{GT}
\in\ell
=
(t)
\ell
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\text{GT}

MARF

Ray Intersection Field

Any

Multi-view consistency

+ \hat{\mathbf q}t

Supervising

=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\ell
\text{GT}
\in\ell
=
(t)
+ \hat{\mathbf q}t
\ell
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\text{GT}

MARF

Ray Intersection Field

Any

Multi-view consistency

Supervising

=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\ell
\text{GT}
\in\ell
=
(t)
+ \hat{\mathbf q}t
\ell
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf x
\mathbf p
:
\text{GT}
r
(
,
)

MARF

Ray Intersection Field

Any

Multi-view consistency

Supervising

Results

\times 10^4

Results

Ours

Ground
Truth

\times 10^4

Results

Ours

Ground
Truth

\times 10^4

Results

Ours

Ground
Truth

\times 10^4

Results

Ours

w/axis

Ground
Truth

\times 10^4

Ours

PRIF

w/axis

Ground
Truth

Results

\times 10^4

Ours

PRIF

w/axis

Ground
Truth

Results

\times 10^4

Ours

PRIF

w/axis

Ground
Truth

Results

\times 10^4

Questions?

Thank you

s.ntnu.no/marf

s.ntnu.no/marf

skipped slides

\mathbb R^4

Ray Fields

?
  • Problem 1: Rays have 4 DoF
    • \(\Rightarrow\) No multi-view consistency or 3D inductive bias!
  • Problem 2: How to uniquely represent a ray?
SE(3)

6 DoF

3D Rigid Bodies

SE(3)
=
\left[{\begin{matrix} \\ \\ \\ \end{matrix}}\right.

6 DoF

\left.{\begin{matrix} \\ \\ \\ \end{matrix}}\right]
SO(3)
SE(3)
=
SO(3)
\left[{\begin{matrix} \\ \\ \\ \end{matrix}}\right.

6 DoF

\left.{\begin{matrix} \\ \\ \\ \end{matrix}}\right]
\mathbf 0
1

3 DoF

\mathbb R^3
SE(3)
=
SO(3)
\left[{\begin{matrix} \\ \\ \\ \end{matrix}}\right.

6 DoF

\left.{\begin{matrix} \\ \\ \\ \end{matrix}}\right]
\mathbf 0
1

3 DoF

3 DoF

\mathbb R^3
\times
SE(3)
=
SO(3)
\mathbb R^3
\times

Compared to rigid bodies

SE(3)
=
SO(3)
\mathbb R^3
\times
SE(3)
=
SO(3)
\mathbb R^3
\times

Compared to rigid bodies

Rays loose two DoF

SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
SO(3)
\mathbb R^3
\times

Rays are invariant to:

SE(3)
=
SO(3)
\mathbb R^3
\times

Compared to rigid bodies

Rays loose two DoF

SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
SO(3)
\mathbb R^3
\times

Rotation about ray

Rotation about ray

Rays are invariant to:

SE(3)
=
SO(3)
\mathbb R^3
\times

Compared to rigid bodies

Rays loose two DoF

SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
S^2
\mathbb R^3
\times

2-Sphere

\mathbb R^2

Rotation about ray

Translation along ray

Rays are invariant to:

SE(3)
=
SO(3)
\mathbb R^3
\times

Compared to rigid bodies

Rays loose two DoF

SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
S^2
\mathbb R^2
\times

2-Sphere

\mathbb R^3

Rotation about ray

Translation along ray

Rays are invariant to:

(assuming no start)

SE(3)
=
SO(3)
\mathbb R^3
\times

Compared to rigid bodies

Rays loose two DoF

SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
S^2
\times

2-Sphere

Somehow

orthogonal
to ray

T^2
\mathbb R^2

Rotation about ray

Translation along ray

Rays are invariant to:

(assuming no start)

SE(3)
=
SO(3)
\mathbb R^3
\times

Compared to rigid bodies

Rays loose two DoF

SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
S^2
T^2
\times

2-Sphere

Somehow

orthogonal
to ray

\mathbb R^2
\mathcal R
=
S^2
T^2
\times

"Hairy ball theorem"

says no

\subset\mathbb R ^4 ?

Without singularities
or discontinuities?

Light Field Networks

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

  • Represents rays using Plücker coordinates

 

\ell = (\hat{\mathbf q}, \mathbf m)
= \hat{\mathbf q}\times \mathbf o
\mathbf o_\perp = \mathbf m\times \hat{\mathbf q}
\mathbb R ^6
\in

Light Field Networks

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

  • Represents rays using Plücker coordinates

 

\ell = (\hat{\mathbf q}, \mathbf m)
\mathbb R ^3
\in
\times
\mathbb R ^3
= \hat{\mathbf q}\times \mathbf o
\mathbf o_\perp = \mathbf m\times \hat{\mathbf q}
S ^2

Light Field Networks

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

  • Represents rays using Plücker coordinates

 

\ell = (\hat{\mathbf q}, \mathbf m)
\mathbb R ^3
\in
\times
S ^2
= \hat{\mathbf q}\times \mathbf o
\mathbf o_\perp = \mathbf m\times \hat{\mathbf q}
\|\hat{\mathbf q}\| = 1
:
\mathbb R ^3
T^2_{\mathbf o_\perp}

Light Field Networks

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

  • Represents rays using Plücker coordinates

 

\ell = (\hat{\mathbf q}, \mathbf m)
\|\hat{\mathbf q}\| = 1
\hat{\mathbf q} \cdot \mathbf m = 0
,
:
T^2_{\mathbf o_\perp}
\in
\times
S ^2
= \hat{\mathbf q}\times \mathbf o
\mathbf o_\perp = \mathbf m\times \hat{\mathbf q}

4 DoF!

\mathbb R ^3

Light Field Networks

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

  • Represents rays using Plücker coordinates
     
  • From ray predict color
    • Fast
    • Simple shading only
    • Geometry?

 

\ell\mapsto(r,g,b)
\ell = (\hat{\mathbf q}, \mathbf m)

Light Field Networks

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

  • Compute depth maps by analyzing epipolar images
    • Expensive \((O(n^3))\)
    • Sparse
    • \(\Rightarrow\) Cannot disentangle materials and illumination

Light Field Networks

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

\ell
\mapsto
(r,g,b)
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\mathbf p
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\mathbf p
\mathbb R ^3
\in

Light Field Networks

V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.

\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
(r,g,b)
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\perp
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\perp
t
\ell(t) =

Primary Ray Intersection Fields

\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
:

B.Y. Feng, Y. Zhang, D. Tang, R. Du, A. Varshney, PRIF: Primary Ray-Based Implicit Function, in: S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022: pp. 138–155. https://doi.org/10.1007/978-3-031-20062-5_9.

\mathbf o_\perp
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\perp
t
\ell(t) =

Primary Ray Intersection Fields

\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf o_\perp
=0

B.Y. Feng, Y. Zhang, D. Tang, R. Du, A. Varshney, PRIF: Primary Ray-Based Implicit Function, in: S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022: pp. 138–155. https://doi.org/10.1007/978-3-031-20062-5_9.

\mathbf m
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\perp
\ell
\mapsto
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
:
t
\ell(t) =

Primary Ray Intersection Fields

\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
=0
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
\to
\{\ell_i\}_{i=1}^n
\mapsto

Fit surface

B.Y. Feng, Y. Zhang, D. Tang, R. Du, A. Varshney, PRIF: Primary Ray-Based Implicit Function, in: S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022: pp. 138–155. https://doi.org/10.1007/978-3-031-20062-5_9.

\mathbf o_\perp