M
A
R
F
A novel neural ray field for
representing 3D shapes
Peder Bergebakken Sundt
Theoharis Theoharis
s.ntnu.no/marf
M
A
R
F
M
A
R
F
Medial
Atom
Ray
Fields
M
A
R
F
Medial
Atom
Ray
Fields
Lambertian
Single network evaluation per pixel.
(Including the normal)
Major speedup for inverse-rendering, and analysis-by-synthesis (Renders in realtime!)
MARFs improve multi-view consistency.
(Not guaranteed in ray fields)
M
A
R
F
Medial
Atom
Ray
Fields
we also get for essentially free
With our novel shape representation
M
A
R
F
Medial
Atom
Ray
Fields
Local Thickness
Unsupervised
Part Segmentation
"Approximate" Normals
We also get for essentially free:
Topological Skeleton
With our novel shape representation
M
A
R
F
Medial
Atom
Ray
Fields
Local Thickness
Subsurface Scattering
\to
M
A
R
F
Medial
Atom
Ray
Fields
With a backward pass we show
that one can also compute
(i.e. not realtime)
M
A
R
F
Medial
Atom
Ray
Fields
True Normals
Curvature
With a backward pass we show
that one can also compute:
Despite the network
having no second derivative
\to
\nabla
(i.e. not realtime)
"Approximate" Normals
M
A
R
F
Medial
Atom
Ray
Fields
Curvature
With a backward pass we show
that one can also compute:
Despite the network
having no second derivative
"Approximate" Normals
True Normals
\to
\nabla
(i.e. not realtime)
M
A
R
F
Medial
Atom
Ray
Fields
Curvature
Anisotropic shading
\to
(Using principal curvatures)
M
A
R
F
Medial
Atom
Ray
Fields
MARFs can represent multiple shapes
in a shared latent space
M
A
R
F
Medial
Atom
Ray
Fields
Latent space traversal
Learned from only 20 shapes
M
A
R
F
Medial
Atom
Ray
Fields
Medial atoms?
Ray Fields?
Neural Fields?
M
A
R
Medial
Atom
Ray
Fields
Medial atoms
Ray Fields
Neural Fields
MARFs
Why Neural Fields?
discrete 3D shape representations
Let's look at the shortcomings of
No topology,
lossy surface reconstruction.
Scales poorly
Fixed topology
or self-intersecting
... map poorly to neural networks!
Discrete 3D shape representations
Points
Voxels
Meshes
O(n ^3)
Functions mapping spatiotemporal coordinates to some quantity
Fields
f
:\mathcal X\to\mathcal Y
Neural
... represented with a neural network
Fields
:\mathcal X\to\mathcal Y
Functions mapping spatiotemporal coordinates to some quantity
f
Neural
Fields
- Resolution Agnostic
- Compact and scales well
- Supports any topology
- "implicit" decoder
Neural Radiance Fields - NeRF
\mapsto
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z
Neural Radiance Fields - NeRF
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z
\mapsto
Neural Radiance Fields - NeRF
\mapsto
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z
Neural Radiance Fields - NeRF
\mapsto
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z
Neural Radiance Fields - NeRF
\int_{0}^{\infty }{L(\mathbf{x} (t),\hat\mathbf{n} (t),\hat\mathbf{v} )\sigma (\mathbf{x} (t))\exp (-\int_{0}^{t}{\sigma (\mathbf{x} (s))ds} )dt}
\(\Rightarrow\) Determine the volume rendering ray integral
Differentiable Ray-Marching
3D Cartesian radiance field
\mapsto
\theta, \phi)
,
(
,
r, g, b
\sigma
)
(
x, y, z
\(\Rightarrow\)
Neural Radiance Fields - NeRF
- Monte-Carlo approximation
- Requires ~100 network evaluations per ray/pixel (slow)
- Has great 3D inductive bias
- By construction
multi-view consistent
Differentiable Ray-Marching
Going faster
4 main strategies
Going faster
4 main strategies
(Offline) Tabulation
Discretization artifacts (aliasing)
Many small networks
Trade compute for memory,
loss of global shape prior
Directly predict
the ray integral
100x speedup,
Learn a distance field
instead of density
10x speedup,
no transparency
O(n^3)
No ray marching!
but difficult
Going faster
Directly predict
the ray integral
100x speedup,
No ray marching!
but difficult
(x, y, z)
\mapsto \mathcal Y
\text{ray}
\cancel{(x, y, z)}
\text{ray}
\ell
- Not multi-view consistent
by construction (4 DoF) - No 3D inductive bias
\mapsto \mathcal Y
\text{ray}
\ell
- Not multi-view consistent
by construction (4 DoF) - No 3D inductive bias
\mapsto \mathcal Y
\ell
(
- Not multi-view consistent
by construction (4 DoF) - No 3D inductive bias
Plücker Coordinate
\text{direction}, \text{moment}
)
No singularities
or discontinuities
\mapsto \mathcal Y
\hat\mathbf q, \mathbf m
No singularities
or discontinuities
Plücker Coordinate
(
)
\mapsto \mathcal Y
Two prior works use this ray encoding
Plücker Coordinate
\hat\mathbf q, \mathbf m
(
)
\ell
\mapsto
\ell
\mathbf p
\in
Intersection Fields
(PRIF)
Networks
\mathbb R ^3
(r,g,b)
\mapsto
Light Field
Primary Ray
Both struggle with discontinuities!
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
B.Y. Feng, Y. Zhang, D. Tang, R. Du, A. Varshney, PRIF: Primary Ray-Based Implicit Function, in: S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022: pp. 138–155. https://doi.org/10.1007/978-3-031-20062-5_9.
\ell
\mapsto
\ell
\mathbf p
\in
Intersection Fields
(PRIF)
\mathbb R ^3
Lipschitz bound!
(r,g,b)
\mapsto
Fuzzy edges
Requires filtering
Primary Ray
Both struggle with discontinuities!
Networks
Light Field
\ell
\mapsto
{\ell\mapsto(r,g,b)}
\mathbf p
\in
Intersection Fields
(PRIF)
Networks
\mathbb R ^3
Lipschitz bound
Light Field
Primary Ray
\ell
\mapsto
{\ell\mapsto(r,g,b)}
\mathbf p
\in
Intersection Fields
(PRIF)
Networks
\mathbb R ^3
Lipschitz bound
\bigg\}
Light Field
Primary Ray
PRIF
Ours
We need to represent 3D shapes
\mathcal O
\subset\mathbb R^3
Figures adapted from
D. Rebain, K. Li, V. Sitzmann, S. Yazdani, K.M. Yi, A. Tagliasacchi, Deep Medial Fields, ArXiv:2106.03804 [Cs]. (2021).
\text{sphere}_{i}
\mathcal O
=
\bigcup\limits_{i=1}^{\infty}
\subset\mathbb R^3
We need to represent 3D shapes
as a sum of spheres
Figures adapted from
D. Rebain, K. Li, V. Sitzmann, S. Yazdani, K.M. Yi, A. Tagliasacchi, Deep Medial Fields, ArXiv:2106.03804 [Cs]. (2021).
The Medial Axis Transform
\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)
A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, A. Telea, 3D Skeletons: A State-of-the-Art Report, Computer Graphics Forum. 35 (2016) 573–597. https://doi.org/10.1111/cgf.12865.
The Medial Axis Transform
\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)
Set of maximally inscribed spheres
Ridges in distance transform
Points with two+ nearest neighbors
Local axis of reflectional symmetry
Defining
A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, A. Telea, 3D Skeletons: A State-of-the-Art Report, Computer Graphics Forum. 35 (2016) 573–597. https://doi.org/10.1111/cgf.12865.
The Medial Axis Transform
\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)
Set of maximally inscribed spheres
Ridges in distance transform
Points with two+ nearest neighbors
Local axis of reflectional symmetry
Defining
A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, A. Telea, 3D Skeletons: A State-of-the-Art Report, Computer Graphics Forum. 35 (2016) 573–597. https://doi.org/10.1111/cgf.12865.
The Medial Axis Transform
\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)
Set of maximally inscribed spheres
Ridges in distance transform
Points with two+ nearest neighbors
Local axis of reflectional symmetry
D. Rebain, B. Angles, J. Valentin, N. Vining, J. Peethambaran, S. Izadi, A. Tagliasacchi, LSMAT Least Squares Medial Axis Transform, Computer Graphics Forum. 38 (2019) 5–18. https://doi.org/10.1111/cgf.13599.
Defining
A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, A. Telea, 3D Skeletons: A State-of-the-Art Report, Computer Graphics Forum. 35 (2016) 573–597. https://doi.org/10.1111/cgf.12865.
The Medial Axis Transform
\text{sphere}_{i}
\mathcal O
=
\{
\}_{i=1}^\infty
\text{MAT}(
)
Set of maximally inscribed spheres
Ridges in distance transform
Points with two+ nearest neighbors
Local axis of reflectional symmetry
D. Rebain, B. Angles, J. Valentin, N. Vining, J. Peethambaran, S. Izadi, A. Tagliasacchi, LSMAT Least Squares Medial Axis Transform, Computer Graphics Forum. 38 (2019) 5–18. https://doi.org/10.1111/cgf.13599.
Defining
Fit for iteration
MARF
\mapsto
\ell
(\mathbf x, r)
Medial Atom
Ray Field
(maximally inscribed sphere)
We propose
The Medial Atom Ray Field
Text
\mapsto
\ell
(\mathbf x, r)
\in\mathbb R ^3\times \mathbb R_{\ge0}
Medial Atom
Ray Field
MARF
We propose
The Medial Atom Ray Field
\mapsto
\ell
Medial Atom
Ray Field
\{(\mathbf x_i, r_i)\}_{i=1}^n
\in(\mathbb R ^3\times \mathbb R_{\ge0}) ^n
s
n
MARF
We propose
The Medial Atom Ray Field
\mapsto
\ell
(\mathbf x, r)
Medial Atom
Ray Field
\{(\mathbf x_i, r_i)\}_{i=1}^n
\in(\mathbb R ^3\times \mathbb R_{\ge0}) ^n
\in
MARF
We propose
The Medial Atom Ray Field
\mapsto
\ell
Medial Atom
Ray Field
\argmin
d(\ell, \mathbf x, r)
(\mathbf x, r)
\{(\mathbf x_i, r_i)\}_{i=1}^n
\in
MARF
We propose
The Medial Atom Ray Field
MARF
\bigg\}
\bigg\}
<
d(\ell, \mathbf x, r)
\ell
\mathbf x_1, r_1
\mathbf x_2, r_2
MARF
d(\ell, \mathbf x, r)
\mathbf x_2, r_2
\mathbf x_1, r_1
\ell
MARF
d(\ell, \mathbf x, r)
\mathbf x_1, r_1
\mathbf x_2, r_2
\ell
Supervising
MARF
\mathbf x_1, r_1
\ell
Supervising
\mathcal L_{\mathbf p}
Training Data
\mathbf x_2, r_2
MARF
\mathbf x_1, r_1
\ell
Supervising
\mathcal L_{\mathbf n}
We show the normal can be computed by
differentiating the network w.r.t. the ray,
\bigg\}
Training Data
MARF
\mathbf x_1, r_1
\ell
Supervising
\mathcal L_{\mathbf n}
We show the normal can be computed by
differentiating the network w.r.t. the ray,
but its much cheaper to draw a line between the atom center and the intersection point!
\bigg\}
-\hat{q}_{1} \left(\frac{\partial \mathbf{p} }{\partial x_{2} } \times \frac{\partial \mathbf{p} }{\partial x_{3} } \right)
-\hat{q}_{2} \left(\frac{\partial \mathbf{p} }{\partial x_{3} } \times \frac{\partial \mathbf{p} }{\partial x_{1} } \right)
-\hat{q}_{3} \left(\frac{\partial \mathbf{p} }{\partial x_{1} } \times \frac{\partial \mathbf{p} }{\partial x_{2} } \right)
MARF
\mathbf x_1, r_1
\ell
Supervising
\bigg\}
\mathcal L_{\mathbf s}
- \(=0\) if the ray should hit
- \(\gt0\) if the ray should miss
- We supervise against
precomputed ground truth
Training Data
MARF
\ell
Supervising
\mathbf x_1, r_1
MARF
\ell
Supervising
\mathbf x_1, r_1
\mathcal L_{\mathbf p}
MARF
\ell
Supervising
\mathbf x_1, r_1
\mathcal L_r
Constant positive
pressure on radius
MARF
\ell
Supervising
\mathbf x_1, r_1
\mathcal L_r
\ell_2
MARF
\ell
Supervising
\mathbf x_1, r_1
\ell_2
MARF
\ell
Supervising
\ell_2
\mathbf x_1, r_1
\mathcal L_{ih}
+\mathcal L_{im}
MARF
\ell
Supervising
\mathbf x_1, r_1
\ell _2
\mathcal L_\text{mv}?
Multi-view consistency
Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle
Supervising
Multi-view consistency
Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle
Supervising
Multi-view consistency
Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle
PRIF
Multi-view consistency
Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle
\bigg\}
PRIF
Multi-view consistency
Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle
\bigg\}
PRIF
MARF
Multi-view consistency
Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle
PRIF
MARF
Multi-view consistency
Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle
MARF
Multi-view consistency
Observation: a point should not change w.r.t. a infinitesimal change in incident viewing angle
\ell
\mapsto
{\ell\mapsto(r,g,b)}
PRIF
Light Fields
MARF
t
\mathbf p
:\ell(t)=
\ell
\mapsto
(\mathbf x, r)
t
(Diffuse only)
Supervising
Multi-view consistency
\ell
\mapsto
\mathbf p
(\mathbf x, r)
Supervising
Multi-view consistency
\ell
\mapsto
\mathbf p
(\mathbf x, r)
Supervising
Multi-view consistency
(t)
\ell
\mapsto
\mathbf p
Supervising
Multi-view consistency
=
\mathbf x_0
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
Supervising
Multi-view consistency
=
\mathbf x_0
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
Supervising
Multi-view consistency
=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
\mathbf x_0
Supervising
Multi-view consistency
=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
\text{GT}
Supervising
Multi-view consistency
\nabla_{\hat {\mathbf q}}
\|
\|
\mathbf p
=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
\text{GT}
Supervising
Multi-view consistency
\nabla_{\hat {\mathbf q}}
\|
\|
=
0
\mathbf p
=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\in\ell
\mathbf p
\mathbf p
\text{GT}
\ell
Supervising
\mapsto
\mathbf p
\ell
=
(t)
+ \hat{\mathbf q}t
\ell
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\text{GT}
\in\ell
=
(t)
\ell
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\text{GT}
MARF
Ray Intersection Field
Any
Multi-view consistency
+ \hat{\mathbf q}t
Supervising
=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\ell
\text{GT}
\in\ell
=
(t)
+ \hat{\mathbf q}t
\ell
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\text{GT}
MARF
Ray Intersection Field
Any
Multi-view consistency
Supervising
=
(t)
+ \hat{\mathbf q}t
\ell
\mapsto
\mathbf p
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf p
\mathbf p
:
\ell
\text{GT}
\in\ell
=
(t)
+ \hat{\mathbf q}t
\ell
\mathcal L_\text{mv}
\nabla_{\hat {\mathbf q}}
=
\|
\|
\mathbf x
\mathbf p
:
\text{GT}
r
(
,
)
MARF
Ray Intersection Field
Any
Multi-view consistency
Supervising
Results
\times 10^4
Results
Ours
Ground
Truth
\times 10^4
Results
Ours
Ground
Truth
\times 10^4
Results
Ours
Ground
Truth
\times 10^4
Results
Ours
w/axis
Ground
Truth
\times 10^4
Ours
PRIF
w/axis
Ground
Truth
Results
\times 10^4
Ours
PRIF
w/axis
Ground
Truth
Results
\times 10^4
Ours
PRIF
w/axis
Ground
Truth
Results
\times 10^4
Questions?
Thank you
s.ntnu.no/marf
s.ntnu.no/marf
skipped slides
\mathbb R^4
Ray Fields
?
- Problem 1: Rays have 4 DoF
- \(\Rightarrow\) No multi-view consistency or 3D inductive bias!
- Problem 2: How to uniquely represent a ray?
SE(3)
6 DoF
3D Rigid Bodies
SE(3)
=
\left[{\begin{matrix} \\ \\ \\ \end{matrix}}\right.
6 DoF
\left.{\begin{matrix} \\ \\ \\ \end{matrix}}\right]
SO(3)
SE(3)
=
SO(3)
\left[{\begin{matrix} \\ \\ \\ \end{matrix}}\right.
6 DoF
\left.{\begin{matrix} \\ \\ \\ \end{matrix}}\right]
\mathbf 0
1
3 DoF
\mathbb R^3
SE(3)
=
SO(3)
\left[{\begin{matrix} \\ \\ \\ \end{matrix}}\right.
6 DoF
\left.{\begin{matrix} \\ \\ \\ \end{matrix}}\right]
\mathbf 0
1
3 DoF
3 DoF
\mathbb R^3
\times
SE(3)
=
SO(3)
\mathbb R^3
\times
Compared to rigid bodies
SE(3)
=
SO(3)
\mathbb R^3
\times
SE(3)
=
SO(3)
\mathbb R^3
\times
Compared to rigid bodies
Rays loose two DoF
SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
SO(3)
\mathbb R^3
\times
Rays are invariant to:
SE(3)
=
SO(3)
\mathbb R^3
\times
Compared to rigid bodies
Rays loose two DoF
SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
SO(3)
\mathbb R^3
\times
Rotation about ray
Rotation about ray
Rays are invariant to:
SE(3)
=
SO(3)
\mathbb R^3
\times
Compared to rigid bodies
Rays loose two DoF
SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
S^2
\mathbb R^3
\times
2-Sphere
\mathbb R^2
Rotation about ray
Translation along ray
Rays are invariant to:
SE(3)
=
SO(3)
\mathbb R^3
\times
Compared to rigid bodies
Rays loose two DoF
SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
S^2
\mathbb R^2
\times
2-Sphere
\mathbb R^3
Rotation about ray
Translation along ray
Rays are invariant to:
(assuming no start)
SE(3)
=
SO(3)
\mathbb R^3
\times
Compared to rigid bodies
Rays loose two DoF
SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
S^2
\times
2-Sphere
Somehow
orthogonal
to ray
T^2
\mathbb R^2
Rotation about ray
Translation along ray
Rays are invariant to:
(assuming no start)
SE(3)
=
SO(3)
\mathbb R^3
\times
Compared to rigid bodies
Rays loose two DoF
SE(3)
=
SO(3)
\mathbb R^3
\times
\mathcal R
=
S^2
T^2
\times
2-Sphere
Somehow
orthogonal
to ray
\mathbb R^2
\mathcal R
=
S^2
T^2
\times
"Hairy ball theorem"
says no
\subset\mathbb R ^4 ?
Without singularities
or discontinuities?
Light Field Networks
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
- Represents rays using Plücker coordinates
\ell = (\hat{\mathbf q}, \mathbf m)
= \hat{\mathbf q}\times \mathbf o
\mathbf o_\perp = \mathbf m\times \hat{\mathbf q}
\mathbb R ^6
\in
Light Field Networks
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
- Represents rays using Plücker coordinates
\ell = (\hat{\mathbf q}, \mathbf m)
\mathbb R ^3
\in
\times
\mathbb R ^3
= \hat{\mathbf q}\times \mathbf o
\mathbf o_\perp = \mathbf m\times \hat{\mathbf q}
S ^2
Light Field Networks
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
- Represents rays using Plücker coordinates
\ell = (\hat{\mathbf q}, \mathbf m)
\mathbb R ^3
\in
\times
S ^2
= \hat{\mathbf q}\times \mathbf o
\mathbf o_\perp = \mathbf m\times \hat{\mathbf q}
\|\hat{\mathbf q}\| = 1
:
\mathbb R ^3
T^2_{\mathbf o_\perp}
Light Field Networks
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
- Represents rays using Plücker coordinates
\ell = (\hat{\mathbf q}, \mathbf m)
\|\hat{\mathbf q}\| = 1
\hat{\mathbf q} \cdot \mathbf m = 0
,
:
T^2_{\mathbf o_\perp}
\in
\times
S ^2
= \hat{\mathbf q}\times \mathbf o
\mathbf o_\perp = \mathbf m\times \hat{\mathbf q}
4 DoF!
\mathbb R ^3
Light Field Networks
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
- Represents rays using Plücker coordinates
- From ray predict color
- Fast
- Simple shading only
- Geometry?
\ell\mapsto(r,g,b)
\ell = (\hat{\mathbf q}, \mathbf m)
Light Field Networks
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
- Compute depth maps by analyzing epipolar images
- Expensive \((O(n^3))\)
- Sparse
- \(\Rightarrow\) Cannot disentangle materials and illumination
Light Field Networks
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
\ell
\mapsto
(r,g,b)
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\mathbf p
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\mathbf p
\mathbb R ^3
\in
Light Field Networks
V. Sitzmann, S. Rezchikov, W.T. Freeman, J.B. Tenenbaum, F. Durand, Light field networks: Neural scene representations with single-evaluation rendering, in: ArXiv, 2021.
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
(r,g,b)
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\perp
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\perp
t
\ell(t) =
Primary Ray Intersection Fields
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf m
=0
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
:
B.Y. Feng, Y. Zhang, D. Tang, R. Du, A. Varshney, PRIF: Primary Ray-Based Implicit Function, in: S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022: pp. 138–155. https://doi.org/10.1007/978-3-031-20062-5_9.
\mathbf o_\perp
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\perp
t
\ell(t) =
Primary Ray Intersection Fields
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
\mathbf o_\perp
=0
B.Y. Feng, Y. Zhang, D. Tang, R. Du, A. Varshney, PRIF: Primary Ray-Based Implicit Function, in: S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022: pp. 138–155. https://doi.org/10.1007/978-3-031-20062-5_9.
\mathbf m
\ell(t) =
\mathbf o
+\hat\mathbf{q}
t
\perp
\ell
\mapsto
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
:
t
\ell(t) =
Primary Ray Intersection Fields
\|\hat{\mathbf q}\| = 1
,
:
T^2
\in
\times
S ^2
\ell
\mapsto
\ell = (\hat{\mathbf q},
)
\mathbf m
\mathbf o_\perp
\hat{\mathbf q} \cdot
=0
\mathbf p
\mathbb R ^3
\in
\subset
\partial\mathcal O
\to
\{\ell_i\}_{i=1}^n
\mapsto
Fit surface
B.Y. Feng, Y. Zhang, D. Tang, R. Du, A. Varshney, PRIF: Primary Ray-Based Implicit Function, in: S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022: pp. 138–155. https://doi.org/10.1007/978-3-031-20062-5_9.
\mathbf o_\perp
MARFs
By pbsds
