A double-covered
probabilistic ray-based
neural shape representation
Peder Bergebakken Sundt
Theoharis Theoharis
3DOR 2026
A double-covered
probabilistic ray-based
neural shape representation
3DOR 2026
A double-covered
probabilistic ray-based
neural shape representation
3DOR 2026
A double-covered
probabilistic ray-based
neural shape representation
3DOR 2026
\operatorname{softmin}
A double-covered
probabilistic ray-based
neural shape representation
3DOR 2026
\operatorname{softmin}
A double-covered
probabilistic ray-based
neural shape representation
3DOR 2026
- Motivation & Preliminaries
- Contributions
- Results
Why?
Where?
What?
Neural Ray Fields
Motivating
Differentiable rendering requires good gradient flow
Why?
Where?
Primitives
Neural Ray Fields
What?
- Analytical gradients
- Approximate gradients
- Approximate rendering
Differentiable rendering requires good gradient flow
\mathbf x
\mathbf y
Neural Fields
Approximate representations?
Motivating
Why?
Where?
Primitives
Neural Ray Fields
What?
Motivation
- Analytical gradients
- Approximate gradients
- Approximate rendering
- "Coordinate-based" neural network
- Map spatial coordinate to some reconstruction signal.
- Multiple evaluations to sample the reconstruction domain
- DeepSDF, NeRF, NGP
\mathbf x
\mathbf y
Neural Fields
f\big(% ljkasdjklasdjklasd
\big) % asldkhasd
why pick one over the other?
Approximate representations?
Differentiable rendering requires good gradient flow
Why?
Where?
Primitives
Neural Ray Fields
What?
Motivation
- Analytical gradients
- Approximate gradients
- Approximate rendering
f\big(% ljkasdjklasdjklasd
\big) % asldkhasd
why pick one over the other?
f\big(
\big)
\text{Ray}
\mathbf y
Neural Ray Fields
Neural Fields
\mathbf x
\mathbf y
- Single evaluation per ray
Differentiable rendering requires good gradient flow
Why?
Where?
Primitives
Neural Ray Fields
What?
Motivation
\mathbf x
\mathbf y
Neural Fields
Why pick one over the other?
f\big(% ljkasdjklasdjklasd
\big) % asldkhasd
f\big(
\big)
The current SOTA
feature great render times.
Why?
Minimize primitive processing
Minimize network evaluations
Image Diffusion
3D Gaussian Splatting
Voxel Grids
Fast Dipole Sums
Implicit Surface / Radiance Fields
f\big(
\big)
f\big(
\big)
Ray Intersection / Light Fields
Ratio tracking
Edge-sampling
Rasterization
Ray Marching
Ray Casting
Neural Fields
Primitives
Speedup
Where?
Neural Ray Fields
What?
(closed-form solution)
(iterative solution)
Neural Ray fields are compact and feature non-diverging memory access,
making them attractive as intersection
shaders in hybrid path-tracing
Figure:
The Vulkan Ray-Tracing Pipeline
Neural Ray Fields
Where?
Why?
Related Works
A simplified timeline
Related Works
PRIF
MARF
PMARF
PDDF
LFN
Light Fields
Intersection Fields
NFD
AutoInt
5D
4D
5D
4D
𝒩-BVH
LSNIF
Hybrid
SRDF
Hybrid
- [AutoInt] Lindell DB, Martel JNP, Wetzstein G. AutoInt: Automatic Integration for Fast Neural Volume Rendering. Proceedings of the conference on computer vision and pattern recognition (CVPR), Nashville, TN, USA: IEEE; 2021, p. 14551–60.
- [LFN] Sitzmann V, Rezchikov S, Freeman B, Tenenbaum J, Durand F. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems 2021;34:19313–25.
- [PRIF] Feng BY, Zhang Y, Tang D, Du R, Varshney A. PRIF: Primary Ray-Based Implicit Function. In: Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors. Computer Vision – ECCV 2022, vol. 13663, Cham: Springer Nature Switzerland; 2022, p. 138–55.
- [MARF] Sundt PB, Theoharis T. MARF: The Medial Atom Ray Field object representation. Computers & Graphics 2023;115:122–36.
- [PMARF] Sundt PB, Theoharis T. Towards multi-view consistency in neural ray fields using parametric medial surfaces. Computers & Graphics 2024;123:103991.
- [NFD] Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.
- [PDDF] Aumentado-Armstrong T, Tsogkas S, Dickinson S, Jepson A. Probabilistic Directed Distance Fields for Ray-Based Shape Representations. IEEE Transac-tions on Pattern Analysis and Machine Intelligence 2025;47:10243–61.
- [SRDF] Zins P, Xu Y, Boyer E, Wuhrer S, Tung T. Multi-View Reconstruction Using Signed Ray Distance Functions (SRDF). 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE; 2023, p. 16696–706.
- [𝒩-BVH] Weier P, Rath A, Michel É, Georgiev I, Slusallek P, Boubekeur T. N-BVH: Neural ray queries with bounding volume hierarchies. InACM SIGGRAPH 2024 conference papers 2024 Jul 13 (pp. 1-11).
- [LSNIF] Fujieda S, Kao CC, Harada T. LSNIF: Locally-Subdivided Neural Intersection Function. Proceedings of the ACM on Computer Graphics and Interactive Techniques. 2025 May 22;8(1):1-8.
A simplified timeline
Orthogonal to our work.
Plücker
embeddings
Plücker
embeddings
Hybrid
PRIF
MARF
PMARF
PDDF
LFN
This Work
Light Fields
Intersection Fields
NFD
AutoInt
5D
4D
5D
4D
𝒩-BVH
LSNIF
Hybrid
SRDF
Hybrid
- [AutoInt] Lindell DB, Martel JNP, Wetzstein G. AutoInt: Automatic Integration for Fast Neural Volume Rendering. Proceedings of the conference on computer vision and pattern recognition (CVPR), Nashville, TN, USA: IEEE; 2021, p. 14551–60.
- [LFN] Sitzmann V, Rezchikov S, Freeman B, Tenenbaum J, Durand F. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems 2021;34:19313–25.
- [PRIF] Feng BY, Zhang Y, Tang D, Du R, Varshney A. PRIF: Primary Ray-Based Implicit Function. In: Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors. Computer Vision – ECCV 2022, vol. 13663, Cham: Springer Nature Switzerland; 2022, p. 138–55.
- [MARF] Sundt PB, Theoharis T. MARF: The Medial Atom Ray Field object representation. Computers & Graphics 2023;115:122–36.
- [PMARF] Sundt PB, Theoharis T. Towards multi-view consistency in neural ray fields using parametric medial surfaces. Computers & Graphics 2024;123:103991.
- [NFD] Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.
- [PDDF] Aumentado-Armstrong T, Tsogkas S, Dickinson S, Jepson A. Probabilistic Directed Distance Fields for Ray-Based Shape Representations. IEEE Transac-tions on Pattern Analysis and Machine Intelligence 2025;47:10243–61.
- [SRDF] Zins P, Xu Y, Boyer E, Wuhrer S, Tung T. Multi-View Reconstruction Using Signed Ray Distance Functions (SRDF). 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE; 2023, p. 16696–706.
- [𝒩-BVH] Weier P, Rath A, Michel É, Georgiev I, Slusallek P, Boubekeur T. N-BVH: Neural ray queries with bounding volume hierarchies. InACM SIGGRAPH 2024 conference papers 2024 Jul 13 (pp. 1-11).
- [LSNIF] Fujieda S, Kao CC, Harada T. LSNIF: Locally-Subdivided Neural Intersection Function. Proceedings of the ACM on Computer Graphics and Interactive Techniques. 2025 May 22;8(1):1-8.
Related Works
A simplified timeline
We iterate on MARF and PMARF, with some improvements
inspired by NFD and PDDF, and some novel.
View overfitting
The 5 (or 4) degrees-of-freedom in rays
permit view-dependent geometry.
The primary challenge:
You can alleviate view overfitting with
dense multi-view supervision.
...but can one do without?
Early work
⇒ Multi-view consistent priors
and regularization
Overfitting on purpose is a no-go.
(x, y, z, \theta, \phi)
(Medial Atom) Ray Fields
A quick refresher
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
\ell
shape
A quick refresher
(Medial Atom) Ray Fields
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
\mapsto
\mathcal R_{\mathcal O}
\ell
t_\ell
:
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
Ray Marching?
shape
\text{Ray Field}
\downarrow
\text{Neural Network}
\downarrow
A quick refresher
\mathcal O \subset \mathbb R^3
\text{Ray Field}
\downarrow
\ell(t) = \mathbf o + \hat{\mathbf q}t
\mapsto
\mathcal R_{\mathcal O}
\ell
t_\ell
:
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{GT}
\text{PRIF}
Lipschitz continuity
A quick refresher
\operatorname{MAT}(\mathcal O)
\mathcal R_{\mathcal O}
:
\ell
\mapsto
t_\ell
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
\{(\mathbf c_{k}, r_{k} )\}_k
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{MARF}
\downarrow
\mathcal M_{\mathcal O}
\Biggl\{
k inscribed spheres
"medial atoms"
depth
A quick refresher
\longleftarrow
"Topological Skeleton"
Medial Axis / Surface
\longleftarrow
Medial atoms are easy to regularize, and provide good multi-view priors
\operatorname{MAT}(\mathcal O)
:
\ell
\mapsto
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{MARF}
\downarrow
\mathcal M_{\mathcal O}
\text{GT}
\text{MARF}
\text{PRIF}
\{(\mathbf c_{k}, r_{k} )\}_k
k inscribed spheres
"medial atoms"
A quick refresher
\longleftarrow
"Topological Skeleton"
Medial Axis / Surface
\longleftarrow
Medial atoms are easy to regularize, and provide good priors / inductive bias
\operatorname{MAT}(\mathcal O)
:
\ell
\mapsto
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{MARF}
\downarrow
\mathcal M_{\mathcal O}
\text{GT}
\text{MARF}
\text{PRIF}
\{(\mathbf c_{k}, r_{k} )\}_k
k inscribed spheres
"medial atoms"
A quick refresher
\longleftarrow
"Topological Skeleton"
Medial Axis / Surface
\longleftarrow
Medial atoms are easy to regularize, and provide good priors / inductive bias
\operatorname{MAT}(\mathcal O)
:
\ell
\mapsto
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
\{(\mathbf c_{k}, r_{k} )\}_k
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{MARF}
\downarrow
\mathcal M_{\mathcal O}
\downarrow
Number of candidate predictions per ray.
\text{GT}
\text{MARF}
\text{PRIF}
A quick refresher
Our Contributions
- Improve gradient flow with
probabilistic blending - Double cover the medial surface
to represent more shapes - Reduce view overfitting
- Improve how the loss is allocated
between candidate predictions - Improve the loss function itself
\cup
\Big\}
=
- Number of atoms predicted per ray
- Number of representable discontinuities
... but increasing k beyond 16 does not improve reconstruction quality much in practice...
Why?
\Big\{
k determines:
hyperparameter
k
The
Atoms obscure each other
I.e. there is poor gradient flow
\text{No loss for B}
\text{All loss goes to B}
Atoms obscure each other
I.e. there is poor gradient flow
Fuzzy rendering
Why not Gaussians?
They works great for color rendering,
Towards
but not for solid surfaces.
Why not Gaussians?
Fuzzy rendering
Gaussians (or metaballs) are...
- Additive: Stacking Gaussians
cause the covered volume to grow.- Cannot use medial axis
properties for regularization
- Cannot use medial axis
- View-dependent if splatted
- Detrimental to training multi-view stability
- or require iterative ray-surface root finding
- Divergent computation, not feed-forward
It works great for color
Towards
Fuzzy rendering
Then what?
\mapsto
\ell
t
=
\operatornamewithlimits{argmin}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
t : \ell(t) \in \partial\mathcal O
\operatornamewithlimits{softmin}
Consider the ray-to-surface mapping
Fuzzy rendering
Then what?
\ell
\mapsto
t
=
\operatornamewithlimits{argmin}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
t : \ell(t) \in \partial\mathcal O
\operatornamewithlimits{softmin}
Consider the ray-to-surface mapping
\Biggl\{
\Biggl\{
Smoothness
through erosion.
Silhouettes unaffected.
View dependent.
Figure:
Softmin atom normals
Atoms cooperate and specialize to
different parts of the shape
Atoms compete to fit the shape
\text{softmin}
\text{softmin}
Train:
Test:
\text{softmin}
\text{argmin}
Train:
Test:
\text{argmin}
\text{argmin}
Train:
Test:
Softmin Temperature T
We start training at (c) then decay to (f)
Controls blending sharpness/entropy
Double-covering
The Medial Axis
First some preliminaries on parametric MARFs
\text{MARF}
Double-covering
The Medial Axis
A ray
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
k atoms
Method
\text{PMARF}
\to
\mathbb R^3\times S^2
\text{MARF}
\text{PMARF}
Double-covering
The Medial Axis
A ray
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^2 \}_k
\{ \mathbb R^3\times\mathbb R^+ \}_k
k atoms
k coordinates
Method
k
In essence, fit k
parametric surfaces
Couples the atom center and radius,
forms a consistent medial surface.
Shape is however limited
to k medial planes.
\text{MARF}
\text{PMARF}
Double-covering
The Medial Axis
A ray
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^2 \}_k
\{ \mathbb R^3\times\mathbb R^+ \}_k
k atoms
k coordinates
Method
k
\leftarrow %ljkasdjklasdjklasdjklasdjlkasdasd
Requires a parameter-domain jump w.r.t input ray
In essence, fit k
parametric surfaces
\text{MARF}
\text{PMARF}
\text{Our } S^2
Double-covering
The Medial Axis
A ray
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^2 \}_k
\{ \mathbb R^3\times\mathbb R^+ \}_k
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k
k atoms
k coordinates
Method
k
k
\leftarrow %ljkasdjklasdjklasdjklasdjlk
The 3D unit sphere
\text{Our } S^2
Double-covering
The Medial Axis
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k
k
\leftarrow %ljkasdjklasdjklasdjklasdjlk
The 3D unit sphere
\text{Our } S^2
Double-covering
The Medial Axis
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k
k
This works for any axis, provided no cycles are formed.
(i.e. on genus-0 subsets)
\leftarrow %ljkasdjklasdjklasdjklasdjlk
The 3D unit sphere
The union of two such
shapes may have any genus.
Double-covering
The Medial Axis
Equation: Objective Function
\text{Our } S^2
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k
Constrain to unit sphere
Maximize utilization
\to % kjasdasdkasdkads
\to
k
\leftarrow %ljkasdjklasdjklasdjklasdjlk
The 3D unit sphere
Double-covering
The Medial Axis
Equation: Objective Function
\to
\text{Our } S^2
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k
\to
Required to leverage
wrap-around sphere topology
\to % kjasdasdkasdkads
k
\leftarrow %ljkasdjklasdjklasdjklasdjlk
The 3D unit sphere
\mathcal L_{\hat{\mathbf u}}
Constrain to unit sphere
Maximize utilization
Dense Data
Data Synthesis
Data Augmentation
Reducing View Overfitting
With a coarse-to-fine training schedule,
to not compromise the final fidelity.
Gaussian Beam Distribution
Slope
Reducing View Overfitting
Inspired by Rebain et al. (2024) we permute rays with a
Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.
Slope
Waist
Reducing View Overfitting
Gaussian Beam Distribution
Inspired by Rebain et al. (2024) we permute rays with a
Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.
Slope
Waist
Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.
Reducing View Overfitting
Gaussian Beam Distribution
We decay from a wide beam to
a tight beam during training.
Inspired by Rebain et al. (2024) we permute rays with a
\mapsto
\mapsto
\text{Ray}
\{ \text{atoms} \}_k
\mapsto
\mapsto
\text{Ray}
\{ \text{atoms} \}_k
\Big\{
\Big\}_k
\Big\{
\Big\}_k
+
\mathcal N(\mathbf 0, \sigma_{\mathbf u}^2 I_3)
\mathcal N(\mathbf 0, \sigma_{\mathbf u}^2 I_2)
+
We permute the predicted
Parameter Domain Coordinates
Reducing View Overfitting
\mapsto
\mapsto
\text{Ray}
\{ \text{atoms} \}_k
\mapsto
\mapsto
\text{Ray}
\{ \text{atoms} \}_k
\mathcal N(\mathbf 0, \sigma_{\mathbf u}^2 I_3)
+
\mathcal N(\mathbf 0, \sigma_{\mathbf u}^2 I_2)
+
\Big\{
\Big\}_k
\Big\{
\Big\}_k
Reducing View Overfitting
with decaying Gaussian noise
\longleftarrow
Works better when we don't
clamp the radial component
We permute the predicted
Parameter Domain Coordinates
Reducing View Overfitting
We decay the softmin blending temperature T
(sharpness/entropy)
Soft/"see through",
good gradient flow
Sharp/high fidelity,
reduced gradient flow
Reducing View Overfitting
We decay the softmin blending temperature T
(sharpness/entropy)
Reducing View Overfitting
We decay the amount of dropout from excessive to minimal,
in effect band-limiting the initial fit to avoid local minima.
Atom Selection
Improving
I.e. how the gradient is allocated between the predicted atoms.
Atom Selection
With softmin the gradient propagate to
multiple atoms, but only true hits.
Furthermore, the first hit still receive
the majority of the signal.
Improving
Furthermore, the first hit still receive
the majority of the signal.
Atom Selection
With softmin the gradient propagate to
multiple atoms, but only true hits.
Atom Selection
0^\circ
15^\circ
0^\circ
15^\circ
How?
The distance metric
Atom Selection
d_{k,\ell} = \begin{cases}
{\hat{\mathbf q}\cdot\left(\mathbf p_{k,\ell}-\mathbf o \right)}
\quad\quad\quad\quad\ \quad
&\text{if }s_{k,\ell}= 0\\
{\infty}
&\text{if }s_{k,\ell}>0\ \wedge\ \exists i\left(s_{i,\ell} = 0\right)\\
{s_{k,\ell}}
&\text{if }\forall i\left(s_{i,\ell} > 0\right)\\
\end{cases}
\longrightarrow
MARF blends atoms
using the argmin of this metric
Miss predictions are discarded
\longleftarrow
\text{MARF}
d_{k,\ell}^\text{GT} = \begin{cases}
{
\left\|
\left(
\mathbf c_{k,\ell}
+
r_{k,\ell}\hat{\mathbf n}_\ell^\text{GT}
\right)
-
\mathbf p_{\ell}^\text{GT}
\right\|
}
&\text{if }s_\ell^\text{GT}= 0\\
{s_{k,\ell}}
&\text{if }s_\ell^\text{GT}> 0\\
\end{cases}
\longleftarrow
Prefers the hit furthest along negative ray direction
\text{Ours}
Supervised
(SAS)
The distance metric
Atom Selection
Supervised
d_{k,\ell}^\text{GT} = \begin{cases}
{
\left\|
\left(
\mathbf c_{k,\ell}
+
r_{k,\ell}\hat{\mathbf n}_\ell^\text{GT}
\right)
-
\mathbf p_{\ell}^\text{GT}
\right\|
}
&\text{if }s_\ell^\text{GT}= 0\\
{s_{k,\ell}}
&\text{if }s_\ell^\text{GT}> 0\\
\end{cases}
\text{Ours }
Invariant of hit/miss predictions,
only the ground truth matters
\longleftarrow
\longleftarrow
This metric however requires ground truths, making it
only apply to training, not to validation or testing.
\longleftarrow
We instead prefer the atom that
has to move the least
⇒ Repair false misses,
improving recall.
⇒ View-invariant
softmin blending.
(SAS)
d_{k,\ell} = \begin{cases}
{\hat{\mathbf q}\cdot\left(\mathbf p_{k,\ell}-\mathbf o \right)}
\quad\quad\quad\quad\ \quad
&\text{if }s_{k,\ell}= 0\\
{\infty}
&\text{if }s_{k,\ell}>0\ \wedge\ \exists i\left(s_{i,\ell} = 0\right)\\
{s_{k,\ell}}
&\text{if }\forall i\left(s_{i,\ell} > 0\right)\\
\end{cases}
\text{MARF}
Figure:
Reconstruction error of
a training view and
a novel view, 15° apart.
Shading: Chamfer distance
to ground truth mesh.
Atom Selection
Supervised
(SAS)
Our method repairs the silhouette missed by MARF
Objective Function
The MARF loss is good for surface details,
but struggles with atom specialization,
false misses/poor recall, and has
a hit/miss imbalance.
Improving the
Atom Displacement Loss
Atom Displacement loss
Euclidean Normal Loss
Early MV loss
Truncated regularization
Parameter domain
regularization
Atom Displacement Loss
\mathcal L_{\mathbf p} =
h_\ell
h_\ell^\text{GT}
\left\|
\mathbf p_\ell
-
\mathbf p_\ell^\text{GT}
\right\|
\mathcal L_{\mathbf n} =
h_\ell
h_\ell^\text{GT}
\left(
1-
\hat{\mathbf n}_\ell
\cdot
\hat{\mathbf n}_\ell^\text{GT}
\right)
Can pull in opposite directions!
\longleftarrow
Applies to true hits only
\text{MARF}
\text{MARF}
Objective Function
Pushes atom along ray
to fit target depth
\longleftarrow
Pivots atom about hit
to fit target surface normal
\longleftarrow
Atom Displacement Loss
\mathcal L_{\mathbf p} =
h_\ell
h_\ell^\text{GT}
\left\|
\mathbf p_\ell
-
\mathbf p_\ell^\text{GT}
\right\|
\mathcal L_{\mathbf n} =
h_\ell
h_\ell^\text{GT}
\left(
1-
\hat{\mathbf n}_\ell
\cdot
\hat{\mathbf n}_\ell^\text{GT}
\right)
Recall our SAS:
\text{MARF}
\text{MARF}
d_{k,\ell}^\text{GT} = \begin{cases}
{
\qquad
\qquad
\qquad
\qquad
\qquad
\quad
}
&\text{if }s_\ell^\text{GT}= 0\\
{s_{k,\ell}}
&\text{if }s_\ell^\text{GT}> 0\\
\end{cases}
\left\|
\left(
\mathbf c_{k,\ell}
+
r_{k,\ell}
\hat{\mathbf n}_\ell^\text{GT}
\right)
-
\mathbf p_\ell^\text{GT}
\right\|
\longleftarrow
Prefers the atom that
has to move the least.
\mathcal L_{\mathbf c} =
h_\ell^\text{GT}
\sum_k {
a_{k,\ell}
}
\text{Ours}
What if we penalize
this distance?
Objective Function
Atom Displacement Loss
\mathcal L_{\mathbf p} =
h_\ell
h_\ell^\text{GT}
\left\|
\mathbf p_\ell
-
\mathbf p_\ell^\text{GT}
\right\|
\mathcal L_{\mathbf n} =
h_\ell
h_\ell^\text{GT}
\left(
1-
\hat{\mathbf n}_\ell
\cdot
\hat{\mathbf n}_\ell^\text{GT}
\right)
Applies to all hits
\longrightarrow
\mathcal L_{\mathbf c} =
h_\ell^\text{GT}
\sum_k {
a_{k,\ell}
}
\left\|
\left(
\mathbf c_{k,\ell}
+
r_{k,\ell}
\hat{\mathbf n}_\ell^\text{GT}
\right)
-
\mathbf p_\ell^\text{GT}
\right\|
Supervises intersection points
and normals jointly
\longrightarrow
\text{MARF}
\text{MARF}
\text{Ours}
Objective Function
Atom Displacement Loss
Euclidean Normal loss
\longrightarrow
Objective Function
Euclidean Normal loss
Atom Displacement Loss
Early Multi-View
Objective Function
Euclidean Normal loss
Atom Displacement Loss
Early Multi-View
\mathcal L_{\mathbf n}^\text{cos} =
h_\ell
h_\ell^\text{GT}
\left(
1-
\hat{\mathbf n_\ell}
\cdot
\hat{\mathbf n_\ell}^\text{GT}
\right)
\left\|
\hat{\mathbf n_\ell}
-
\hat{\mathbf n_\ell}^\text{GT}
\right\|
h_\ell
h_\ell^\text{GT}
\mathcal L_{\mathbf n}^\text{euc} =
Cosine Distance
Euclidean Distance
Yin R, Chen Y, Karaoglu S, Gevers T. Ray-Distance Volume Rendering for Neural Scene Reconstruction. In: Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G, editors. Computer Vision – ECCV 2024, Cham: Springer Nature Switzerland; 2025, p. 377–94.
https://doi.org/10.1007/978-3-031-72630-9_22
Smooth and stable
Sharp, but unstable
Objective Function
Normal loss
Early Multi-View
First, a small recap
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}
Double backpropagation
\longleftarrow % kasdasdasdkasdkasd
Objective Function
Early Multi-View
... how does differentiating w.r.t. the ray direction help?
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
\xmapsto{\text{Ray Field}}
\mathbf p_\ell
Ray
Hit
: \mathbf o = \mathbf p_\ell^\text{GT}
\ell =
+ \hat{\mathbf q}t
\mathbf o
Objective Function
Early Multi-View
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
\xmapsto[\text{Perp foot}]{\text{Plücker}}
\xmapsto{\text{Ray Field}}
\mathbf p_\ell
Ray
Hit
4D embedding
Global along-ray translation invariance
by construction through normalization
(\mathbf o_\perp, \hat{\mathbf q})
(\mathbf m, \hat{\mathbf q})
or
\longleftarrow
: \mathbf o = \mathbf p_\ell^\text{GT}
\ell =
+ \hat{\mathbf q}t
\mathbf o
\longleftarrow
Ray origin is normalized
before network ever sees it,
\mathbf p_\ell^\text{GT}
but is still a part of the
auto-differentiation graph!
Objective Function
Early Multi-View
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
\xmapsto[\text{Perp foot}]{\text{Plücker}}
\xmapsto{\text{Ray Field}}
\mathbf p_\ell
\mathbf p_\ell^\text{GT}
Ray
Hit
4D embedding
(\mathbf o_\perp, \hat{\mathbf q})
(\mathbf m, \hat{\mathbf q})
or
\longleftarrow
: \mathbf o = \mathbf p_\ell^\text{GT}
\ell =
+ \hat{\mathbf q}t
\mathbf o
\longleftarrow
penalizing changes w.r.t. viewpoint.
\mathcal L_\text{mv}
\Rightarrow
Objective Function
Early Multi-View
Global along-ray translation invariance
by construction through normalization
Ray origin is normalized
before network ever sees it,
but is still a part of the
auto-differentiation graph!
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2
\mathbf p_\ell %kjashdkjahsdkjahdkajhsd
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}
Objective Function
Early Multi-View
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}
Fix intersected atom center and radius, not just the intersection point.
\longleftarrow %asdasd
Objective Function
Early Multi-View
\text{Any MARF}
\left\| \nabla_{\hat{\mathbf q}} \mathbf c_\ell \right\|^2
+
\left\| \nabla_{\hat{\mathbf q}} r_\ell \right\|^2
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
: \mathbf o = \mathbf p_\ell^\text{GT}
\Bigg)\Big\|^2
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\left\| \nabla_{\hat{\mathbf q}} \mathbf c_\ell \right\|^2
+
\left\| \nabla_{\hat{\mathbf q}} r_\ell \right\|^2
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
Ray Decoder
Wastes capacity to accommodate
a multi-view inconsistent ray decoder
\longleftarrow
\text{Any MARF}
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}
: \mathbf o = \mathbf p_\ell^\text{GT}
Objective Function
Early Multi-View
\text{Any PMARF}
Medial Surface Decoders
\left\|
\nabla_{\hat{\mathbf q}}
\sum_{i=1}^k
a_{i,\ell}
\mathbf u_{i,\ell}
\right\|^2
\left\| \nabla_{\hat{\mathbf q}} \mathbf c_\ell \right\|^2
+
\left\| \nabla_{\hat{\mathbf q}} r_\ell \right\|^2
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\text{Any MARF}
\mathcal L_\text{mv} =
h_\ell
h_\ell^\text{GT}
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2
- Skip lots of double backprop
- Multi-view consistent
ray decoder
Weighted with softmin weights
\longleftarrow %asdasd
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}
: \mathbf o = \mathbf p_\ell^\text{GT}
: \mathbf o = \mathbf p_\ell^\text{GT}
Objective Function
Early Multi-View
Truncated Regularization
\text{Any PMARF}
\text{(ours, early)}
Objective Function
Truncated Regularization
Early Multi-View
A brief recap
Objective Function
Truncated Regularization
Early Multi-View
\mathcal L_r
\mathcal L_\sigma
adds a constant positive pressure on atom radii (MAT maximality).
limits per-candidate area of influence, to specialize atoms to separate parts and handle discontinuities.
\ell
there is a single trivial solution
when no other loss apply.
Failure mode:
Objective Function
Truncated Regularization
Early Multi-View
\{
0.3
\ell
\delta_{i,\ell} = \left[s_{i,\ell} \le 0.3\right]
\mathcal L_r
\mathcal L_\sigma
adds a constant positive pressure on atom radii (MAT maximality).
limits per-candidate area of influence, to specialize atoms to separate parts and handle discontinuities.
there is a single trivial solution
when no other loss apply.
Failure mode:
The majority of predictions miss,
from a per-candidate perspective.
Objective Function
Truncated Regularization
Early Multi-View
\{
0.3
\ell
\delta_{i,\ell} = \left[s_{i,\ell} \le 0.3\right]
\mathcal L_r
\mathcal L_\sigma
adds a constant positive pressure on atom radii (MAT maximality).
limits per-candidate area of influence, to specialize atoms to separate parts and handle discontinuities.
there is a single trivial solution
when no other loss apply.
Failure mode:
The majority of predictions miss,
from a per-candidate perspective.
- Reconstruction quality
- Scaling behavior
- Ablations studies
- Training and
rendering speed
\times
3 architectures
2 setups
( + PRIF )
Results
\Bigg\} % 78932789123978123789123789
S^2\text{-parametric} % ljasdjasdjklasd
\mathbb R^2\text{-parametric} % ljasdjasdjklasd
\text{Non-parametric} % ljasdjasdjklasd
\bigg\{ % 78932789123978123789123789
\text{Our loss \& setup} % ljasdjasdjklasd
\text{Baseline loss \& setup} % ljasdjasdjklasd
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Depth-based
(non-medial)
\{
Medial baselines
\Big\{
Baseline
with our S²
\{
Our softmin & improved loss
\big\{
Chamfer Distance
Cosine Similarity
Medial Atom Normals
Differential Normals
Intersection over Union
Precision
Recall
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Results
Reconstruction
Table: Reconstruction scores. Average of 19 shapes.
Scaling
Same correctness, with improved capacity and recall.
Best overall.
\leftarrow %ljadsadsjklasd
Start to fall behind.
Has a "indecisive"
fit from the higher gradient flow.
\leftarrow %ljadsadsjklasd
Results
Reconstruction
Scaling
Renders
Results
Scaling
Renders
\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}
\text{PMARF}
\text{MARF}
\text{PRIF}
\text{Ours}
\text{Ours}
\text{Ours}
fails to generalize
multi-view stable
multi-view stable
multi-view stable
multi-view stable
multi-view stable
degeneracies near
overhangs
degeneracies near
overhangs
limited topology
limited topology
Results
Scaling
Renders
\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}
\text{PMARF}
\text{MARF}
\text{PRIF}
\text{Ours}
\text{Ours}
\text{Ours}
fails to generalize
multi-view stable
multi-view stable
multi-view stable
multi-view stable
multi-view stable
degeneracies near
overhangs
degeneracies near
overhangs
missing silhouette
missing silhouette
missing silhouette
limited topology
limited topology
Results
Scaling
Renders
\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}
\text{PMARF}
\text{MARF}
\text{PRIF}
\text{Ours}
\text{Ours}
\text{Ours}
fails to generalize
multi-view stable
multi-view stable
multi-view stable
multi-view stable
multi-view stable
degeneracies near
overhangs
degeneracies near
overhangs
missing silhouette
missing silhouette
missing silhouette
limited topology
limited topology
problems with inorganic shapes
that feature sharp angles
Renders
Reconstruction
Results
Reconstruction
Specialized
Specialized
Entangled atoms
Limited topology
Vanishing radius
\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}
Results
Reconstruction
Specialized
Specialized
Limited topology
Vanishing radius
\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}
A novel-view problem,
training views are unaffected.
Likely a ray-decoder interpolation issue.
\leftarrow
This hurdle may be fixed with:
Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.
Entangled atoms
Results
Ablations
Table: Ablation studies
Table: Ablation studies
Results
Ablations
Table: Ablation studies
Table: Ablation studies
Results
Ablations
Table: Ablation studies
Table: Ablation studies
Sundt PB, Theoharis T. Towards multi-view consistency in neural ray fields using parametric medial surfaces. Computers & Graphics 2024;123:103991.
k-means init
Agnostic init
Data
Results
Ablations
Table: Ablation studies
Table: Ablation studies
\to
but is benefitial on PMARF baselines.
Reinforces characteristics observed earlier.
Detrimental on Our R² configuration,
Results
Ablations
Table: Ablation studies
Table: Ablation studies
argmin
softmin
data
Results
Ablations
Table: Ablation studies
Table: Ablation studies
⇒ Our Atom Displacement Loss requires Supervised Atom Selection.
our SAS mitigates the issue.
Penalizing normals early is unstable (MARF baseline also avoids it),
Results
Ablations
Table: Ablation studies
Table: Ablation studies
Results
Ablations
Table: Ablation studies
Table: Ablation studies
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2
Results
Ablations
Table: Ablation studies
Table: Ablation studies
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2
Results
Ablations
Table: Ablation studies
Table: Ablation studies
Strictly improved
Detrimental?
Primarily on mechanical shapes
Results
Ablations
Table: Ablation studies
Table: Ablation studies
Ray augmentation improves fidelity and recall.
And so does parameter domain augmentation.
Results
Ablations
Table: Ablation studies
Table: Ablation studies
And so does parameter domain augmentation.
Ray augmentation improves fidelity and recall.
Results
Ablations
Table: Ablation studies
\longrightarrow
Like MARF baseline
Hits only
\longrightarrow
Table: Ablation studies
Our 0.3 threshold strikes a sweetspot between regularizing all rays or regularizing hits only
Results
Ablations
Table: Ablation studies
Results
Ablations
Table: Ablation studies
Results
Ablations
Table: Ablation studies
Results
Ablations
Table: Ablation studies
Excessive
Insufficient
Sweetspot?
Decaying dropout
\tiny\{
Fixed dropout
\Big\{
Our decay from excessive to minimal avoids
local minima without sacrificing fidelity,
but rather improve it.
Results
Ablations
Table: Ablation studies
Cosine and Euclidean
Euclidean-only
Cosine-only
Stable, but smooth
Detailed, but unstable
A balance
Results
Runtime
Table: Timing Results
Ours have more losses, more gradients.
S² has k more activations and a vector normalization.
Results
Runtime
Table: Timing Results
Performance is reclaimed by our early multi-view loss.
Ours have more losses, more gradients.
S² has k more activations and a vector normalization.
Results
Runtime
Table: Timing Results
Parametric networks excel when compute bound.
Softmin blending is less divergent than Argmin lookups.
Results
Runtime
Table: Timing Results
Large but simple MLPs win out when not compute bound.
Parametric networks excel when compute bound.
Softmin blending is less divergent than Argmin lookups.
Thanks
-
A new double-covering parametrization
-
Expands representation capacity
-
-
Probabilistic blending
-
Improves gradient flow and scaling
-
-
New loss functions
-
Increases fidelity, stability and recall
-
-
Faster and improved multi-view stabilization
-
Reduce training times while improving multi-view stability
-
-
Coarse-to-fine training schedule
-
Improves convergence and multi-view generalization
-
Our Contributions Summaried
s2-marf
By pbsds
