A double-covered

probabilistic ray-based

neural shape representation

Peder Bergebakken Sundt

Theoharis Theoharis

3DOR 2026

A double-covered

probabilistic ray-based

neural shape representation

3DOR 2026

A double-covered

probabilistic ray-based

neural shape representation

3DOR 2026

A double-covered

probabilistic ray-based

neural shape representation

3DOR 2026

\operatorname{softmin}

A double-covered

probabilistic ray-based

neural shape representation

3DOR 2026

\operatorname{softmin}

A double-covered

probabilistic ray-based

neural shape representation

3DOR 2026

  • Motivation & Preliminaries
  • Contributions
  • Results

Why?

Where?

What?

Neural Ray Fields

Motivating

Differentiable rendering requires good gradient flow

Why?

Where?

Primitives

Neural Ray Fields

What?

  • Analytical gradients
  • Approximate gradients
  • Approximate rendering

Differentiable rendering requires good gradient flow

\mathbf x
\mathbf y

Neural Fields

Approximate representations?

Motivating

Why?

Where?

Primitives

Neural Ray Fields

What?

Motivation

  • Analytical gradients
  • Approximate gradients
  • Approximate rendering
  • "Coordinate-based" neural network
  • Map spatial coordinate to some reconstruction signal.
  • Multiple evaluations to sample the reconstruction domain
    • DeepSDF, NeRF, NGP
\mathbf x
\mathbf y

Neural Fields

f\big(% ljkasdjklasdjklasd
\big) % asldkhasd

why pick one over the other?

Approximate representations?

Differentiable rendering requires good gradient flow

Why?

Where?

Primitives

Neural Ray Fields

What?

Motivation

  • Analytical gradients
  • Approximate gradients
  • Approximate rendering
f\big(% ljkasdjklasdjklasd
\big) % asldkhasd

why pick one over the other?

f\big(
\big)
\text{Ray}
\mathbf y

Neural Ray Fields

Neural Fields

\mathbf x
\mathbf y
  • Single evaluation per ray

Differentiable rendering requires good gradient flow

Why?

Where?

Primitives

Neural Ray Fields

What?

Motivation

\mathbf x
\mathbf y

Neural Fields

Why pick one over the other?

f\big(% ljkasdjklasdjklasd
\big) % asldkhasd
f\big(
\big)

The current SOTA
feature great render times.

Why?

Minimize primitive processing

Minimize network evaluations

Image Diffusion

3D Gaussian Splatting

Voxel Grids

Fast Dipole Sums

Implicit Surface / Radiance Fields

f\big(
\big)
f\big(
\big)

Ray Intersection / Light Fields

Ratio tracking

https://doi.org/10.1145/3528223.3530073

Edge-sampling

https://doi.org/10.1145/3272127.3275109

Rasterization

Ray Marching

Ray Casting

Neural Fields

Primitives

Speedup

Where?

https://doi.org/10.1007/978-3-031-73232-4_25

https://doi.org/10.1109/CVPR52733.2024.00765

Neural Ray Fields

What?

(closed-form solution)

(iterative solution)

Neural Ray fields are compact and feature non-diverging memory access,

making them attractive as intersection
shaders in hybrid path-tracing

Figure: 
The Vulkan Ray-Tracing Pipeline

Neural Ray Fields

Where?

Why?

Related Works

A simplified timeline

Related Works

PRIF

MARF

PMARF

PDDF

LFN

Light Fields

Intersection Fields

NFD

AutoInt

5D

4D

5D

4D

𝒩-BVH

LSNIF

Hybrid

SRDF

Hybrid

  • [AutoInt] Lindell DB, Martel JNP, Wetzstein G. AutoInt: Automatic Integration for Fast Neural Volume Rendering. Proceedings of the conference on computer vision and pattern recognition (CVPR), Nashville, TN, USA: IEEE; 2021, p. 14551–60.
  • [LFN] Sitzmann V, Rezchikov S, Freeman B, Tenenbaum J, Durand F. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems 2021;34:19313–25.
  • [PRIF] Feng BY, Zhang Y, Tang D, Du R, Varshney A. PRIF: Primary Ray-Based Implicit Function. In: Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors. Computer Vision – ECCV 2022, vol. 13663, Cham: Springer Nature Switzerland; 2022, p. 138–55.
  • [MARF] Sundt PB, Theoharis T. MARF: The Medial Atom Ray Field object representation. Computers & Graphics 2023;115:122–36.
  • [PMARF] Sundt PB, Theoharis T. Towards multi-view consistency in neural ray fields using parametric medial surfaces. Computers & Graphics 2024;123:103991.
  • [NFD] Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.
  • [PDDF] Aumentado-Armstrong T, Tsogkas S, Dickinson S, Jepson A. Probabilistic Directed Distance Fields for Ray-Based Shape Representations. IEEE Transac-tions on Pattern Analysis and Machine Intelligence 2025;47:10243–61.
  • [SRDF] Zins P, Xu Y, Boyer E, Wuhrer S, Tung T. Multi-View Reconstruction Using Signed Ray Distance Functions (SRDF). 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE; 2023, p. 16696–706.
  • [𝒩-BVH] Weier P, Rath A, Michel É, Georgiev I, Slusallek P, Boubekeur T. N-BVH: Neural ray queries with bounding volume hierarchies. InACM SIGGRAPH 2024 conference papers 2024 Jul 13 (pp. 1-11).
  • [LSNIF] Fujieda S, Kao CC, Harada T. LSNIF: Locally-Subdivided Neural Intersection Function. Proceedings of the ACM on Computer Graphics and Interactive Techniques. 2025 May 22;8(1):1-8.

A simplified timeline

Orthogonal to our work.

Plücker
embeddings

Plücker
embeddings

Hybrid

PRIF

MARF

PMARF

PDDF

LFN

This Work

Light Fields

Intersection Fields

NFD

AutoInt

5D

4D

5D

4D

𝒩-BVH

LSNIF

Hybrid

SRDF

Hybrid

  • [AutoInt] Lindell DB, Martel JNP, Wetzstein G. AutoInt: Automatic Integration for Fast Neural Volume Rendering. Proceedings of the conference on computer vision and pattern recognition (CVPR), Nashville, TN, USA: IEEE; 2021, p. 14551–60.
  • [LFN] Sitzmann V, Rezchikov S, Freeman B, Tenenbaum J, Durand F. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems 2021;34:19313–25.
  • [PRIF] Feng BY, Zhang Y, Tang D, Du R, Varshney A. PRIF: Primary Ray-Based Implicit Function. In: Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors. Computer Vision – ECCV 2022, vol. 13663, Cham: Springer Nature Switzerland; 2022, p. 138–55.
  • [MARF] Sundt PB, Theoharis T. MARF: The Medial Atom Ray Field object representation. Computers & Graphics 2023;115:122–36.
  • [PMARF] Sundt PB, Theoharis T. Towards multi-view consistency in neural ray fields using parametric medial surfaces. Computers & Graphics 2024;123:103991.
  • [NFD] Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.
  • [PDDF] Aumentado-Armstrong T, Tsogkas S, Dickinson S, Jepson A. Probabilistic Directed Distance Fields for Ray-Based Shape Representations. IEEE Transac-tions on Pattern Analysis and Machine Intelligence 2025;47:10243–61.
  • [SRDF] Zins P, Xu Y, Boyer E, Wuhrer S, Tung T. Multi-View Reconstruction Using Signed Ray Distance Functions (SRDF). 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE; 2023, p. 16696–706.
  • [𝒩-BVH] Weier P, Rath A, Michel É, Georgiev I, Slusallek P, Boubekeur T. N-BVH: Neural ray queries with bounding volume hierarchies. InACM SIGGRAPH 2024 conference papers 2024 Jul 13 (pp. 1-11).
  • [LSNIF] Fujieda S, Kao CC, Harada T. LSNIF: Locally-Subdivided Neural Intersection Function. Proceedings of the ACM on Computer Graphics and Interactive Techniques. 2025 May 22;8(1):1-8.

Related Works

A simplified timeline

We iterate on MARF and PMARF, with some improvements
inspired by NFD and PDDF, and some novel.

View overfitting

The 5 (or 4) degrees-of-freedom in rays
permit view-dependent geometry.

The primary challenge:

You can alleviate view overfitting with
dense multi-view supervision.

...but can one do without?

Early work

⇒ Multi-view consistent priors
and regularization

Overfitting on purpose is a no-go.

(x, y, z, \theta, \phi)

(Medial Atom) Ray Fields

A quick refresher

\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
\ell

shape

A quick refresher

(Medial Atom) Ray Fields

\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
\mapsto
\mathcal R_{\mathcal O}
\ell
t_\ell
:
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}

Ray Marching?

shape

\text{Ray Field}
\downarrow
\text{Neural Network}
\downarrow

A quick refresher

\mathcal O \subset \mathbb R^3
\text{Ray Field}
\downarrow
\ell(t) = \mathbf o + \hat{\mathbf q}t
\mapsto
\mathcal R_{\mathcal O}
\ell
t_\ell
:
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{GT}
\text{PRIF}

Lipschitz continuity

A quick refresher

\operatorname{MAT}(\mathcal O)
\mathcal R_{\mathcal O}
:
\ell
\mapsto
t_\ell
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
\{(\mathbf c_{k}, r_{k} )\}_k
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{MARF}
\downarrow
\mathcal M_{\mathcal O}
\Biggl\{

k inscribed spheres

"medial atoms"

depth

A quick refresher

\longleftarrow

"Topological Skeleton"

Medial Axis / Surface

\longleftarrow

Medial atoms are easy to regularize, and provide good multi-view priors

\operatorname{MAT}(\mathcal O)
:
\ell
\mapsto
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{MARF}
\downarrow
\mathcal M_{\mathcal O}
\text{GT}
\text{MARF}
\text{PRIF}
\{(\mathbf c_{k}, r_{k} )\}_k

k inscribed spheres

"medial atoms"

A quick refresher

\longleftarrow

"Topological Skeleton"

Medial Axis / Surface

\longleftarrow

Medial atoms are easy to regularize, and provide good priors / inductive bias

\operatorname{MAT}(\mathcal O)
:
\ell
\mapsto
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{MARF}
\downarrow
\mathcal M_{\mathcal O}
\text{GT}
\text{MARF}
\text{PRIF}
\{(\mathbf c_{k}, r_{k} )\}_k

k inscribed spheres

"medial atoms"

A quick refresher

\longleftarrow

"Topological Skeleton"

Medial Axis / Surface

\longleftarrow

Medial atoms are easy to regularize, and provide good priors / inductive bias

\operatorname{MAT}(\mathcal O)
:
\ell
\mapsto
\mathcal O \subset \mathbb R^3
\ell(t) = \mathbf o + \hat{\mathbf q}t
\{(\mathbf c_{k}, r_{k} )\}_k
=
\operatornamewithlimits{argmin}_{t\ :\ \ell(t) \in \partial\mathcal O}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
\text{MARF}
\downarrow
\mathcal M_{\mathcal O}
\downarrow

Number of candidate predictions per ray.

\text{GT}
\text{MARF}
\text{PRIF}

A quick refresher

Our Contributions

  • Improve gradient flow with
    probabilistic blending
  • Double cover the medial surface
    to represent more shapes
  • Reduce view overfitting
  • Improve how the loss is allocated
    between candidate predictions
  • Improve the loss function itself
\cup
\Big\}
=
  • Number of atoms predicted per ray
  • Number of representable discontinuities

... but increasing k beyond 16 does not improve reconstruction quality much in practice...

Why?

\Big\{

k determines:

hyperparameter

k

The





Atoms obscure each other

I.e. there is poor gradient flow

\text{No loss for B}
\text{All loss goes to B}





Atoms obscure each other

I.e. there is poor gradient flow

Fuzzy rendering

Why not Gaussians?

They works great for color rendering,

Towards

but not for solid surfaces.

Why not Gaussians?

Fuzzy rendering

Gaussians (or metaballs) are...

  • Additive: Stacking Gaussians
    cause the covered volume to grow.
    • Cannot use medial axis
      properties for regularization
  • View-dependent if splatted
    • Detrimental to training multi-view stability
  • or require iterative ray-surface root finding
    • Divergent computation, not feed-forward

It works great for color

 

Towards

Fuzzy rendering

Then what?

\mapsto
\ell
t
=
\operatornamewithlimits{argmin}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
t : \ell(t) \in \partial\mathcal O
\operatornamewithlimits{softmin}

Consider the ray-to-surface mapping

Fuzzy rendering

Then what?

\ell
\mapsto
t
=
\operatornamewithlimits{argmin}
{\hat{\mathbf q}\cdot\left(\ell\left(t\right)-\mathbf o \right)}
t : \ell(t) \in \partial\mathcal O
\operatornamewithlimits{softmin}

Consider the ray-to-surface mapping

\Biggl\{
\Biggl\{

Smoothness
through erosion.

Silhouettes unaffected.

View dependent.

Figure:

Softmin atom normals

Atoms cooperate and specialize to
different parts of the shape

Atoms compete to fit the shape

\text{softmin}
\text{softmin}

Train:

Test:

\text{softmin}
\text{argmin}

Train:

Test:

\text{argmin}
\text{argmin}

Train:

Test:

Softmin Temperature T

We start training at (c) then decay to (f)

Controls blending sharpness/entropy

Double-covering

The Medial Axis

First some preliminaries on parametric MARFs

\text{MARF}

Double-covering

The Medial Axis

A ray

\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k

k atoms

Method

\text{PMARF}
\to
\mathbb R^3\times S^2
\text{MARF}
\text{PMARF}

Double-covering

The Medial Axis

A ray

\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^2 \}_k
\{ \mathbb R^3\times\mathbb R^+ \}_k

k atoms

k coordinates

Method

k

In essence, fit k
parametric surfaces

Couples the atom center and radius,
forms a consistent medial surface.

Shape is however limited
to k medial planes.

\text{MARF}
\text{PMARF}

Double-covering

The Medial Axis

A ray

\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^2 \}_k
\{ \mathbb R^3\times\mathbb R^+ \}_k

k atoms

k coordinates

Method

k
\leftarrow %ljkasdjklasdjklasdjklasdjlkasdasd

Requires a parameter-domain jump w.r.t input ray

In essence, fit k
parametric surfaces

\text{MARF}
\text{PMARF}
\text{Our } S^2

Double-covering

The Medial Axis

A ray

\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^2 \}_k
\{ \mathbb R^3\times\mathbb R^+ \}_k
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k

k atoms

k coordinates

Method

k
k
\leftarrow %ljkasdjklasdjklasdjklasdjlk

The 3D unit sphere

\text{Our } S^2

Double-covering

The Medial Axis

\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k
k
\leftarrow %ljkasdjklasdjklasdjklasdjlk

The 3D unit sphere

\text{Our } S^2

Double-covering

The Medial Axis

\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k
k

This works for any axis, provided no cycles are formed.

(i.e. on genus-0 subsets)

\leftarrow %ljkasdjklasdjklasdjklasdjlk

The 3D unit sphere

The union of two such
shapes may have any genus.

Double-covering

The Medial Axis

Equation: Objective Function

\text{Our } S^2
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k

Constrain to unit sphere

Maximize utilization

\to % kjasdasdkasdkads
\to
k
\leftarrow %ljkasdjklasdjklasdjklasdjlk

The 3D unit sphere

Double-covering

The Medial Axis

Equation: Objective Function

\to
\text{Our } S^2
\to
\to
\mathbb R^3\times S^2
\{ \mathbb R^3\times\mathbb R^+ \}_k
\{ S^2 \}_k
\to

Required to leverage
wrap-around sphere topology

\to % kjasdasdkasdkads
k
\leftarrow %ljkasdjklasdjklasdjklasdjlk

The 3D unit sphere

\mathcal L_{\hat{\mathbf u}}

Constrain to unit sphere

Maximize utilization

Dense Data

Data Synthesis

Data Augmentation

Reducing View Overfitting

With a coarse-to-fine training schedule,
to not compromise the final fidelity.

Gaussian Beam Distribution

Slope

Reducing View Overfitting

Inspired by Rebain et al. (2024) we permute rays with a

Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.

Slope

Waist

Reducing View Overfitting

Gaussian Beam Distribution

Inspired by Rebain et al. (2024) we permute rays with a

Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.

Slope

Waist

Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.

Reducing View Overfitting

Gaussian Beam Distribution

We decay from a wide beam to
a tight beam during training.

Inspired by Rebain et al. (2024) we permute rays with a

\mapsto
\mapsto
\text{Ray}
\{ \text{atoms} \}_k
\mapsto
\mapsto
\text{Ray}
\{ \text{atoms} \}_k
\Big\{
\Big\}_k
\Big\{
\Big\}_k
+
\mathcal N(\mathbf 0, \sigma_{\mathbf u}^2 I_3)
\mathcal N(\mathbf 0, \sigma_{\mathbf u}^2 I_2)
+

We permute the predicted
Parameter Domain Coordinates

Reducing View Overfitting

\mapsto
\mapsto
\text{Ray}
\{ \text{atoms} \}_k
\mapsto
\mapsto
\text{Ray}
\{ \text{atoms} \}_k
\mathcal N(\mathbf 0, \sigma_{\mathbf u}^2 I_3)
+
\mathcal N(\mathbf 0, \sigma_{\mathbf u}^2 I_2)
+
\Big\{
\Big\}_k
\Big\{
\Big\}_k

Reducing View Overfitting

with decaying Gaussian noise

\longleftarrow

Works better when we don't
clamp the radial component

We permute the predicted
Parameter Domain Coordinates

Reducing View Overfitting

We decay the softmin blending temperature T

(sharpness/entropy)

Soft/"see through",
good gradient flow

Sharp/high fidelity,
reduced gradient flow

Reducing View Overfitting

We decay the softmin blending temperature T

(sharpness/entropy)

Reducing View Overfitting

We decay the amount of dropout from excessive to minimal,

in effect band-limiting the initial fit to avoid local minima.

Atom Selection

Improving

I.e. how the gradient is allocated between the predicted atoms.

Atom Selection

With softmin the gradient propagate to
multiple atoms, but only true hits.

Furthermore, the first hit still receive
the majority of the signal.

Improving

Furthermore, the first hit still receive
the majority of the signal.

Atom Selection

With softmin the gradient propagate to
multiple atoms, but only true hits.

Atom Selection

0^\circ
15^\circ
0^\circ
15^\circ

How?

The distance metric

Atom Selection

d_{k,\ell} = \begin{cases} {\hat{\mathbf q}\cdot\left(\mathbf p_{k,\ell}-\mathbf o \right)} \quad\quad\quad\quad\ \quad &\text{if }s_{k,\ell}= 0\\ {\infty} &\text{if }s_{k,\ell}>0\ \wedge\ \exists i\left(s_{i,\ell} = 0\right)\\ {s_{k,\ell}} &\text{if }\forall i\left(s_{i,\ell} > 0\right)\\ \end{cases}
\longrightarrow

MARF blends atoms
using the argmin of this metric

Miss predictions are discarded

\longleftarrow
\text{MARF}
d_{k,\ell}^\text{GT} = \begin{cases} { \left\| \left( \mathbf c_{k,\ell} + r_{k,\ell}\hat{\mathbf n}_\ell^\text{GT} \right) - \mathbf p_{\ell}^\text{GT} \right\| } &\text{if }s_\ell^\text{GT}= 0\\ {s_{k,\ell}} &\text{if }s_\ell^\text{GT}> 0\\ \end{cases}
\longleftarrow

Prefers the hit furthest along negative ray direction

\text{Ours}

Supervised

(SAS)

The distance metric

Atom Selection

Supervised

d_{k,\ell}^\text{GT} = \begin{cases} { \left\| \left( \mathbf c_{k,\ell} + r_{k,\ell}\hat{\mathbf n}_\ell^\text{GT} \right) - \mathbf p_{\ell}^\text{GT} \right\| } &\text{if }s_\ell^\text{GT}= 0\\ {s_{k,\ell}} &\text{if }s_\ell^\text{GT}> 0\\ \end{cases}
\text{Ours }

Invariant of hit/miss predictions,

only the ground truth matters

\longleftarrow
\longleftarrow

This metric however requires ground truths, making it
only apply to training, not to validation or testing.

\longleftarrow

We instead prefer the atom that
has to move the least

⇒ Repair false misses,
    improving recall.

⇒ View-invariant
    softmin blending.

(SAS)

d_{k,\ell} = \begin{cases} {\hat{\mathbf q}\cdot\left(\mathbf p_{k,\ell}-\mathbf o \right)} \quad\quad\quad\quad\ \quad &\text{if }s_{k,\ell}= 0\\ {\infty} &\text{if }s_{k,\ell}>0\ \wedge\ \exists i\left(s_{i,\ell} = 0\right)\\ {s_{k,\ell}} &\text{if }\forall i\left(s_{i,\ell} > 0\right)\\ \end{cases}
\text{MARF}

Figure:

Reconstruction error of
a training view and
a novel view, 15° apart.

Shading: Chamfer distance
to ground truth mesh.

Atom Selection

Supervised

(SAS)

Our method repairs the silhouette missed by MARF 

Objective Function

The MARF loss is good for surface details,
but struggles with atom specialization,
false misses/poor recall, and has
a hit/miss imbalance.

Improving the

 Atom Displacement Loss 

Atom Displacement loss

Euclidean Normal Loss

Early MV loss

Truncated regularization

Parameter domain
regularization

Atom Displacement Loss

\mathcal L_{\mathbf p} = h_\ell h_\ell^\text{GT} \left\| \mathbf p_\ell - \mathbf p_\ell^\text{GT} \right\|
\mathcal L_{\mathbf n} = h_\ell h_\ell^\text{GT} \left( 1- \hat{\mathbf n}_\ell \cdot \hat{\mathbf n}_\ell^\text{GT} \right)

Can pull in opposite directions!

\longleftarrow

Applies to true hits only  

\text{MARF}
\text{MARF}

Objective Function

Pushes atom along ray
to fit target depth

\longleftarrow

Pivots atom about hit
to fit target surface normal

\longleftarrow

Atom Displacement Loss

\mathcal L_{\mathbf p} = h_\ell h_\ell^\text{GT} \left\| \mathbf p_\ell - \mathbf p_\ell^\text{GT} \right\|
\mathcal L_{\mathbf n} = h_\ell h_\ell^\text{GT} \left( 1- \hat{\mathbf n}_\ell \cdot \hat{\mathbf n}_\ell^\text{GT} \right)

Recall our SAS:

\text{MARF}
\text{MARF}
d_{k,\ell}^\text{GT} = \begin{cases} { \qquad \qquad \qquad \qquad \qquad \quad } &\text{if }s_\ell^\text{GT}= 0\\ {s_{k,\ell}} &\text{if }s_\ell^\text{GT}> 0\\ \end{cases}
\left\| \left( \mathbf c_{k,\ell} + r_{k,\ell} \hat{\mathbf n}_\ell^\text{GT} \right) - \mathbf p_\ell^\text{GT} \right\|
\longleftarrow

Prefers the atom that
has to move the least.

\mathcal L_{\mathbf c} = h_\ell^\text{GT} \sum_k { a_{k,\ell} }
\text{Ours}

What if we penalize
this distance?

Objective Function

Atom Displacement Loss

\mathcal L_{\mathbf p} = h_\ell h_\ell^\text{GT} \left\| \mathbf p_\ell - \mathbf p_\ell^\text{GT} \right\|
\mathcal L_{\mathbf n} = h_\ell h_\ell^\text{GT} \left( 1- \hat{\mathbf n}_\ell \cdot \hat{\mathbf n}_\ell^\text{GT} \right)

Applies to all hits

\longrightarrow
\mathcal L_{\mathbf c} = h_\ell^\text{GT} \sum_k { a_{k,\ell} }
\left\| \left( \mathbf c_{k,\ell} + r_{k,\ell} \hat{\mathbf n}_\ell^\text{GT} \right) - \mathbf p_\ell^\text{GT} \right\|

Supervises intersection points
and normals jointly

\longrightarrow
\text{MARF}
\text{MARF}
\text{Ours}

Objective Function

Atom Displacement Loss

Euclidean Normal loss

\longrightarrow

Objective Function

Euclidean Normal loss

Atom Displacement Loss

Early Multi-View

Objective Function

Euclidean Normal loss

Atom Displacement Loss

Early Multi-View

\mathcal L_{\mathbf n}^\text{cos} =
h_\ell h_\ell^\text{GT}
\left( 1- \hat{\mathbf n_\ell} \cdot \hat{\mathbf n_\ell}^\text{GT} \right)
\left\| \hat{\mathbf n_\ell} - \hat{\mathbf n_\ell}^\text{GT} \right\|
h_\ell h_\ell^\text{GT}
\mathcal L_{\mathbf n}^\text{euc} =

Cosine Distance

Euclidean Distance

Yin R, Chen Y, Karaoglu S, Gevers T. Ray-Distance Volume Rendering for Neural Scene Reconstruction. In: Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G, editors. Computer Vision – ECCV 2024, Cham: Springer Nature Switzerland; 2025, p. 377–94.
https://doi.org/10.1007/978-3-031-72630-9_22

Smooth and stable

Sharp, but unstable

Objective Function

Normal loss

Early Multi-View

First, a small recap

\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}

Double backpropagation

\longleftarrow % kasdasdasdkasdkasd

Objective Function

Early Multi-View

... how does differentiating w.r.t. the ray direction help?

\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
\xmapsto{\text{Ray Field}}
\mathbf p_\ell

Ray

Hit

: \mathbf o = \mathbf p_\ell^\text{GT}
\ell =
+ \hat{\mathbf q}t
\mathbf o

Objective Function

Early Multi-View

\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
\xmapsto[\text{Perp foot}]{\text{Plücker}}
\xmapsto{\text{Ray Field}}
\mathbf p_\ell

Ray

Hit

4D embedding

Global along-ray translation invariance
by construction through normalization

(\mathbf o_\perp, \hat{\mathbf q})
(\mathbf m, \hat{\mathbf q})

or

\longleftarrow
: \mathbf o = \mathbf p_\ell^\text{GT}
\ell =
+ \hat{\mathbf q}t
\mathbf o
\longleftarrow

Ray origin is normalized
before network ever sees it,

\mathbf p_\ell^\text{GT}

but is still a part of the
auto-differentiation graph!

Objective Function

Early Multi-View

\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
\xmapsto[\text{Perp foot}]{\text{Plücker}}
\xmapsto{\text{Ray Field}}
\mathbf p_\ell
\mathbf p_\ell^\text{GT}

Ray

Hit

4D embedding

(\mathbf o_\perp, \hat{\mathbf q})
(\mathbf m, \hat{\mathbf q})

or

\longleftarrow
: \mathbf o = \mathbf p_\ell^\text{GT}
\ell =
+ \hat{\mathbf q}t
\mathbf o
\longleftarrow

penalizing changes w.r.t. viewpoint.

\mathcal L_\text{mv}
\Rightarrow

Objective Function

Early Multi-View

Global along-ray translation invariance
by construction through normalization

Ray origin is normalized
before network ever sees it,

but is still a part of the
auto-differentiation graph!

\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2
\mathbf p_\ell %kjashdkjahsdkjahdkajhsd
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}

Objective Function

Early Multi-View

\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}

Fix intersected atom center and radius, not just the intersection point.

\longleftarrow %asdasd

Objective Function

Early Multi-View

\text{Any MARF}
\left\| \nabla_{\hat{\mathbf q}} \mathbf c_\ell \right\|^2 + \left\| \nabla_{\hat{\mathbf q}} r_\ell \right\|^2
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
: \mathbf o = \mathbf p_\ell^\text{GT}
\Bigg)\Big\|^2
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\left\| \nabla_{\hat{\mathbf q}} \mathbf c_\ell \right\|^2 + \left\| \nabla_{\hat{\mathbf q}} r_\ell \right\|^2
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}

Ray Decoder

Wastes capacity to accommodate
a multi-view inconsistent ray decoder

\longleftarrow
\text{Any MARF}
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}
: \mathbf o = \mathbf p_\ell^\text{GT}

Objective Function

Early Multi-View

\text{Any PMARF}

Medial Surface Decoders

\left\| \nabla_{\hat{\mathbf q}} \sum_{i=1}^k a_{i,\ell} \mathbf u_{i,\ell} \right\|^2
\left\| \nabla_{\hat{\mathbf q}} \mathbf c_\ell \right\|^2 + \left\| \nabla_{\hat{\mathbf q}} r_\ell \right\|^2
\left\| \nabla_{\hat{\mathbf q}} \mathbf p_\ell \right\|^2
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\text{Any MARF}
\mathcal L_\text{mv} = h_\ell h_\ell^\text{GT}
\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2
  • Skip lots of double backprop
  • Multi-view consistent
    ray decoder

Weighted with softmin weights

\longleftarrow %asdasd
\mathrm{Any\ 4D\ Ray\ Field}
: \mathbf o = \mathbf p_\ell^\text{GT}
: \mathbf o = \mathbf p_\ell^\text{GT}
: \mathbf o = \mathbf p_\ell^\text{GT}

Objective Function

Early Multi-View

Truncated Regularization

\text{Any PMARF}
\text{(ours, early)}

Objective Function

Truncated Regularization

Early Multi-View

A brief recap

Objective Function

Truncated Regularization

Early Multi-View

\mathcal L_r
\mathcal L_\sigma

adds a constant positive pressure on atom radii (MAT maximality).

limits per-candidate area of influence, to specialize atoms to separate parts and handle discontinuities.

\ell

there is a single trivial solution
when no other loss apply.

Failure mode: 

Objective Function

Truncated Regularization

Early Multi-View

\{
0.3
\ell
\delta_{i,\ell} = \left[s_{i,\ell} \le 0.3\right]
\mathcal L_r
\mathcal L_\sigma

adds a constant positive pressure on atom radii (MAT maximality).

limits per-candidate area of influence, to specialize atoms to separate parts and handle discontinuities.

there is a single trivial solution
when no other loss apply.

Failure mode: 

The majority of predictions miss,
from a per-candidate perspective.

Objective Function

Truncated Regularization

Early Multi-View

\{
0.3
\ell
\delta_{i,\ell} = \left[s_{i,\ell} \le 0.3\right]
\mathcal L_r
\mathcal L_\sigma

adds a constant positive pressure on atom radii (MAT maximality).

limits per-candidate area of influence, to specialize atoms to separate parts and handle discontinuities.

there is a single trivial solution
when no other loss apply.

Failure mode: 

The majority of predictions miss,
from a per-candidate perspective.

  • Reconstruction quality
  • Scaling behavior
  • Ablations studies
  • Training and
    rendering speed
\times

3 architectures

2 setups

( + PRIF )

Results

\Bigg\} % 78932789123978123789123789
S^2\text{-parametric} % ljasdjasdjklasd
\mathbb R^2\text{-parametric} % ljasdjasdjklasd
\text{Non-parametric} % ljasdjasdjklasd
\bigg\{ % 78932789123978123789123789
\text{Our loss \& setup} % ljasdjasdjklasd
\text{Baseline loss \& setup} % ljasdjasdjklasd

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Depth-based
(non-medial)

\{

Medial baselines

\Big\{

Baseline
with our S²

\{

Our softmin & improved loss

\big\{

Chamfer Distance

Cosine Similarity

Medial Atom Normals

Differential Normals

Intersection over Union

Precision

Recall

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Results

Reconstruction

Table: Reconstruction scores. Average of 19 shapes.

Scaling

Same correctness, with improved capacity and recall.

Best overall.

\leftarrow %ljadsadsjklasd

Start to fall behind.

Has a "indecisive"
fit from the higher gradient flow.

\leftarrow %ljadsadsjklasd

Results

Reconstruction

Scaling

Renders

Results

Scaling

Renders

\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}
\text{PMARF}
\text{MARF}
\text{PRIF}
\text{Ours}
\text{Ours}
\text{Ours}

fails to generalize

multi-view stable

multi-view stable

multi-view stable

multi-view stable

multi-view stable

degeneracies near
overhangs

degeneracies near
overhangs

limited topology

limited topology

Results

Scaling

Renders

\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}
\text{PMARF}
\text{MARF}
\text{PRIF}
\text{Ours}
\text{Ours}
\text{Ours}

fails to generalize

multi-view stable

multi-view stable

multi-view stable

multi-view stable

multi-view stable

degeneracies near
overhangs

degeneracies near
overhangs

missing silhouette

missing silhouette

missing silhouette

limited topology

limited topology

Results

Scaling

Renders

\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}
\text{PMARF}
\text{MARF}
\text{PRIF}
\text{Ours}
\text{Ours}
\text{Ours}

fails to generalize

multi-view stable

multi-view stable

multi-view stable

multi-view stable

multi-view stable

degeneracies near
overhangs

degeneracies near
overhangs

missing silhouette

missing silhouette

missing silhouette

limited topology

limited topology

problems with inorganic shapes
that feature sharp angles

Renders

Reconstruction

Results

Reconstruction

Specialized

Specialized

Entangled atoms

Limited topology

Vanishing radius

\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}

Results

Reconstruction

Specialized

Specialized

Limited topology

Vanishing radius

\text{Non-parametric}
\mathbb R^2\text{-parametric}
S^2\text{-parametric}

A novel-view problem,
training views are unaffected.

Likely a ray-decoder interpolation issue.

\leftarrow


This hurdle may be fixed with:

 

Rebain D, Yazdani S, Yi KM, Tagliasacchi A. Neural fields as distributions: Signal processing beyond Euclidean space. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 Jun 16 (pp. 4274-4283). IEEE.

Entangled atoms

Results

Ablations

Table: Ablation studies

Table: Ablation studies

Results

Ablations

Table: Ablation studies

Table: Ablation studies

Results

Ablations

Table: Ablation studies

Table: Ablation studies

Sundt PB, Theoharis T. Towards multi-view consistency in neural ray fields using parametric medial surfaces. Computers & Graphics 2024;123:103991.

k-means init

Agnostic init

Data

Results

Ablations

Table: Ablation studies

Table: Ablation studies

\to

but is benefitial on PMARF baselines.

Reinforces characteristics observed earlier.

Detrimental on Our R² configuration,

Results

Ablations

Table: Ablation studies

Table: Ablation studies

argmin

softmin

data

Results

Ablations

Table: Ablation studies

Table: Ablation studies

⇒ Our Atom Displacement Loss requires Supervised Atom Selection.

our SAS mitigates the issue.

Penalizing normals early is unstable (MARF baseline also avoids it),

Results

Ablations

Table: Ablation studies

Table: Ablation studies

Results

Ablations

Table: Ablation studies

Table: Ablation studies

\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2

Results

Ablations

Table: Ablation studies

Table: Ablation studies

\Big\| \nabla_{\hat{\mathbf q}} \Bigg(
\Bigg)\Big\|^2

Results

Ablations

Table: Ablation studies

Table: Ablation studies

Strictly improved

Detrimental?

Primarily on mechanical shapes

Results

Ablations

Table: Ablation studies

Table: Ablation studies

Ray augmentation improves fidelity and recall.

And so does parameter domain augmentation.

Results

Ablations

Table: Ablation studies

Table: Ablation studies

And so does parameter domain augmentation.

Ray augmentation improves fidelity and recall.

Results

Ablations

Table: Ablation studies

\longrightarrow

Like MARF baseline

Hits only

\longrightarrow

Table: Ablation studies

Our 0.3 threshold strikes a sweetspot between regularizing all rays or regularizing hits only

Results

Ablations

Table: Ablation studies

Results

Ablations

Table: Ablation studies

Results

Ablations

Table: Ablation studies

Results

Ablations

Table: Ablation studies

Excessive

Insufficient

Sweetspot?

Decaying dropout

\tiny\{

Fixed dropout

\Big\{

Our decay from excessive to minimal avoids

local minima without sacrificing fidelity,

but rather improve it.

Results

Ablations

Table: Ablation studies

Cosine and Euclidean

Euclidean-only

Cosine-only

Stable, but smooth

Detailed, but unstable

A balance

Results

Runtime

Table: Timing Results

Ours have more losses, more gradients.

S² has k more activations and a vector normalization.

Results

Runtime

Table: Timing Results

Performance is reclaimed by our early multi-view loss.

Ours have more losses, more gradients.

S² has k more activations and a vector normalization.

Results

Runtime

Table: Timing Results

Parametric networks excel when compute bound.

Softmin blending is less divergent than Argmin lookups.

Results

Runtime

Table: Timing Results

Large but simple MLPs win out when not compute bound.

Parametric networks excel when compute bound.

Softmin blending is less divergent than Argmin lookups.

Thanks

  • A new double-covering parametrization

    • Expands representation capacity

  • Probabilistic blending

    • Improves gradient flow and scaling

  • New loss functions

    • Increases fidelity, stability and recall

  • Faster and improved multi-view stabilization

    • Reduce training times while improving multi-view stability

  • Coarse-to-fine training schedule

    • Improves convergence and multi-view generalization

Our Contributions Summaried