# Reconstructing 3D geometry from Neural Radiance Field (NeRF)

Tao Pang

• Hold right mouse button: pan.
• Mouse wheel: zoom.

## What is NeRF?

Training data: posed images

$$(x, y, z)$$: point in the 3D volume containing the scene.

$$d$$: direction at which the camera looks at the point.

(x_i, y_i, z_i)
d
\sigma = f_{\sigma, \theta}(x, y, z)

Goal: render new views.

Intensity:

Learns

Color(RGB):

C(x, y, z, d) = f_{RGB, \theta}(x, y, z, d)

Volumetric rendering

The color of a pixel is the weighted sum of the colors along the ray.

C_{\text{pixel}} = \sum_i w_i C(x_i, y_i, z_i, d)

The weight $$w_i$$ of a point is high if:

• it has high intensity $$\sigma_i$$ ($$\alpha = 1$$), and
• it is not blocked by another high intensity point.

Depth can also be estimated from NeRF:

NeRF:

z_{\text{pixel}} = \sum_i w_i z_i

(similar to $$\alpha$$)

w_0 = 1
w_1 = 0
(x_0, y_0, z_0)
(x_1, y_1, z_1)

camera ray

Pixel (a)

z = z_0

camera ray

Pixel (b)

w_0 = 0.5
w_1 = 0.5
(x_0, y_0, z_0)
(x_1, y_1, z_1)
z = (z_0 + z_1) / 2

## Color depends on direction.

• To capture effects such as specular reflection, the direction $$d$$ in $$C(x, y, z, d)$$ is essential.

Pre-trained

$$C(x, y, z)$$

$$C(x, y, z, d)$$

## Accurate geometry depends on direction too!

Pre-trained

• It turns out that making color a function of direction $$d$$ also improves the quality of learned scene geometry.
• Geometry can be reconstructed from RGBD images synthesized for different views. (The authors' website took a different approach)

$$C(x, y, z)$$

$$C(x, y, z, d)$$

Brighter

darker

Pixel (a)

Pixel (b)

## Mesh from TSDF volume

Pre-trained

$$C(x, y, z)$$

$$C(x, y, z, d)$$

## Mesh from Marching cubes

• Bullet One
• Bullet Two
• Bullet Three

Pre-trained

Position only

Position and direction

Pixel a

Pixel b

• Bullet One
• Bullet Two
• Bullet Three