Tao Pang

- Hold right mouse button: pan.
- Mouse wheel: zoom.

Training data: posed images

\((x, y, z)\): point in the 3D volume containing the scene.

\(d\): direction at which the camera looks at the point.

(x_i, y_i, z_i)

d

\sigma = f_{\sigma, \theta}(x, y, z)

Goal: render new views.

Intensity:

Learns

Color(RGB):

C(x, y, z, d) = f_{RGB, \theta}(x, y, z, d)

Volumetric rendering

The color of a pixel is the weighted sum of the colors along the ray.

C_{\text{pixel}} = \sum_i w_i C(x_i, y_i, z_i, d)

The weight \(w_i\) of a point is high if:

- it has high intensity \(\sigma_i\) (\(\alpha = 1\)), and
- it is not blocked by another high intensity point.

Depth can also be estimated from NeRF:

**NeRF:**

z_{\text{pixel}} = \sum_i w_i z_i

(similar to \(\alpha\))

w_0 = 1

w_1 = 0

(x_0, y_0, z_0)

(x_1, y_1, z_1)

camera ray

Pixel (a)

z = z_0

camera ray

Pixel (b)

w_0 = 0.5

w_1 = 0.5

(x_0, y_0, z_0)

(x_1, y_1, z_1)

z = (z_0 + z_1) / 2

- To capture effects such as specular reflection, the direction \(d\) in \(C(x, y, z, d)\) is essential.

Pre-trained

\(C(x, y, z)\)

\(C(x, y, z, d)\)

Pre-trained

- It turns out that making color a function of direction \(d\) also improves the quality of learned scene geometry.
- Geometry can be reconstructed from RGBD images synthesized for different views. (The authors' website took a different approach)

\(C(x, y, z)\)

\(C(x, y, z, d)\)

Brighter

darker

Pixel (a)

Pixel (b)

Pre-trained

\(C(x, y, z)\)

\(C(x, y, z, d)\)

- Bullet One
- Bullet Two
- Bullet Three

Pre-trained

Position only

Position and direction

Pixel a

Pixel b

- Bullet One
- Bullet Two
- Bullet Three