Reconstructing 3D geometry from Neural Radiance Field (NeRF)
Tao Pang
- Hold right mouse button: pan.
- Mouse wheel: zoom.
What is NeRF?
Training data: posed images
(x,y,z): point in the 3D volume containing the scene.
d: direction at which the camera looks at the point.
(xi,yi,zi)
(x_i, y_i, z_i)
d
d
σ=fσ,θ(x,y,z)
\sigma = f_{\sigma, \theta}(x, y, z)
Goal: render new views.
Intensity:
Learns
Color(RGB):
C(x,y,z,d)=fRGB,θ(x,y,z,d)
C(x, y, z, d) = f_{RGB, \theta}(x, y, z, d)
Volumetric rendering
The color of a pixel is the weighted sum of the colors along the ray.
Cpixel=∑iwiC(xi,yi,zi,d)
C_{\text{pixel}} = \sum_i w_i C(x_i, y_i, z_i, d)
The weight wi of a point is high if:
- it has high intensity σi (α=1), and
- it is not blocked by another high intensity point.
Depth can also be estimated from NeRF:
NeRF:
zpixel=∑iwizi
z_{\text{pixel}} = \sum_i w_i z_i
(similar to α)
w0=1
w_0 = 1
w1=0
w_1 = 0
(x0,y0,z0)
(x_0, y_0, z_0)
(x1,y1,z1)
(x_1, y_1, z_1)
camera ray
Pixel (a)
z=z0
z = z_0
camera ray
Pixel (b)
w0=0.5
w_0 = 0.5
w1=0.5
w_1 = 0.5
(x0,y0,z0)
(x_0, y_0, z_0)
(x1,y1,z1)
(x_1, y_1, z_1)
z=(z0+z1)/2
z = (z_0 + z_1) / 2
Color depends on direction.
- To capture effects such as specular reflection, the direction d in C(x,y,z,d) is essential.
Pre-trained
C(x,y,z)
C(x,y,z,d)
Accurate geometry depends on direction too!
Pre-trained
- It turns out that making color a function of direction d also improves the quality of learned scene geometry.
- Geometry can be reconstructed from RGBD images synthesized for different views. (The authors' website took a different approach)
C(x,y,z)
C(x,y,z,d)
Position-only formulation learns color variation by adding extraneous geometry!
Brighter
darker
Pixel (a)
Pixel (b)
Mesh from TSDF volume
Pre-trained
C(x,y,z)
C(x,y,z,d)
Mesh from Marching cubes
- Bullet One
- Bullet Two
- Bullet Three
Pre-trained
Position only
Position and direction
Backup slides
Pixel a
Pixel b
Title Text
Title Text
Title Text
- Bullet One
- Bullet Two
- Bullet Three
Reconstructing 3D geometry from Neural Radiance Field (NeRF) Tao Pang Hold right mouse button: pan. Mouse wheel: zoom.