These slides:
https://s.ntnu.no/irobot
F for fullscreen
SPACE to advance
ESC to zoom out
Slides progress downward,
then to next column
Single-View 3D Shape Completion
for Robotic Grasping of Objects
via Deep Neural Fields
A MsC by
Peder Bergebakken Sundt
Motivation
Motivation
?
Panda Emika 7-DoF robot arm
With
Intel Realsense 3D vision
Single-View 3D Shape Completion
for Robotic Grasping of Objects
via Deep Neural Fields
A MsC by
Peder Bergebakken Sundt
Affected by
point order,
no topology.
Scales poorly
Either limited
topologically or
self-intersecting
... all map poorly to neural networks!
Explicit 3D shape representations
Points
Voxels
Meshes
A new concept, first explored in 2019
"We have many names for the things we love:"
- [Deep] Neural Fields
- Coordinate-based Neural Networks
- Implicit Representation Network
Park JJ, Florence P, Straub J, Newcombe R, Lovegrove S. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA: IEEE; 2019, p. 165–74. https://doi.org/10.1109/CVPR.2019.00025.
"We have many names for the things we love:"
- [Deep] Neural Fields
- Coordinate-based Neural Networks
- Implicit Representation Network
Park JJ, Florence P, Straub J, Newcombe R, Lovegrove S. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA: IEEE; 2019, p. 165–74. https://doi.org/10.1109/CVPR.2019.00025.
A new concept, first explored in 2019
Query
coordinates
Value at
coordinate
An example:
2D RGB Field
\Phi _{\theta } \space :\space \mathbb{R} ^{2} \rightarrow \mathbb{R} ^{3}
Trained on (x,y,r,g,b) tuples
3D Implicit Neural Representations have
several benefits:
- They represent shapes as continuous functions.
- Continuous -> arbitrary reconstruction resolution
- Perfect for partial data, enables single view shape completion
- The network size not tied to the resolution of 3D models.
- Networks only scale with shape complexity, not size!
- Seamlessly allow for learning latent spaces of functions.
- Enables shape completion drawing from learned priors.
f(\bold{x} )=y\space :\mathbb{R} ^3\rightarrow \mathbb{R}
f(\cdot )=0
\|\nabla _{\bold{x} } f\|_{2} =1
Signed Distance Functions (SDF)
Implicit Isosurface
Metric Constraint
Moving to 3D...
f(\bold{x} )=y\space :\mathbb{R} ^3\rightarrow \mathbb{R}
Signed Distance Functions (SDF)
\Phi_\theta(\bold{x} )=y\space :\mathbb{R} ^3\rightarrow \mathbb{R}
Signed Distance Functions (SDF)
Train it on (x, y) tuples
Signed Distance Functions (SDF)
Marching Cubes
+
\Phi_\theta(\bold{x} )=y\space :\mathbb{R} ^3\rightarrow \mathbb{R}
Train it on (x, y) tuples
3D models provided by
The YCB dataset
Training data
Signed Distances
Gradients
Positive SDF
Negative SDF
⬤
⬤
Free-space
Near-surface
⬤
⬤
Training data
Mesh
\Rightarrow
f(\bold x)
\nabla_{\bold x}f(\bold x)
Gradients
Positive SDF
Negative SDF
⬤
⬤
Free-space
Near-surface
⬤
⬤
\mathcal{L} (y,\hat{y} )=
|y-\hat{y} |
+\ \ \ (1 - \langle
\nabla_{\bold x}y,
\nabla_{\bold x}\hat y
\rangle)
(Cosine similarity)
Mesh
Loss
Signed Distances
Requires us to compute
the derivative of the network itself
ReLU-based
Sinusoidal
\max (0, x)
\sin (x)
- Piecewise linear
- Second derivative is zero!
Activation Function
SIREN
\sin (\omega_0 x)
- Piecewise linear
- Second derivative is zero!
- Can represent complex signals
- The derivative of a SIREN
is another SIREN!
Activation Function
ReLU-based
\max (0, x)
Ground-
Truth
ReLU
ReLU
w/gradients
SIREN
w/gradients
ReLU-based networks
SIREN networks
Learning a useful space of
prior shapes
Coded decoders
Embedding more than one shape
Learning latent spaces
Auto-encoders map poorly
to learning implicit functions...
... and treat the latent vectors as learnable parameters!
-> Just skip the encoder!
Keep a database
of codes per object
\sum_y \mathcal{L}(y, \hat{y})
Add a regularizing cost to each latent code in the auto-decoder database:
+
\lambda\frac{1}{|\Omega |} \sum_{i\in \Omega }^{}{\|\bold{z} _{i} \|_2 }
Problem: Learned latent vectors drift apart!
Pulls each code (z) towards 0, and
incentivise a spherical distribution
\Rightarrow
(Reconstruction loss)
(Latent code regularization)
and generalization.
Pose?
\approx
\text{SDF}_i(\bold x)
\Phi_\theta(\bold x, \bold{z}_i)
We need:
- Rotation (R)
- Scale (s)
- Translation (t)
Pose
\approx
s\cdot \text{SDF}_i (\bold{R} (s\cdot \bold{x} )+\bold{t} )
\Phi _{\theta } (\bold{x} ,\bold{R} ,s,\bold{t} , \bold{z}_i)
We need:
- Rotation (R)
- Scale (s)
- Translation (t)
Then we train with random transformations,
and discover the pose via gradient decent at test time
Single-view data
We sample SDF points from two distributions:
- Near-surface samples: For surface details
- Ambient space samples: Improves generalization
The these two distributions
are balanced 90% / 10%
Single-view data
Hit
Miss
Camera
⬤
⬤
⬤
Positive
Negative
⬤
⬤
Sampling pipeline
Single-view data
T-SNE of learned latent vectors
Shape completion:
- Start near the mean class vector (classifier needed)
- Further optimize the vector w.r.t. partial observation
using gradient decent
(start)
(end)
A Search
BigBIRD Scanner
Real data?
\Rightarrow
!
From the YCB data- and object set
BigBIRD Scanner
Color
Depth
\Rightarrow
!
Positive SDF
Negative SDF
Unit sphere
Reconstruction
volume
⬤
⬤
⬤
⬤
Bounding box?
Ground-
Truth
To be solved...
Testing Occlusions
Questions?
Tips?
https://s.ntnu.no/irobot
This is is the basis for a paper in the works.
Irobot
By pbsds
