Erik Jansson
WASP and KAW postdoctoral fellow in the Cambridge Image Analysis Group and CRA at Wolfson College.
Geometric shape matching
slides.com/erikjansson
About Me!
PhD (project researcher) from Chalmers in Gothenburg
Same department where Mike did his postdoc
Me
Recent PhD from Chalmers in Gothenburg: Same department where Mike did his postdoc
Shape analysis and matching problems
Structure-preserving numerics for geometric mechanics
With Mike
About Me!
Elliptic SPDEs on manifolds
for sampling of random fields
Recent PhD from Chalmers in Gothenburg: Same department where Mike did his postdoc
Shape analysis and matching problems
Structure-preserving numerics for geometric mechanics
Elliptic SPDEs on manifolds
for sampling of random fields
With Mike
About Me!
Joint work with:
- Klas Modin (Chalmers/University of Gothenburg)
- Ozan Öktem (Royal Institute of Technology, Stockholm)
- Jonathan Krook (Royal Institute of Technology, Stockholm)
Talk based on:
asdasdasd
But what is a shape?
Shape matching
Point cloud on torus
Function on \([0,1]^2\)
Density on \([0,1]\)
Anything a diffeomorphism can act on!
What is a shape?
(Diffeomorphism = smooth mapping that goes from domain to itself and has a smooth inverse)
A warp is determined by a time-dependent vector field \(v\)
A vector field generates a curve of diffeomorphisms \(\gamma: [0,1] \mapsto G = \operatorname{Diff}(M)\) by
\(\dot \gamma(t) = \nu(t, \gamma(t)), t \in [0,1], \gamma(0) = \operatorname{Id} \)
The optimal vector field is a minimizer of an energy!
What is an optimal warp?
The energy functional
\underbrace{\mathcal{D}(\gamma(1).A,B)}_{\text{Matching energy}}+\underbrace{d_G(\gamma,e)}_{\text{Regularization}}
Canonical choice! Use right-invariant metric
\underbrace{\mathcal{D}(\gamma(1).A,B)}_{\text{Matching energy}}+\underbrace{\int_0^1 \int_M L\nu(t) \cdot \nu(t)\mathrm{d}x\mathrm{d}t}_{\text{Regularization}}
The energy functional
\underbrace{\mathcal{D}(\gamma(1).A,B)}_{\text{Matching energy}}+\underbrace{\int_0^1 \int_M L\nu(t) \cdot \nu(t)\mathrm{d}x\mathrm{d}t}_{\text{Regularization}}
The energy functional
Expensive and difficult optimization problem.
However: Can be reduced to a dynamic (= differential equation) formulation!
\underbrace{\mathcal{D}(\gamma(1).A,B)}_{\text{Matching energy}}
+\hspace{2.4pt}\underbrace{\int_0^1 \int_M L\nu(t) \cdot \nu(t)\mathrm{d}x\mathrm{d}t}_{\text{Regularization}}
Only depends on final value of \(\gamma(t)\)
Changes depending on application
The energy functional
Expensive and difficult optimization problem.
However: Can be reduced to a dynamic (= differential equation) formulation!
Only depends on final value of \(\gamma(t)\)
Depends on whole path
Changes depending on application
Key question: How does \(\nu\) evolve?
(Answer: Use calculus of variations, but with which energy?)
\underbrace{\mathcal{D}(\gamma(1).A,B)}_{\text{Matching energy}}
+\hspace{2.28pt}\underbrace{\int_0^1 \int_M L\nu(t) \cdot \nu(t)\mathrm{d}x\mathrm{d}t}_{\text{Regularization}}
The energy functional
Expensive and difficult optimization problem.
However: Can be reduced to a dynamic (= differential equation) formulation!
Evolution of vector field does not care about matching!
Cannonball does not care about target
Shooting analogy
Use the Lagrangian \({\int_0^T \int_M L\nu(t) \cdot \nu(t)\mathrm{d}x\mathrm{d}t}_{}\)
Calculus of variations gives:
~\dot m = \operatorname{ad}_\nu^* m~ \\
m = L\nu
The EPDiff equation
Matching by shooting
A picture says more...
~\dot m = \operatorname{ad}_\nu^* m~ \\
m = L\nu
Shape analysis made simpler
The ingredients you really need:
A Lie group of deformations \(G\) acting on a set of shapes \(V\) (metric space)
\mathcal{D}(\gamma(1).A,B)
Flexible choice
\langle \dot \gamma, \dot \gamma\rangle_\gamma = (\dot \gamma \circ \gamma^{-1}, \dot \gamma \circ \gamma^{-1})_{\mathfrak{g}}
Right-invariant metric (not so flexible)
Traditional application: medical image analysis:
Compute warp
- Energy of deformation = distance between images
- Allows for statistical analysis, clustering etc
Intuition by Movie
Compute warp
Towards my application:
indirect matching
Here, \(A\) and \(B\) are assumed to be elements in the same space.
What if we cannot directly observe B?
Indirect matching
Include a forward operator \(\mathcal{F}\colon V \to W\)
\underbrace{\mathcal{D}_W(\mathcal{F}(\gamma(1).A),B)}_{\text{Matching energy}}+\underbrace{\int_0^1 \int_M L\nu(t) \cdot \nu(t)\mathrm{d}x\mathrm{d}t}_{\text{Regularization}}
Initial shape: \(A \in V\)
Target shape: \(B \in W\)
Doesn't affect mathematical framework!
"Reconstructs" the \(\gamma(1).A\) that best would map to \(B\)
Shape for Cryo-EM
Single-particle Homogeneous Cryo-EM in 30 seconds (by a maths person)
Electron
microscope
Low dose - low SnR
Many images
Shape for Cryo-EM
Proteins in 30 seconds (by a mathematician)
In this work: forget about everything but the \(C_\alpha\)s
Relative positions
Shape for Cryo-EM
Goal:
Given noisy images
Reconstruct protein conformation
Idea: Use shape to deform a "prior", i.e., same protein in other conformation
Shape for Cryo-EM
Shape space: space of relative positions \(V = \mathbb{R}^{3N}\)
Data space - 2D images, \(L^2(\mathbb{R}^2)\) (very noisy!)
However: We want rigid deformations, so \(G = \operatorname{SO}(3)^N\), and \(\operatorname{dim}(G) \ll \operatorname{dim}(\operatorname{Diff}(M)) = \infty\)
Todo: Construct forward model, decide on energy, compute gradients, set up optimization routine
Shape for Cryo-EM
Forward model building
Shape for Cryo-EM
Forward model building
Shape for Cryo-EM
Similarity score is clear. For now: use x-y-z projections (3 images)
\mathcal E(\nu) = \frac{1}{2} \sum_{i \in \{1,2,3\}}\left\| \mathcal{F}_i(\Phi(\gamma(1),w)) - y_i \right\|^2_{L^2(\mathbb R^2)} + \lambda \int_0^1 \operatorname{Tr}(\nu^T\mathbb{I}\nu)\\
\dot \gamma = \nu \gamma, \gamma(0) = e
Shape for Cryo-EM
Similarity score is clear. For now: use x-y-z projections (3 images)
\dot m = [m,\nu], m = \mathbb{I} \nu \\
\dot \gamma = \nu \gamma, \gamma(0) = e
- Guess \(\nu(0)\) (or path sampled at discrete points)
- Integrate
- Evaluate \(\mathcal{E}\) and compute \(\nabla_{\nu(0)} \mathcal{E}\)
- Update \(\nu(0)\) with favorite algorithm (A few GD steps and then L-BFGS-B)
Gradient is available (even easy to compute)
(but ugly)
Shape for Cryo-EM
Gradient is available
Shape for Cryo-EM
Deformation
What we want
What we see
Where we start
Shape for Cryo-EM
Oh no
but....
Shape for Cryo-EM
x
y
z
Deformed
Target
Template
Shape for Cryo-EM
Shape for Cryo-EM
Hope!
Turn up number of images used
3 \rightarrow 300
Decent in the directions we have projections
Shape for Cryo-EM
Deformation
What we want
What we see
Where we start
Shape for Cryo-EM
Deformation path
Note: Deformation makes no physical sense
Shape for Cryo-EM
Increase noise
Increase #projections
Quantative measure: Procrustes score
Procrustes analysis
Two point clouds \(\mathbf x = (x_i)_{i=1}^K\) and \(\mathbf y = (y_i)_{i=1}^K\)
Rotate, reflect, scale and translate ("superimpose") \(\mathbf x\) to be as close in mean-square sense to \(\mathbf y\) as possible
As close as we can get: Procrustes distance (early statistical shape analysis measure)
- Global pose: We assume it is known
- Side chains: We cut away the side chains
- Faster methods: Even if we have the gradient, not fast!
Shape analysis seems promising
Shape for Cryo-EM
Shape for Gaussian random fields?
Elliptic SPDEs on manifolds
for sampling of random fields
Shape analysis and matching problems
Shapes analysis gives a way to compute diffeomorphisms!
Question: Can learning deformations of random fields be seen as an indirect matching problem?
Z: \text{stationary}\\
\gamma \in \operatorname{Diff}(\mathbb T^2) \\
Z\circ \gamma^{-1}, \text{non-stationary}
Modern ML-based methods allows for the learning of \(\gamma\) with guarantees!
Shape for Gaussian random fields?
Shape for Gaussian random fields?
Z: \text{stationary}\\
\gamma \in \operatorname{Diff}(\mathbb T^2) \\
Z\circ \gamma^{-1}, \text{non-stationary}
Question: What is a good matching functional to compare your deformed model to observational data?
Question: What is suitable data?
- Group: \(G = \operatorname{Diff}(\mathbb{T}^2)\),
- Shape space: \(V = \text{set of random fields}\)
- Data space: \(W = \text{observations of non-stationary field}\)
In conclusion
- Is a framework that allows one to find optimal deformations
- Has applications in medical image analysis, protein conformation reconstruction
- Has seen recent ML-based methods for solving expensive computations, opens new applications: Maybe for GRFs?
Shape analysis...
Fontainebleau
By Erik Jansson
