Tomancak Lab, Group Meeting
\text{Tomancak Lab, Group Meeting}
Dec 15, 2020
\text{Dec 15, 2020}
Early development of an organism
\text{Early development of an organism}
Nuclei
\text{Nuclei}
Membrane
\text{Membrane}
Light sheet Imaging
\text{Light sheet Imaging}
Time-lapse recording of Embryo 1
\text{Time-lapse recording of Embryo 1}
Early development of an organism
\text{Early development of an organism}
Nuclei
\text{Nuclei}
Membrane
\text{Membrane}
Light sheet Imaging
\text{Light sheet Imaging}
Time-lapse recording of Embryo 1
\text{Time-lapse recording of Embryo 1}
Confocal Imaging
\text{Confocal Imaging}
Embryo 2
\text{Embryo 2}
Embryo ...
\text{Embryo ...}
Embryo N
\text{Embryo N}
Nuclei
\text{Nuclei}
Gene
\text{Gene}
Membrane
\text{Membrane}
Our model organism
\text{Our model organism}
Platynereis dumerillii
\text{\textit{Platynereis dumerillii}}
Light Sheet Imaging of Live Embryo
\text{Light Sheet Imaging of Live Embryo}
Confocal Images of specimens
\text{Confocal Images of specimens}
Adult worm
\text{Adult worm}
Embryo @ 16 hpf*
\text{Embryo @ 16 hpf*}
hpf*: hours post fertilization
\text{hpf*: hours post fertilization}
at 16 hpf using ISH*
\text{at 16 hpf using ISH*}
ISH*: in situ hybridization
\text{ISH*: \textit{in situ} hybridization}
Adult worm
\text{Adult worm}
Our problem statement
\text{Our problem statement}
Our problem statement
\text{Our problem statement}
Intramodal Registration
\text{Intramodal Registration}
Intermodal Registration
\text{Intermodal Registration}
* tp: time point
\text{* tp: time point}
Challenges in identifying nuclei correspondence
\text{Challenges in identifying nuclei correspondence}
Live Embryo:
\text{Live Embryo:}
Nuclei
\text{Nuclei}
Cell nuclei positions were fitted to an
\text{Cell nuclei positions were fitted to an}
ellipsoid
\text{ellipsoid}
situ
situ
Nuclei
\text{Nuclei}
Cell nuclei positions were fitted to an
\text{Cell nuclei positions were fitted to an}
ellipsoid
\text{ellipsoid}
In−
In-
specimen:
\text{specimen:}
How to identify cell-to-cell correspondences?
\text{How to identify cell-to-cell correspondences?}
Morphology
\text{Morphology}
Appearance
\text{Appearance}
Geometric Arrangement
\text{Geometric Arrangement}
Use distinctively shaped nuclei to align and match
\text{Use distinctively shaped nuclei to align and match}
Use neighborhood as a feature to identify corrsponding nuclei
\text{Use neighborhood as a feature to identify corrsponding nuclei}
Our approach
\text{Our approach}
σ=6
\sigma=6
σ=7
\sigma=7
σ=5
\sigma=5
σ=8
\sigma=8
1. Evaluate ∇2Gσ⊛I
\text{1. Evaluate }\nabla^{2} G_{\sigma} \circledast \mathbf{I}
Our approach
\text{Our approach}
σ=6
\sigma=6
σ=7
\sigma=7
σ=5
\sigma=5
σ=8
\sigma=8
1. Evaluate ∇2Gσ⊛I
\text{1. Evaluate }\nabla^{2} G_{\sigma} \circledast \mathbf{I}
2. Find local minima
\text{2. Find local minima }
in x, y, z, σ space
\text{in x, y, z, $\sigma$ space}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Belongie et al.“Shape matching and object recognition using shape contexts"
\text{Belongie et al.``Shape matching and object recognition using shape contexts"}
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
\text{IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002}
Z’
\text{Z'}
Our approach
\text{Our approach}
Y’
\text{Y'}
Z’
\text{Z'}
Our approach
\text{Our approach}
Z’
\text{Z'}
Y’
\text{Y'}
X’
\text{X'}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
3
3
11
11
15
15
6
6
12
12
4
4
0
0
9
9
0
0
0
0
22
22
3
3
5
5
17
17
11
11
19
19
Bins:
\text{Bins: }
1
\text{1}
2
\text{2}
3
\text{3}
4
\text{4}
...
\text{...}
...
\text{...}
K
\text{K}
6
6
6
6
6
6
0
0
0
0
0
0
2
2
2
2
4
4
4
4
4
4
4
4
20
20
18
18
14
14
13
13
16
16
16
16
9
9
19
19
Nucleus 1 @ (12, 34, 89)
\text{Nucleus 1 @ (12, 34, 89)}
Nucleus 2 @ (26, 49, 92)
\text{Nucleus 2 @ (26, 49, 92)}
Nucleus 3 @ (68, 93, 21)
\text{Nucleus 3 @ (68, 93, 21)}
Nucleus 4 @ (81, 32, 15)
\text{Nucleus 4 @ (81, 32, 15)}
Nucleus ... @ (..., ..., ...)
\text{Nucleus ... @ (..., ..., ...)}
Our approach
\text{Our approach}
0.03
0.03
0.1
0.1
0.1
0.1
0.06
0.06
0.1
0.1
0.01
0.01
0
0
0.03
0.03
0
0
0
0
0.2
0.2
0.01
0.01
0.02
0.02
0.2
0.2
0.1
0.1
0.2
0.2
Bins:
\text{Bins: }
1
\text{1}
2
\text{2}
3
\text{3}
4
\text{4}
...
\text{...}
...
\text{...}
K
\text{K}
0.1
0.1
0.02
0.02
0.02
0.02
0
0
0
0
0
0
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.2
0.2
0.2
0.2
0.03
0.03
0.2
0.2
Nucleus 1 @ (12, 34, 89)
\text{Nucleus 1 @ (12, 34, 89)}
Nucleus 2 @ (26, 49, 92)
\text{Nucleus 2 @ (26, 49, 92)}
Nucleus 3 @ (68, 93, 21)
\text{Nucleus 3 @ (68, 93, 21)}
Nucleus 4 @ (81, 32, 15)
\text{Nucleus 4 @ (81, 32, 15)}
Nucleus ... @ (..., ..., ...)
\text{Nucleus ... @ (..., ..., ...)}
∑k=1k=Kh(k)=1
\sum^{k=K}_{k=1} h(k) =1
Our approach
\text{Our approach}
...
\text{...}
Cij:=C(pi,qj)
C_{ij} := C(p_{i}, q_{j})
1
\text{1}
1’
\text{1'}
2
\text{2}
2’
\text{2'}
3
\text{3}
3’
\text{3'}
4
\text{4}
4’
\text{4'}
2
\text{2}
...
\text{...}
p1
p_{1}
p2
p_{2}
p3
p_{3}
p4
p_{4}
p...
p_{...}
pM
p_{M}
q1
q_{1}
q2
q_{2}
q3
q_{3}
q4
q_{4}
qM
q_{M}
q...
q_{...}
=21∑k=1Khi(k)+hj(k)(hi(k)−hj(k))2
= \frac{1}{2} \sum_{k=1}^{K} \frac{(h_{i}(k) -h_{j}(k))^{2}}{h_{i}(k) + h_{j}(k)}
0.24
\text{0.24}
0.19
\text{0.19}
0.21
\text{0.21}
0.12
\text{0.12}
0.15
\text{0.15}
0.18
\text{0.18}
0.06
\text{0.06}
0.23
\text{0.23}
0.16
\text{0.16}
0.07
\text{0.07}
0.16
\text{0.16}
0.13
\text{0.13}
0.29
\text{0.29}
0.15
\text{0.15}
0.28
\text{0.28}
0.20
\text{0.20}
0.11
\text{0.11}
0.14
\text{0.14}
0.18
\text{0.18}
0.13
\text{0.13}
0.24
\text{0.24}
0.26
\text{0.26}
0.12
\text{0.12}
0.09
\text{0.09}
0.14
\text{0.14}
0.17
\text{0.17}
0.23
\text{0.23}
0.26
\text{0.26}
0.19
\text{0.19}
0.22
\text{0.22}
0.14
\text{0.14}
0.16
\text{0.16}
0.23
\text{0.23}
0.21
\text{0.21}
0.11
\text{0.11}
0.09
\text{0.09}
Our approach
\text{Our approach}
Apply RANSAC to estimate
\textit{Apply RANSAC to estimate}
coarse transform using inliers
\textit{coarse transform using inliers}
1x′1y′1z′2x′2y′2z′3x′3y′3z′4x′4y′4z′=A×1x1y1z2x2y2z3x3y3z4x4y4z
\begin{bmatrix}
1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\
1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\
1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\
\end{bmatrix}=
A \times
\begin{bmatrix}
1_{x} & 2_{x} & 3_{x} & 4_{x}\\
1_{y} & 2_{y} & 3_{y} & 4_{y}\\
1_{z} & 2_{z} & 3_{z} & 4_{z}\\
\end{bmatrix}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Our approach
\text{Our approach}
“Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.”
\text{``Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.''}
Fischer and Bolles, Communications of the ACM (1981)
\text{Fischer and Bolles, Communications of the ACM (1981)}
Count Inliers
\text{Count Inliers}
Apply RANSAC to estimate
\textit{Apply RANSAC to estimate}
coarse transform using inliers
\textit{coarse transform using inliers}
1x′1y′1z′2x′2y′2z′3x′3y′3z′4x′4y′4z′=A×1x1y1z2x2y2z3x3y3z4x4y4z
\begin{bmatrix}
1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\
1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\
1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\
\end{bmatrix}=
A \times
\begin{bmatrix}
1_{x} & 2_{x} & 3_{x} & 4_{x}\\
1_{y} & 2_{y} & 3_{y} & 4_{y}\\
1_{z} & 2_{z} & 3_{z} & 4_{z}\\
\end{bmatrix}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Our approach
\text{Our approach}
Repeat
\text{Repeat}
Count Inliers
\text{Count Inliers}
Apply RANSAC to estimate
\textit{Apply RANSAC to estimate}
coarse transform using inliers
\textit{coarse transform using inliers}
1x′1y′1z′2x′2y′2z′3x′3y′3z′4x′4y′4z′=A×1x1y1z2x2y2z3x3y3z4x4y4z
\begin{bmatrix}
1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\
1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\
1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\
\end{bmatrix}=
A \times
\begin{bmatrix}
1_{x} & 2_{x} & 3_{x} & 4_{x}\\
1_{y} & 2_{y} & 3_{y} & 4_{y}\\
1_{z} & 2_{z} & 3_{z} & 4_{z}\\
\end{bmatrix}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Our approach
\text{Our approach}
...
\text{...}
...
\text{...}
p1∗
p^{*}_{1}
p2∗
p^{*}_{2}
p3∗
p^{*}_{3}
p4∗
p^{*}_{4}
p...∗
p^{*}_{...}
pM∗
p^{*}_{M}
q1
q_{1}
q2
q_{2}
q3
q_{3}
q4
q_{4}
qN
q_{N}
q...
q_{...}
10.1
\text{10.1}
2.6
\text{2.6}
11.3
\text{11.3}
89.7
\text{89.7}
14.2
\text{14.2}
90.8
\text{90.8}
64.2
\text{64.2}
23.5
\text{23.5}
16.9
\text{16.9}
70.8
\text{70.8}
16.7
\text{16.7}
32.4
\text{32.4}
9.7
\text{9.7}
15.4
\text{15.4}
28.3
\text{28.3}
20.1
\text{20.1}
11.6
\text{11.6}
14.9
\text{14.9}
18.1
\text{18.1}
3.7
\text{3.7}
24.1
\text{24.1}
36.4
\text{36.4}
12.2
\text{12.2}
9.5
\text{9.5}
14.2
\text{14.2}
17.5
\text{17.5}
23.45
\text{23.45}
6.7
\text{6.7}
19.9
\text{19.9}
22.1
\text{22.1}
14.8
\text{14.8}
16.4
\text{16.4}
63.7
\text{63.7}
21.5
\text{21.5}
81.1
\text{81.1}
9.6
\text{9.6}
Cij:=C(pi∗,qj)
C_{ij} := C(p^{*}_{i}, q_{j})
=(pi,x∗−qj,x)2+(pi,y∗−qj,y)2+(pi,z∗−qj,z)2
= \sqrt{\left( p^{*}_{i,x} -q_{j,x} \right)^{2} + \left( p^{*}_{i,y} -q_{j,y} \right)^{2} + \left( p^{*}_{i,z} -q_{j,z} \right)^{2}}
1
\text{1}
5’
\text{5'}
1’. 4’
\text{1'. 4'}
3
\text{3}
2’
\text{2'}
4
\text{4}
6’
\text{6'}
2
\text{2}
3’
\text{3'}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Apply ICP to obtain a tighter fit
\textit{Apply ICP to obtain a tighter fit}
Our approach
\text{Our approach}
Apply ICP to obtain a tighter fit
\textit{Apply ICP to obtain a tighter fit}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Estimate Affine Transform
\text{Estimate Affine Transform}
1x′1y′1z′2x′2y′2z′………Mx′My′Mz′=A×1x1y1z2x2y2z………MxMyMz
\begin{bmatrix}
1^{'}_{x} & 2^{'}_{x} & \ldots & M^{'}_{x}\\
1^{'}_{y} & 2^{'}_{y} & \ldots & M^{'}_{y}\\
1^{'}_{z} & 2^{'}_{z} & \ldots & M^{'}_{z}\\
\end{bmatrix}=
A \times
\begin{bmatrix}
1_{x} & 2_{x} & \ldots & M_{x}\\
1_{y} & 2_{y} &\ldots & M_{y}\\
1_{z} & 2_{z} & \ldots & M_{z}\\
\end{bmatrix}
Our approach
\text{Our approach}
Apply ICP to obtain a tighter fit
\textit{Apply ICP to obtain a tighter fit}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Estimate Affine Transform
\text{Estimate Affine Transform}
Repeat upto Convergence
\text{Repeat upto Convergence}
Our approach
\text{Our approach}
...
\text{...}
...
\text{...}
p1∗
p^{*}_{1}
p2∗
p^{*}_{2}
p3∗
p^{*}_{3}
p4∗
p^{*}_{4}
p...∗
p^{*}_{...}
pM∗
p^{*}_{M}
q2
q_{2}
q3
q_{3}
q4
q_{4}
qN
q_{N}
q...
q_{...}
4.1
\text{4.1}
2.6
\text{2.6}
11.3
\text{11.3}
89.7
\text{89.7}
14.2
\text{14.2}
90.8
\text{90.8}
64.2
\text{64.2}
7.8
\text{7.8}
16.9
\text{16.9}
70.8
\text{70.8}
16.7
\text{16.7}
32.4
\text{32.4}
9.7
\text{9.7}
15.4
\text{15.4}
8.3
\text{8.3}
20.1
\text{20.1}
11.6
\text{11.6}
14.9
\text{14.9}
18.1
\text{18.1}
3.7
\text{3.7}
24.1
\text{24.1}
36.4
\text{36.4}
12.2
\text{12.2}
9.5
\text{9.5}
14.2
\text{14.2}
17.5
\text{17.5}
23.45
\text{23.45}
6.7
\text{6.7}
19.9
\text{19.9}
22.1
\text{22.1}
14.8
\text{14.8}
16.4
\text{16.4}
63.7
\text{63.7}
21.5
\text{21.5}
81.1
\text{81.1}
9.6
\text{9.6}
q1
q_{1}
to obtain one-to-one matching
\textit{to obtain one-to-one matching }
Cij:=C(pi∗,qj)
C_{ij} := C(p^{*}_{i}, q_{j})
=(pi,x∗−qj,x)2+(pi,y∗−qj,y)2+(pi,z∗−qj,z)2
= \sqrt{\left( p^{*}_{i,x} -q_{j,x} \right)^{2} + \left( p^{*}_{i,y} -q_{j,y} \right)^{2} + \left( p^{*}_{i,z} -q_{j,z} \right)^{2}}
X^=Xarg min∑i=1M∑j=1NCijXij
\hat{X}= \underset{X}{\text{arg min}} \sum_{i=1}^{M} \sum_{j=1}^{N} C_{ij} X_{ij}
where Xij∈{0,1}
\text{ where } X_{ij} \in \{0, 1\}
s.t.∑k=1k=MXik=1
\text{ s.t.} \sum_{k=1}^{k=M} X_{ik} = 1
∑k=1k=NXkj≤1
\sum_{k=1}^{k=N} X_{kj} \leq 1
Our approach - at the time of BIC∗
\text{Our approach - at the time of $BIC^{*}$}
Apply Hungarian Algorithm
\textit{Apply Hungarian Algorithm}
∗BioImage Computing, ECCV 2020
^{*}\text{BioImage Computing, ECCV 2020}
Intra-modal
\text{Intra-modal}
Inter-modal
\text{Inter-modal}
registration
\text{registration}
Before
\text{Before}
Results (
\text{Results (}
Intra-modal
\text{Intra-modal}
Inter-modal
\text{Inter-modal}
registration)
\text{registration)}
after
\text{\textit{after}}
Results (
\text{Results (}
registration)
\text{registration)}
after
\text{\textit{after}}
Conclusions - at the time of BIC
\text{Conclusions - at the time of BIC}
Intra-modal
\text{Intra-modal}
Inter-modal
\text{Inter-modal}
We demonstrated a pipeline for registering volumetric images of embryos
\text{We demonstrated a pipeline for registering volumetric images of embryos}
by establishing correspondences between cells
\text{by establishing correspondences between cells}
Our approach produces lower average registration error than baselines on real data
\text{Our approach produces lower average registration error than baselines on real data}
We developed a FIJI & Napari plugin for easier useability
\text{We developed a FIJI \& Napari plugin for easier useability }
BUT there is error in matching nuclei
\text{BUT there is error in matching nuclei}
Modify Last Step - Benefit from Tracking GT data
\text{Modify Last Step - Benefit from Tracking GT data}
Inter-modal
\text{Inter-modal}
t = T
\text{t = T }
t = T + K
\text{t = T + K}
…
\ldots
…
\ldots
t = T - K
\text{t = T - K}
CT−K
C^{T-K}
CT
C^{T}
CT+K
C^{T+K}
…
\ldots
…
\ldots
Use CT−K, …, CT, …, CT+K to determine the best X that respects the tracking!
\text{Use $C^{T-K}$, \ldots, $C^{T}$, \ldots, $C^{T+K}$ to determine the best $X$ that respects the tracking!}
Instance Segmentation
\text{Instance Segmentation}
Instance Segmentation
\text{Instance Segmentation}
Unique Segmentation of Each Nucleus/Cell
\text{Unique Segmentation of Each Nucleus/Cell}
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
16
16
17
17
EmbedSeg 2D∗
\text{EmbedSeg 2D$^{*}$}
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
16
16
17
17
Pixels belonging to one object should be embedded together
\text{Pixels belonging to one object should be embedded together}
Pixels belonging to different objects should be embedded separately
\text{Pixels belonging to different objects should be embedded separately }
∗Under review at ISBI, 2021
*\text{Under review at ISBI, 2021}
“Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth”
\text{``Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth''}
Neven et al, CVPR, June 2019
\text{Neven et al, CVPR, June 2019}
Results
\text{Results}
Why Another Method?
\text{Why Another Method?}
CellPose addresses 3D segmentation through a proxy approach
\text{CellPose addresses 3D segmentation through a proxy approach}
StarDist3D consumes large GPU memory and has blocky appearance
\text{StarDist3D consumes large GPU memory and has blocky appearance}
Why Another Method?
\text{Why Another Method?}
CellPose addresses 3D segmentation through a proxy approach
\text{CellPose addresses 3D segmentation through a proxy approach}
StarDist3D consumes large GPU memory and has blocky appearance
\text{StarDist3D consumes large GPU memory and has blocky appearance}
Annotating Platynereis Data
\text{Annotating Platynereis Data}
Future directions
\text{Future directions}
Predict instance shapes on both live embryo and in-situ images at 16, 20 and 24 hours
\text{Predict instance shapes on both live embryo and in-situ images at 16, 20 and 24 hours}
Use morphology information in addition to geometric arrangement for matching
\text{Use morphology information in addition to geometric arrangement for matching}
Embryo at 16 hpf
\text{Embryo at 16 hpf}
Nuclei
\text{Nuclei}
Pax3/7
\text{Pax3/7}
Embryo at 20 hpf
\text{Embryo at 20 hpf}
Nuclei
\text{Nuclei}
Pax3/7
\text{Pax3/7}
See if the network can learn a neighborhood descriptor on its own
\text{See if the network can learn a neighborhood descriptor on its own}
Thank You for Listening!
\text{Thank You for Listening!}
T o m a n c a k L a b , G r o u p M e e t i n g \text{Tomancak Lab, Group Meeting} D e c 1 5 , 2 0 2 0 \text{Dec 15, 2020}
Group Meeting, Tomancak Lab
By Manan Lalit
Group Meeting, Tomancak Lab
Dec 15, 2020
- 221
