Tomancak Lab, Group Meeting
\text{Tomancak Lab, Group Meeting}
Tomancak Lab, Group Meeting
\text{Tomancak Lab, Group Meeting}
Dec 15, 2020
\text{Dec 15, 2020}
Dec 15, 2020
\text{Dec 15, 2020}
Early development of an organism
\text{Early development of an organism}
Early development of an organism
\text{Early development of an organism}
Nuclei
\text{Nuclei}
Nuclei
\text{Nuclei}
Membrane
\text{Membrane}
Membrane
\text{Membrane}
Light sheet Imaging
\text{Light sheet Imaging}
Light sheet Imaging
\text{Light sheet Imaging}
Time-lapse recording of Embryo 1
\text{Time-lapse recording of Embryo 1}
Time-lapse recording of Embryo 1
\text{Time-lapse recording of Embryo 1}
Early development of an organism
\text{Early development of an organism}
Early development of an organism
\text{Early development of an organism}
Nuclei
\text{Nuclei}
Nuclei
\text{Nuclei}
Membrane
\text{Membrane}
Membrane
\text{Membrane}
Light sheet Imaging
\text{Light sheet Imaging}
Light sheet Imaging
\text{Light sheet Imaging}
Time-lapse recording of Embryo 1
\text{Time-lapse recording of Embryo 1}
Time-lapse recording of Embryo 1
\text{Time-lapse recording of Embryo 1}
Confocal Imaging
\text{Confocal Imaging}
Confocal Imaging
\text{Confocal Imaging}
Embryo 2
\text{Embryo 2}
Embryo 2
\text{Embryo 2}
Embryo ...
\text{Embryo ...}
Embryo ...
\text{Embryo ...}
Embryo N
\text{Embryo N}
Embryo N
\text{Embryo N}
Nuclei
\text{Nuclei}
Nuclei
\text{Nuclei}
Gene
\text{Gene}
Gene
\text{Gene}
Membrane
\text{Membrane}
Membrane
\text{Membrane}
Our model organism
\text{Our model organism}
Our model organism
\text{Our model organism}
Platynereis
dumerillii
\text{\textit{Platynereis dumerillii}}
Platynereis dumerillii
\text{\textit{Platynereis dumerillii}}
Light Sheet Imaging of Live Embryo
\text{Light Sheet Imaging of Live Embryo}
Light Sheet Imaging of Live Embryo
\text{Light Sheet Imaging of Live Embryo}
Confocal Images of specimens
\text{Confocal Images of specimens}
Confocal Images of specimens
\text{Confocal Images of specimens}
Adult worm
\text{Adult worm}
Adult worm
\text{Adult worm}
Embryo @ 16 hpf*
\text{Embryo @ 16 hpf*}
Embryo @ 16 hpf*
\text{Embryo @ 16 hpf*}
hpf*: hours post fertilization
\text{hpf*: hours post fertilization}
hpf*: hours post fertilization
\text{hpf*: hours post fertilization}
at 16 hpf using ISH*
\text{at 16 hpf using ISH*}
at 16 hpf using ISH*
\text{at 16 hpf using ISH*}
ISH*:
in
situ
hybridization
\text{ISH*: \textit{in situ} hybridization}
ISH*:
in situ
hybridization
\text{ISH*: \textit{in situ} hybridization}
Adult worm
\text{Adult worm}
Adult worm
\text{Adult worm}
Our problem statement
\text{Our problem statement}
Our problem statement
\text{Our problem statement}
Our problem statement
\text{Our problem statement}
Our problem statement
\text{Our problem statement}
Intramodal Registration
\text{Intramodal Registration}
Intramodal Registration
\text{Intramodal Registration}
Intermodal Registration
\text{Intermodal Registration}
Intermodal Registration
\text{Intermodal Registration}
* tp: time point
\text{* tp: time point}
* tp: time point
\text{* tp: time point}
Challenges in identifying nuclei correspondence
\text{Challenges in identifying nuclei correspondence}
Challenges in identifying nuclei correspondence
\text{Challenges in identifying nuclei correspondence}
Live Embryo:
\text{Live Embryo:}
Live Embryo:
\text{Live Embryo:}
Nuclei
\text{Nuclei}
Nuclei
\text{Nuclei}
Cell nuclei positions were fitted to an
\text{Cell nuclei positions were fitted to an}
Cell nuclei positions were fitted to an
\text{Cell nuclei positions were fitted to an}
ellipsoid
\text{ellipsoid}
ellipsoid
\text{ellipsoid}
s
i
t
u
situ
s
i
t
u
situ
Nuclei
\text{Nuclei}
Nuclei
\text{Nuclei}
Cell nuclei positions were fitted to an
\text{Cell nuclei positions were fitted to an}
Cell nuclei positions were fitted to an
\text{Cell nuclei positions were fitted to an}
ellipsoid
\text{ellipsoid}
ellipsoid
\text{ellipsoid}
I
n
−
In-
I
n
−
In-
specimen:
\text{specimen:}
specimen:
\text{specimen:}
How to identify cell-to-cell correspondences?
\text{How to identify cell-to-cell correspondences?}
How to identify cell-to-cell correspondences?
\text{How to identify cell-to-cell correspondences?}
Morphology
\text{Morphology}
Morphology
\text{Morphology}
Appearance
\text{Appearance}
Appearance
\text{Appearance}
Geometric Arrangement
\text{Geometric Arrangement}
Geometric Arrangement
\text{Geometric Arrangement}
Use distinctively shaped nuclei to align and match
\text{Use distinctively shaped nuclei to align and match}
Use distinctively shaped nuclei to align and match
\text{Use distinctively shaped nuclei to align and match}
Use neighborhood as a feature to identify corrsponding nuclei
\text{Use neighborhood as a feature to identify corrsponding nuclei}
Use neighborhood as a feature to identify corrsponding nuclei
\text{Use neighborhood as a feature to identify corrsponding nuclei}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
σ
=
6
\sigma=6
σ
=
6
\sigma=6
σ
=
7
\sigma=7
σ
=
7
\sigma=7
σ
=
5
\sigma=5
σ
=
5
\sigma=5
σ
=
8
\sigma=8
σ
=
8
\sigma=8
1. Evaluate
∇
2
G
σ
⊛
I
\text{1. Evaluate }\nabla^{2} G_{\sigma} \circledast \mathbf{I}
1. Evaluate
∇
2
G
σ
⊛
I
\text{1. Evaluate }\nabla^{2} G_{\sigma} \circledast \mathbf{I}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
σ
=
6
\sigma=6
σ
=
6
\sigma=6
σ
=
7
\sigma=7
σ
=
7
\sigma=7
σ
=
5
\sigma=5
σ
=
5
\sigma=5
σ
=
8
\sigma=8
σ
=
8
\sigma=8
1. Evaluate
∇
2
G
σ
⊛
I
\text{1. Evaluate }\nabla^{2} G_{\sigma} \circledast \mathbf{I}
1. Evaluate
∇
2
G
σ
⊛
I
\text{1. Evaluate }\nabla^{2} G_{\sigma} \circledast \mathbf{I}
2. Find local minima
\text{2. Find local minima }
2. Find local minima
\text{2. Find local minima }
in x, y, z,
σ
space
\text{in x, y, z, $\sigma$ space}
in x, y, z,
σ
space
\text{in x, y, z, $\sigma$ space}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Belongie et al.“Shape matching and object recognition using shape contexts"
\text{Belongie et al.``Shape matching and object recognition using shape contexts"}
Belongie et al.“Shape matching and object recognition using shape contexts"
\text{Belongie et al.``Shape matching and object recognition using shape contexts"}
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
\text{IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002}
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
\text{IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002}
Z’
\text{Z'}
Z’
\text{Z'}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Y’
\text{Y'}
Y’
\text{Y'}
Z’
\text{Z'}
Z’
\text{Z'}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Z’
\text{Z'}
Z’
\text{Z'}
Y’
\text{Y'}
Y’
\text{Y'}
X’
\text{X'}
X’
\text{X'}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
3
3
3
3
11
11
11
11
15
15
15
15
6
6
6
6
12
12
12
12
4
4
4
4
0
0
0
0
9
9
9
9
0
0
0
0
0
0
0
0
22
22
22
22
3
3
3
3
5
5
5
5
17
17
17
17
11
11
11
11
19
19
19
19
Bins:
\text{Bins: }
Bins:
\text{Bins: }
1
\text{1}
1
\text{1}
2
\text{2}
2
\text{2}
3
\text{3}
3
\text{3}
4
\text{4}
4
\text{4}
...
\text{...}
...
\text{...}
...
\text{...}
...
\text{...}
K
\text{K}
K
\text{K}
6
6
6
6
6
6
6
6
6
6
6
6
0
0
0
0
0
0
0
0
0
0
0
0
2
2
2
2
2
2
2
2
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
20
20
20
20
18
18
18
18
14
14
14
14
13
13
13
13
16
16
16
16
16
16
16
16
9
9
9
9
19
19
19
19
Nucleus 1 @ (12, 34, 89)
\text{Nucleus 1 @ (12, 34, 89)}
Nucleus 1 @ (12, 34, 89)
\text{Nucleus 1 @ (12, 34, 89)}
Nucleus 2 @ (26, 49, 92)
\text{Nucleus 2 @ (26, 49, 92)}
Nucleus 2 @ (26, 49, 92)
\text{Nucleus 2 @ (26, 49, 92)}
Nucleus 3 @ (68, 93, 21)
\text{Nucleus 3 @ (68, 93, 21)}
Nucleus 3 @ (68, 93, 21)
\text{Nucleus 3 @ (68, 93, 21)}
Nucleus 4 @ (81, 32, 15)
\text{Nucleus 4 @ (81, 32, 15)}
Nucleus 4 @ (81, 32, 15)
\text{Nucleus 4 @ (81, 32, 15)}
Nucleus ... @ (..., ..., ...)
\text{Nucleus ... @ (..., ..., ...)}
Nucleus ... @ (..., ..., ...)
\text{Nucleus ... @ (..., ..., ...)}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
0.03
0.03
0.03
0.03
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.06
0.06
0.06
0.06
0.1
0.1
0.1
0.1
0.01
0.01
0.01
0.01
0
0
0
0
0.03
0.03
0.03
0.03
0
0
0
0
0
0
0
0
0.2
0.2
0.2
0.2
0.01
0.01
0.01
0.01
0.02
0.02
0.02
0.02
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.2
0.2
0.2
0.2
Bins:
\text{Bins: }
Bins:
\text{Bins: }
1
\text{1}
1
\text{1}
2
\text{2}
2
\text{2}
3
\text{3}
3
\text{3}
4
\text{4}
4
\text{4}
...
\text{...}
...
\text{...}
...
\text{...}
...
\text{...}
K
\text{K}
K
\text{K}
0.1
0.1
0.1
0.1
0.02
0.02
0.02
0.02
0.02
0.02
0.02
0.02
0
0
0
0
0
0
0
0
0
0
0
0
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.03
0.03
0.03
0.03
0.2
0.2
0.2
0.2
Nucleus 1 @ (12, 34, 89)
\text{Nucleus 1 @ (12, 34, 89)}
Nucleus 1 @ (12, 34, 89)
\text{Nucleus 1 @ (12, 34, 89)}
Nucleus 2 @ (26, 49, 92)
\text{Nucleus 2 @ (26, 49, 92)}
Nucleus 2 @ (26, 49, 92)
\text{Nucleus 2 @ (26, 49, 92)}
Nucleus 3 @ (68, 93, 21)
\text{Nucleus 3 @ (68, 93, 21)}
Nucleus 3 @ (68, 93, 21)
\text{Nucleus 3 @ (68, 93, 21)}
Nucleus 4 @ (81, 32, 15)
\text{Nucleus 4 @ (81, 32, 15)}
Nucleus 4 @ (81, 32, 15)
\text{Nucleus 4 @ (81, 32, 15)}
Nucleus ... @ (..., ..., ...)
\text{Nucleus ... @ (..., ..., ...)}
Nucleus ... @ (..., ..., ...)
\text{Nucleus ... @ (..., ..., ...)}
∑
k
=
1
k
=
K
h
(
k
)
=
1
\sum^{k=K}_{k=1} h(k) =1
∑
k
=
1
k
=
K
h
(
k
)
=
1
\sum^{k=K}_{k=1} h(k) =1
Our approach
\text{Our approach}
Our approach
\text{Our approach}
...
\text{...}
...
\text{...}
C
i
j
:
=
C
(
p
i
,
q
j
)
C_{ij} := C(p_{i}, q_{j})
C
ij
:=
C
(
p
i
,
q
j
)
C_{ij} := C(p_{i}, q_{j})
1
\text{1}
1
\text{1}
1’
\text{1'}
1’
\text{1'}
2
\text{2}
2
\text{2}
2’
\text{2'}
2’
\text{2'}
3
\text{3}
3
\text{3}
3’
\text{3'}
3’
\text{3'}
4
\text{4}
4
\text{4}
4’
\text{4'}
4’
\text{4'}
2
\text{2}
2
\text{2}
...
\text{...}
...
\text{...}
p
1
p_{1}
p
1
p_{1}
p
2
p_{2}
p
2
p_{2}
p
3
p_{3}
p
3
p_{3}
p
4
p_{4}
p
4
p_{4}
p
.
.
.
p_{...}
p
...
p_{...}
p
M
p_{M}
p
M
p_{M}
q
1
q_{1}
q
1
q_{1}
q
2
q_{2}
q
2
q_{2}
q
3
q_{3}
q
3
q_{3}
q
4
q_{4}
q
4
q_{4}
q
M
q_{M}
q
M
q_{M}
q
.
.
.
q_{...}
q
...
q_{...}
=
1
2
∑
k
=
1
K
(
h
i
(
k
)
−
h
j
(
k
)
)
2
h
i
(
k
)
+
h
j
(
k
)
= \frac{1}{2} \sum_{k=1}^{K} \frac{(h_{i}(k) -h_{j}(k))^{2}}{h_{i}(k) + h_{j}(k)}
=
2
1
∑
k
=
1
K
h
i
(
k
)
+
h
j
(
k
)
(
h
i
(
k
)
−
h
j
(
k
)
)
2
= \frac{1}{2} \sum_{k=1}^{K} \frac{(h_{i}(k) -h_{j}(k))^{2}}{h_{i}(k) + h_{j}(k)}
0.24
\text{0.24}
0.24
\text{0.24}
0.19
\text{0.19}
0.19
\text{0.19}
0.21
\text{0.21}
0.21
\text{0.21}
0.12
\text{0.12}
0.12
\text{0.12}
0.15
\text{0.15}
0.15
\text{0.15}
0.18
\text{0.18}
0.18
\text{0.18}
0.06
\text{0.06}
0.06
\text{0.06}
0.23
\text{0.23}
0.23
\text{0.23}
0.16
\text{0.16}
0.16
\text{0.16}
0.07
\text{0.07}
0.07
\text{0.07}
0.16
\text{0.16}
0.16
\text{0.16}
0.13
\text{0.13}
0.13
\text{0.13}
0.29
\text{0.29}
0.29
\text{0.29}
0.15
\text{0.15}
0.15
\text{0.15}
0.28
\text{0.28}
0.28
\text{0.28}
0.20
\text{0.20}
0.20
\text{0.20}
0.11
\text{0.11}
0.11
\text{0.11}
0.14
\text{0.14}
0.14
\text{0.14}
0.18
\text{0.18}
0.18
\text{0.18}
0.13
\text{0.13}
0.13
\text{0.13}
0.24
\text{0.24}
0.24
\text{0.24}
0.26
\text{0.26}
0.26
\text{0.26}
0.12
\text{0.12}
0.12
\text{0.12}
0.09
\text{0.09}
0.09
\text{0.09}
0.14
\text{0.14}
0.14
\text{0.14}
0.17
\text{0.17}
0.17
\text{0.17}
0.23
\text{0.23}
0.23
\text{0.23}
0.26
\text{0.26}
0.26
\text{0.26}
0.19
\text{0.19}
0.19
\text{0.19}
0.22
\text{0.22}
0.22
\text{0.22}
0.14
\text{0.14}
0.14
\text{0.14}
0.16
\text{0.16}
0.16
\text{0.16}
0.23
\text{0.23}
0.23
\text{0.23}
0.21
\text{0.21}
0.21
\text{0.21}
0.11
\text{0.11}
0.11
\text{0.11}
0.09
\text{0.09}
0.09
\text{0.09}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Apply
RANSAC
to
estimate
\textit{Apply RANSAC to estimate}
Apply RANSAC to estimate
\textit{Apply RANSAC to estimate}
coarse
transform
using
inliers
\textit{coarse transform using inliers}
coarse transform using inliers
\textit{coarse transform using inliers}
[
1
x
′
2
x
′
3
x
′
4
x
′
1
y
′
2
y
′
3
y
′
4
y
′
1
z
′
2
z
′
3
z
′
4
z
′
]
=
A
×
[
1
x
2
x
3
x
4
x
1
y
2
y
3
y
4
y
1
z
2
z
3
z
4
z
]
\begin{bmatrix} 1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\ 1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\ 1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\ \end{bmatrix}= A \times \begin{bmatrix} 1_{x} & 2_{x} & 3_{x} & 4_{x}\\ 1_{y} & 2_{y} & 3_{y} & 4_{y}\\ 1_{z} & 2_{z} & 3_{z} & 4_{z}\\ \end{bmatrix}
1
x
′
1
y
′
1
z
′
2
x
′
2
y
′
2
z
′
3
x
′
3
y
′
3
z
′
4
x
′
4
y
′
4
z
′
=
A
×
1
x
1
y
1
z
2
x
2
y
2
z
3
x
3
y
3
z
4
x
4
y
4
z
\begin{bmatrix} 1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\ 1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\ 1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\ \end{bmatrix}= A \times \begin{bmatrix} 1_{x} & 2_{x} & 3_{x} & 4_{x}\\ 1_{y} & 2_{y} & 3_{y} & 4_{y}\\ 1_{z} & 2_{z} & 3_{z} & 4_{z}\\ \end{bmatrix}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
“Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.”
\text{``Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.''}
“Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.”
\text{``Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.''}
Fischer and Bolles, Communications of the ACM (1981)
\text{Fischer and Bolles, Communications of the ACM (1981)}
Fischer and Bolles, Communications of the ACM (1981)
\text{Fischer and Bolles, Communications of the ACM (1981)}
Count Inliers
\text{Count Inliers}
Count Inliers
\text{Count Inliers}
Apply
RANSAC
to
estimate
\textit{Apply RANSAC to estimate}
Apply RANSAC to estimate
\textit{Apply RANSAC to estimate}
coarse
transform
using
inliers
\textit{coarse transform using inliers}
coarse transform using inliers
\textit{coarse transform using inliers}
[
1
x
′
2
x
′
3
x
′
4
x
′
1
y
′
2
y
′
3
y
′
4
y
′
1
z
′
2
z
′
3
z
′
4
z
′
]
=
A
×
[
1
x
2
x
3
x
4
x
1
y
2
y
3
y
4
y
1
z
2
z
3
z
4
z
]
\begin{bmatrix} 1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\ 1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\ 1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\ \end{bmatrix}= A \times \begin{bmatrix} 1_{x} & 2_{x} & 3_{x} & 4_{x}\\ 1_{y} & 2_{y} & 3_{y} & 4_{y}\\ 1_{z} & 2_{z} & 3_{z} & 4_{z}\\ \end{bmatrix}
1
x
′
1
y
′
1
z
′
2
x
′
2
y
′
2
z
′
3
x
′
3
y
′
3
z
′
4
x
′
4
y
′
4
z
′
=
A
×
1
x
1
y
1
z
2
x
2
y
2
z
3
x
3
y
3
z
4
x
4
y
4
z
\begin{bmatrix} 1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\ 1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\ 1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\ \end{bmatrix}= A \times \begin{bmatrix} 1_{x} & 2_{x} & 3_{x} & 4_{x}\\ 1_{y} & 2_{y} & 3_{y} & 4_{y}\\ 1_{z} & 2_{z} & 3_{z} & 4_{z}\\ \end{bmatrix}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Repeat
\text{Repeat}
Repeat
\text{Repeat}
Count Inliers
\text{Count Inliers}
Count Inliers
\text{Count Inliers}
Apply
RANSAC
to
estimate
\textit{Apply RANSAC to estimate}
Apply RANSAC to estimate
\textit{Apply RANSAC to estimate}
coarse
transform
using
inliers
\textit{coarse transform using inliers}
coarse transform using inliers
\textit{coarse transform using inliers}
[
1
x
′
2
x
′
3
x
′
4
x
′
1
y
′
2
y
′
3
y
′
4
y
′
1
z
′
2
z
′
3
z
′
4
z
′
]
=
A
×
[
1
x
2
x
3
x
4
x
1
y
2
y
3
y
4
y
1
z
2
z
3
z
4
z
]
\begin{bmatrix} 1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\ 1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\ 1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\ \end{bmatrix}= A \times \begin{bmatrix} 1_{x} & 2_{x} & 3_{x} & 4_{x}\\ 1_{y} & 2_{y} & 3_{y} & 4_{y}\\ 1_{z} & 2_{z} & 3_{z} & 4_{z}\\ \end{bmatrix}
1
x
′
1
y
′
1
z
′
2
x
′
2
y
′
2
z
′
3
x
′
3
y
′
3
z
′
4
x
′
4
y
′
4
z
′
=
A
×
1
x
1
y
1
z
2
x
2
y
2
z
3
x
3
y
3
z
4
x
4
y
4
z
\begin{bmatrix} 1^{'}_{x} & 2^{'}_{x} & 3^{'}_{x} & 4^{'}_{x}\\ 1^{'}_{y} & 2^{'}_{y} & 3^{'}_{y} & 4^{'}_{y}\\ 1^{'}_{z} & 2^{'}_{z} & 3^{'}_{z} & 4^{'}_{z}\\ \end{bmatrix}= A \times \begin{bmatrix} 1_{x} & 2_{x} & 3_{x} & 4_{x}\\ 1_{y} & 2_{y} & 3_{y} & 4_{y}\\ 1_{z} & 2_{z} & 3_{z} & 4_{z}\\ \end{bmatrix}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Sample 4 pairs of correspondences
\text{Sample 4 pairs of correspondences}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
...
\text{...}
...
\text{...}
...
\text{...}
...
\text{...}
p
1
∗
p^{*}_{1}
p
1
∗
p^{*}_{1}
p
2
∗
p^{*}_{2}
p
2
∗
p^{*}_{2}
p
3
∗
p^{*}_{3}
p
3
∗
p^{*}_{3}
p
4
∗
p^{*}_{4}
p
4
∗
p^{*}_{4}
p
.
.
.
∗
p^{*}_{...}
p
...
∗
p^{*}_{...}
p
M
∗
p^{*}_{M}
p
M
∗
p^{*}_{M}
q
1
q_{1}
q
1
q_{1}
q
2
q_{2}
q
2
q_{2}
q
3
q_{3}
q
3
q_{3}
q
4
q_{4}
q
4
q_{4}
q
N
q_{N}
q
N
q_{N}
q
.
.
.
q_{...}
q
...
q_{...}
10.1
\text{10.1}
10.1
\text{10.1}
2.6
\text{2.6}
2.6
\text{2.6}
11.3
\text{11.3}
11.3
\text{11.3}
89.7
\text{89.7}
89.7
\text{89.7}
14.2
\text{14.2}
14.2
\text{14.2}
90.8
\text{90.8}
90.8
\text{90.8}
64.2
\text{64.2}
64.2
\text{64.2}
23.5
\text{23.5}
23.5
\text{23.5}
16.9
\text{16.9}
16.9
\text{16.9}
70.8
\text{70.8}
70.8
\text{70.8}
16.7
\text{16.7}
16.7
\text{16.7}
32.4
\text{32.4}
32.4
\text{32.4}
9.7
\text{9.7}
9.7
\text{9.7}
15.4
\text{15.4}
15.4
\text{15.4}
28.3
\text{28.3}
28.3
\text{28.3}
20.1
\text{20.1}
20.1
\text{20.1}
11.6
\text{11.6}
11.6
\text{11.6}
14.9
\text{14.9}
14.9
\text{14.9}
18.1
\text{18.1}
18.1
\text{18.1}
3.7
\text{3.7}
3.7
\text{3.7}
24.1
\text{24.1}
24.1
\text{24.1}
36.4
\text{36.4}
36.4
\text{36.4}
12.2
\text{12.2}
12.2
\text{12.2}
9.5
\text{9.5}
9.5
\text{9.5}
14.2
\text{14.2}
14.2
\text{14.2}
17.5
\text{17.5}
17.5
\text{17.5}
23.45
\text{23.45}
23.45
\text{23.45}
6.7
\text{6.7}
6.7
\text{6.7}
19.9
\text{19.9}
19.9
\text{19.9}
22.1
\text{22.1}
22.1
\text{22.1}
14.8
\text{14.8}
14.8
\text{14.8}
16.4
\text{16.4}
16.4
\text{16.4}
63.7
\text{63.7}
63.7
\text{63.7}
21.5
\text{21.5}
21.5
\text{21.5}
81.1
\text{81.1}
81.1
\text{81.1}
9.6
\text{9.6}
9.6
\text{9.6}
C
i
j
:
=
C
(
p
i
∗
,
q
j
)
C_{ij} := C(p^{*}_{i}, q_{j})
C
ij
:=
C
(
p
i
∗
,
q
j
)
C_{ij} := C(p^{*}_{i}, q_{j})
=
(
p
i
,
x
∗
−
q
j
,
x
)
2
+
(
p
i
,
y
∗
−
q
j
,
y
)
2
+
(
p
i
,
z
∗
−
q
j
,
z
)
2
= \sqrt{\left( p^{*}_{i,x} -q_{j,x} \right)^{2} + \left( p^{*}_{i,y} -q_{j,y} \right)^{2} + \left( p^{*}_{i,z} -q_{j,z} \right)^{2}}
=
(
p
i
,
x
∗
−
q
j
,
x
)
2
+
(
p
i
,
y
∗
−
q
j
,
y
)
2
+
(
p
i
,
z
∗
−
q
j
,
z
)
2
= \sqrt{\left( p^{*}_{i,x} -q_{j,x} \right)^{2} + \left( p^{*}_{i,y} -q_{j,y} \right)^{2} + \left( p^{*}_{i,z} -q_{j,z} \right)^{2}}
1
\text{1}
1
\text{1}
5’
\text{5'}
5’
\text{5'}
1’. 4’
\text{1'. 4'}
1’. 4’
\text{1'. 4'}
3
\text{3}
3
\text{3}
2’
\text{2'}
2’
\text{2'}
4
\text{4}
4
\text{4}
6’
\text{6'}
6’
\text{6'}
2
\text{2}
2
\text{2}
3’
\text{3'}
3’
\text{3'}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Apply
ICP
to
obtain
a
tighter
fit
\textit{Apply ICP to obtain a tighter fit}
Apply ICP to obtain a tighter fit
\textit{Apply ICP to obtain a tighter fit}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Apply
ICP
to
obtain
a
tighter
fit
\textit{Apply ICP to obtain a tighter fit}
Apply ICP to obtain a tighter fit
\textit{Apply ICP to obtain a tighter fit}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Estimate Affine Transform
\text{Estimate Affine Transform}
Estimate Affine Transform
\text{Estimate Affine Transform}
[
1
x
′
2
x
′
…
M
x
′
1
y
′
2
y
′
…
M
y
′
1
z
′
2
z
′
…
M
z
′
]
=
A
×
[
1
x
2
x
…
M
x
1
y
2
y
…
M
y
1
z
2
z
…
M
z
]
\begin{bmatrix} 1^{'}_{x} & 2^{'}_{x} & \ldots & M^{'}_{x}\\ 1^{'}_{y} & 2^{'}_{y} & \ldots & M^{'}_{y}\\ 1^{'}_{z} & 2^{'}_{z} & \ldots & M^{'}_{z}\\ \end{bmatrix}= A \times \begin{bmatrix} 1_{x} & 2_{x} & \ldots & M_{x}\\ 1_{y} & 2_{y} &\ldots & M_{y}\\ 1_{z} & 2_{z} & \ldots & M_{z}\\ \end{bmatrix}
1
x
′
1
y
′
1
z
′
2
x
′
2
y
′
2
z
′
…
…
…
M
x
′
M
y
′
M
z
′
=
A
×
1
x
1
y
1
z
2
x
2
y
2
z
…
…
…
M
x
M
y
M
z
\begin{bmatrix} 1^{'}_{x} & 2^{'}_{x} & \ldots & M^{'}_{x}\\ 1^{'}_{y} & 2^{'}_{y} & \ldots & M^{'}_{y}\\ 1^{'}_{z} & 2^{'}_{z} & \ldots & M^{'}_{z}\\ \end{bmatrix}= A \times \begin{bmatrix} 1_{x} & 2_{x} & \ldots & M_{x}\\ 1_{y} & 2_{y} &\ldots & M_{y}\\ 1_{z} & 2_{z} & \ldots & M_{z}\\ \end{bmatrix}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
Apply
ICP
to
obtain
a
tighter
fit
\textit{Apply ICP to obtain a tighter fit}
Apply ICP to obtain a tighter fit
\textit{Apply ICP to obtain a tighter fit}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Find Nearest Neighbor Correspondences
\text{Find Nearest Neighbor Correspondences}
Estimate Affine Transform
\text{Estimate Affine Transform}
Estimate Affine Transform
\text{Estimate Affine Transform}
Repeat upto Convergence
\text{Repeat upto Convergence}
Repeat upto Convergence
\text{Repeat upto Convergence}
Our approach
\text{Our approach}
Our approach
\text{Our approach}
...
\text{...}
...
\text{...}
...
\text{...}
...
\text{...}
p
1
∗
p^{*}_{1}
p
1
∗
p^{*}_{1}
p
2
∗
p^{*}_{2}
p
2
∗
p^{*}_{2}
p
3
∗
p^{*}_{3}
p
3
∗
p^{*}_{3}
p
4
∗
p^{*}_{4}
p
4
∗
p^{*}_{4}
p
.
.
.
∗
p^{*}_{...}
p
...
∗
p^{*}_{...}
p
M
∗
p^{*}_{M}
p
M
∗
p^{*}_{M}
q
2
q_{2}
q
2
q_{2}
q
3
q_{3}
q
3
q_{3}
q
4
q_{4}
q
4
q_{4}
q
N
q_{N}
q
N
q_{N}
q
.
.
.
q_{...}
q
...
q_{...}
4.1
\text{4.1}
4.1
\text{4.1}
2.6
\text{2.6}
2.6
\text{2.6}
11.3
\text{11.3}
11.3
\text{11.3}
89.7
\text{89.7}
89.7
\text{89.7}
14.2
\text{14.2}
14.2
\text{14.2}
90.8
\text{90.8}
90.8
\text{90.8}
64.2
\text{64.2}
64.2
\text{64.2}
7.8
\text{7.8}
7.8
\text{7.8}
16.9
\text{16.9}
16.9
\text{16.9}
70.8
\text{70.8}
70.8
\text{70.8}
16.7
\text{16.7}
16.7
\text{16.7}
32.4
\text{32.4}
32.4
\text{32.4}
9.7
\text{9.7}
9.7
\text{9.7}
15.4
\text{15.4}
15.4
\text{15.4}
8.3
\text{8.3}
8.3
\text{8.3}
20.1
\text{20.1}
20.1
\text{20.1}
11.6
\text{11.6}
11.6
\text{11.6}
14.9
\text{14.9}
14.9
\text{14.9}
18.1
\text{18.1}
18.1
\text{18.1}
3.7
\text{3.7}
3.7
\text{3.7}
24.1
\text{24.1}
24.1
\text{24.1}
36.4
\text{36.4}
36.4
\text{36.4}
12.2
\text{12.2}
12.2
\text{12.2}
9.5
\text{9.5}
9.5
\text{9.5}
14.2
\text{14.2}
14.2
\text{14.2}
17.5
\text{17.5}
17.5
\text{17.5}
23.45
\text{23.45}
23.45
\text{23.45}
6.7
\text{6.7}
6.7
\text{6.7}
19.9
\text{19.9}
19.9
\text{19.9}
22.1
\text{22.1}
22.1
\text{22.1}
14.8
\text{14.8}
14.8
\text{14.8}
16.4
\text{16.4}
16.4
\text{16.4}
63.7
\text{63.7}
63.7
\text{63.7}
21.5
\text{21.5}
21.5
\text{21.5}
81.1
\text{81.1}
81.1
\text{81.1}
9.6
\text{9.6}
9.6
\text{9.6}
q
1
q_{1}
q
1
q_{1}
to
obtain
one-to-one
matching
\textit{to obtain one-to-one matching }
to obtain one-to-one matching
\textit{to obtain one-to-one matching }
C
i
j
:
=
C
(
p
i
∗
,
q
j
)
C_{ij} := C(p^{*}_{i}, q_{j})
C
ij
:=
C
(
p
i
∗
,
q
j
)
C_{ij} := C(p^{*}_{i}, q_{j})
=
(
p
i
,
x
∗
−
q
j
,
x
)
2
+
(
p
i
,
y
∗
−
q
j
,
y
)
2
+
(
p
i
,
z
∗
−
q
j
,
z
)
2
= \sqrt{\left( p^{*}_{i,x} -q_{j,x} \right)^{2} + \left( p^{*}_{i,y} -q_{j,y} \right)^{2} + \left( p^{*}_{i,z} -q_{j,z} \right)^{2}}
=
(
p
i
,
x
∗
−
q
j
,
x
)
2
+
(
p
i
,
y
∗
−
q
j
,
y
)
2
+
(
p
i
,
z
∗
−
q
j
,
z
)
2
= \sqrt{\left( p^{*}_{i,x} -q_{j,x} \right)^{2} + \left( p^{*}_{i,y} -q_{j,y} \right)^{2} + \left( p^{*}_{i,z} -q_{j,z} \right)^{2}}
X
^
=
arg min
X
∑
i
=
1
M
∑
j
=
1
N
C
i
j
X
i
j
\hat{X}= \underset{X}{\text{arg min}} \sum_{i=1}^{M} \sum_{j=1}^{N} C_{ij} X_{ij}
X
^
=
X
arg min
∑
i
=
1
M
∑
j
=
1
N
C
ij
X
ij
\hat{X}= \underset{X}{\text{arg min}} \sum_{i=1}^{M} \sum_{j=1}^{N} C_{ij} X_{ij}
where
X
i
j
∈
{
0
,
1
}
\text{ where } X_{ij} \in \{0, 1\}
where
X
ij
∈
{
0
,
1
}
\text{ where } X_{ij} \in \{0, 1\}
s.t.
∑
k
=
1
k
=
M
X
i
k
=
1
\text{ s.t.} \sum_{k=1}^{k=M} X_{ik} = 1
s.t.
∑
k
=
1
k
=
M
X
ik
=
1
\text{ s.t.} \sum_{k=1}^{k=M} X_{ik} = 1
∑
k
=
1
k
=
N
X
k
j
≤
1
\sum_{k=1}^{k=N} X_{kj} \leq 1
∑
k
=
1
k
=
N
X
kj
≤
1
\sum_{k=1}^{k=N} X_{kj} \leq 1
Our approach - at the time of
B
I
C
∗
\text{Our approach - at the time of $BIC^{*}$}
Our approach - at the time of
B
I
C
∗
\text{Our approach - at the time of $BIC^{*}$}
Apply
Hungarian
Algorithm
\textit{Apply Hungarian Algorithm}
Apply Hungarian Algorithm
\textit{Apply Hungarian Algorithm}
∗
BioImage Computing, ECCV 2020
^{*}\text{BioImage Computing, ECCV 2020}
∗
BioImage Computing, ECCV 2020
^{*}\text{BioImage Computing, ECCV 2020}
Intra-modal
\text{Intra-modal}
Intra-modal
\text{Intra-modal}
Inter-modal
\text{Inter-modal}
Inter-modal
\text{Inter-modal}
registration
\text{registration}
registration
\text{registration}
Before
\text{Before}
Before
\text{Before}
Results (
\text{Results (}
Results (
\text{Results (}
Intra-modal
\text{Intra-modal}
Intra-modal
\text{Intra-modal}
Inter-modal
\text{Inter-modal}
Inter-modal
\text{Inter-modal}
registration)
\text{registration)}
registration)
\text{registration)}
after
\text{\textit{after}}
after
\text{\textit{after}}
Results (
\text{Results (}
Results (
\text{Results (}
registration)
\text{registration)}
registration)
\text{registration)}
after
\text{\textit{after}}
after
\text{\textit{after}}
Conclusions - at the time of BIC
\text{Conclusions - at the time of BIC}
Conclusions - at the time of BIC
\text{Conclusions - at the time of BIC}
Intra-modal
\text{Intra-modal}
Intra-modal
\text{Intra-modal}
Inter-modal
\text{Inter-modal}
Inter-modal
\text{Inter-modal}
We demonstrated a pipeline for registering volumetric images of embryos
\text{We demonstrated a pipeline for registering volumetric images of embryos}
We demonstrated a pipeline for registering volumetric images of embryos
\text{We demonstrated a pipeline for registering volumetric images of embryos}
by establishing correspondences between cells
\text{by establishing correspondences between cells}
by establishing correspondences between cells
\text{by establishing correspondences between cells}
Our approach produces lower average registration error than baselines on real data
\text{Our approach produces lower average registration error than baselines on real data}
Our approach produces lower average registration error than baselines on real data
\text{Our approach produces lower average registration error than baselines on real data}
We developed a FIJI & Napari plugin for easier useability
\text{We developed a FIJI \& Napari plugin for easier useability }
We developed a FIJI & Napari plugin for easier useability
\text{We developed a FIJI \& Napari plugin for easier useability }
BUT there is error in matching nuclei
\text{BUT there is error in matching nuclei}
BUT there is error in matching nuclei
\text{BUT there is error in matching nuclei}
Modify Last Step - Benefit from Tracking GT data
\text{Modify Last Step - Benefit from Tracking GT data}
Modify Last Step - Benefit from Tracking GT data
\text{Modify Last Step - Benefit from Tracking GT data}
Inter-modal
\text{Inter-modal}
Inter-modal
\text{Inter-modal}
t = T
\text{t = T }
t = T
\text{t = T }
t = T + K
\text{t = T + K}
t = T + K
\text{t = T + K}
…
\ldots
…
\ldots
…
\ldots
…
\ldots
t = T - K
\text{t = T - K}
t = T - K
\text{t = T - K}
C
T
−
K
C^{T-K}
C
T
−
K
C^{T-K}
C
T
C^{T}
C
T
C^{T}
C
T
+
K
C^{T+K}
C
T
+
K
C^{T+K}
…
\ldots
…
\ldots
…
\ldots
…
\ldots
Use
C
T
−
K
,
…
,
C
T
,
…
,
C
T
+
K
to determine the best
X
that respects the tracking!
\text{Use $C^{T-K}$, \ldots, $C^{T}$, \ldots, $C^{T+K}$ to determine the best $X$ that respects the tracking!}
Use
C
T
−
K
,
…
,
C
T
,
…
,
C
T
+
K
to determine the best
X
that respects the tracking!
\text{Use $C^{T-K}$, \ldots, $C^{T}$, \ldots, $C^{T+K}$ to determine the best $X$ that respects the tracking!}
Instance Segmentation
\text{Instance Segmentation}
Instance Segmentation
\text{Instance Segmentation}
Instance Segmentation
\text{Instance Segmentation}
Instance Segmentation
\text{Instance Segmentation}
Unique Segmentation of Each Nucleus/Cell
\text{Unique Segmentation of Each Nucleus/Cell}
Unique Segmentation of Each Nucleus/Cell
\text{Unique Segmentation of Each Nucleus/Cell}
1
1
1
1
2
2
2
2
3
3
3
3
4
4
4
4
5
5
5
5
6
6
6
6
7
7
7
7
8
8
8
8
9
9
9
9
10
10
10
10
11
11
11
11
12
12
12
12
13
13
13
13
14
14
14
14
15
15
15
15
16
16
16
16
17
17
17
17
EmbedSeg 2D
∗
\text{EmbedSeg 2D$^{*}$}
EmbedSeg 2D
∗
\text{EmbedSeg 2D$^{*}$}
1
1
1
1
2
2
2
2
3
3
3
3
4
4
4
4
5
5
5
5
6
6
6
6
7
7
7
7
8
8
8
8
9
9
9
9
10
10
10
10
11
11
11
11
12
12
12
12
13
13
13
13
14
14
14
14
15
15
15
15
16
16
16
16
17
17
17
17
Pixels belonging to one object should be embedded together
\text{Pixels belonging to one object should be embedded together}
Pixels belonging to one object should be embedded together
\text{Pixels belonging to one object should be embedded together}
Pixels belonging to different objects should be embedded separately
\text{Pixels belonging to different objects should be embedded separately }
Pixels belonging to different objects should be embedded separately
\text{Pixels belonging to different objects should be embedded separately }
∗
Under review at ISBI, 2021
*\text{Under review at ISBI, 2021}
∗
Under review at ISBI, 2021
*\text{Under review at ISBI, 2021}
“Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth”
\text{``Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth''}
“Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth”
\text{``Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth''}
Neven et al, CVPR, June 2019
\text{Neven et al, CVPR, June 2019}
Neven et al, CVPR, June 2019
\text{Neven et al, CVPR, June 2019}
Results
\text{Results}
Results
\text{Results}
Why Another Method?
\text{Why Another Method?}
Why Another Method?
\text{Why Another Method?}
CellPose addresses 3D segmentation through a proxy approach
\text{CellPose addresses 3D segmentation through a proxy approach}
CellPose addresses 3D segmentation through a proxy approach
\text{CellPose addresses 3D segmentation through a proxy approach}
StarDist3D consumes large GPU memory and has blocky appearance
\text{StarDist3D consumes large GPU memory and has blocky appearance}
StarDist3D consumes large GPU memory and has blocky appearance
\text{StarDist3D consumes large GPU memory and has blocky appearance}
Why Another Method?
\text{Why Another Method?}
Why Another Method?
\text{Why Another Method?}
CellPose addresses 3D segmentation through a proxy approach
\text{CellPose addresses 3D segmentation through a proxy approach}
CellPose addresses 3D segmentation through a proxy approach
\text{CellPose addresses 3D segmentation through a proxy approach}
StarDist3D consumes large GPU memory and has blocky appearance
\text{StarDist3D consumes large GPU memory and has blocky appearance}
StarDist3D consumes large GPU memory and has blocky appearance
\text{StarDist3D consumes large GPU memory and has blocky appearance}
Annotating Platynereis Data
\text{Annotating Platynereis Data}
Annotating Platynereis Data
\text{Annotating Platynereis Data}
Future directions
\text{Future directions}
Future directions
\text{Future directions}
Predict instance shapes on both live embryo and in-situ images at 16, 20 and 24 hours
\text{Predict instance shapes on both live embryo and in-situ images at 16, 20 and 24 hours}
Predict instance shapes on both live embryo and in-situ images at 16, 20 and 24 hours
\text{Predict instance shapes on both live embryo and in-situ images at 16, 20 and 24 hours}
Use morphology information in addition to geometric arrangement for matching
\text{Use morphology information in addition to geometric arrangement for matching}
Use morphology information in addition to geometric arrangement for matching
\text{Use morphology information in addition to geometric arrangement for matching}
Embryo at 16 hpf
\text{Embryo at 16 hpf}
Embryo at 16 hpf
\text{Embryo at 16 hpf}
Nuclei
\text{Nuclei}
Nuclei
\text{Nuclei}
Pax3/7
\text{Pax3/7}
Pax3/7
\text{Pax3/7}
Embryo at 20 hpf
\text{Embryo at 20 hpf}
Embryo at 20 hpf
\text{Embryo at 20 hpf}
Nuclei
\text{Nuclei}
Nuclei
\text{Nuclei}
Pax3/7
\text{Pax3/7}
Pax3/7
\text{Pax3/7}
See if the network can learn a neighborhood descriptor on its own
\text{See if the network can learn a neighborhood descriptor on its own}
See if the network can learn a neighborhood descriptor on its own
\text{See if the network can learn a neighborhood descriptor on its own}
Thank You for Listening!
\text{Thank You for Listening!}
Thank You for Listening!
\text{Thank You for Listening!}
Resume presentation
T o m a n c a k L a b , G r o u p M e e t i n g \text{Tomancak Lab, Group Meeting} D e c 1 5 , 2 0 2 0 \text{Dec 15, 2020}
Made with Slides.com
BESbswy
BESbswy
BESbswy
BESbswy