Weekly Chat
\text{Weekly Chat}
March 2, 2021
\text{March 2, 2021}
Fourier Shape Descriptor
\text{Fourier Shape Descriptor}
SORT
\text{SORT}
3d Visualization
\text{3d Visualization}
Graph Matching
\text{Graph Matching}
MitoEM
\text{MitoEM}
Fourier Shape Descriptor
\text{Fourier Shape Descriptor}
Fourier Shape Descriptor
\text{Fourier Shape Descriptor}
Fourier Shape Descriptor - idea is ...
\text{Fourier Shape Descriptor - idea is ...}
6×4 predictions p.p
6 \times 4 \text{ predictions p.p}
Fourier Shape Descriptor - idea is ...
\text{Fourier Shape Descriptor - idea is ...}
6×4 predictions p.p
6 \times 4 \text{ predictions p.p}
IDFT
\text{IDFT}
x & y locations in cartesian space
\text{x \& y locations in cartesian space}
Fourier Shape Descriptor - idea is ...
\text{Fourier Shape Descriptor - idea is ...}
6×4 predictions p.p
6 \times 4 \text{ predictions p.p}
IDFT
\text{IDFT}
x & y locations in cartesian space
\text{x \& y locations in cartesian space}
differentiable IOU loss
\text{differentiable IOU loss}
Fourier Shape Descriptor - idea is ...
\text{Fourier Shape Descriptor - idea is ...}
6×4 predictions p.p
6 \times 4 \text{ predictions p.p}
IDFT
\text{IDFT}
x & y locations in cartesian space
\text{x \& y locations in cartesian space}
differentiable IOU loss
\text{differentiable IOU loss}
translation invariance loss
\text{translation invariance loss}
Fourier Shape Descriptor - idea is ...
\text{Fourier Shape Descriptor - idea is ...}
6×4 predictions p.p
6 \times 4 \text{ predictions p.p}
IDFT
\text{IDFT}
x & y locations in cartesian space
\text{x \& y locations in cartesian space}
differentiable IOU loss
\text{differentiable IOU loss}
translation invariance loss
\text{translation invariance loss}
foreground loss
\text{foreground loss}
Fourier Shape Descriptor - idea is end-to-end training
\text{Fourier Shape Descriptor - idea is end-to-end training}
6×4 predictions p.p
6 \times 4 \text{ predictions p.p}
IDFT
\text{IDFT}
x & y locations in cartesian space
\text{x \& y locations in cartesian space}
differentiable IOU loss
\text{differentiable IOU loss}
translation invariance loss
\text{translation invariance loss}
foreground loss
\text{foreground loss}
Fourier Shape Descriptor - questions
\text{Fourier Shape Descriptor - questions}
How to compute interior area of points defining contour in differentiable manner?
\text{How to compute interior area of points defining contour in differentiable manner?}
Are there unique solutions which the network can achieve during training?
\text{Are there unique solutions which the network can achieve during training?}
Fourier Shape Descriptor
\text{Fourier Shape Descriptor}
SORT
\text{SORT}
3d Visualization
\text{3d Visualization}
Graph Matching
\text{Graph Matching}
MitoEM
\text{MitoEM}
SORT
\text{SORT}
T=0
\text{T=0}
50
\text{50}
100
\text{100}
150
\text{150}
200
\text{200}
250
\text{250}
350
\text{350}
300
\text{300}
600
\text{600}
SORT
\text{SORT}
Scheme Three
\text{Scheme Three}
T=0
\text{T=0}
T=1
\text{T=1}
T=2
\text{T=2}
Hard Assignment
\text{Hard Assignment}
Real Time
\text{Real Time}
Local Optima
\text{Local Optima}
Correct for mitosis - conservation of mass!
\text{Correct for mitosis - conservation of mass!}
...
\text{...}
...
\text{...}
p1∗
p^{*}_{1}
p2∗
p^{*}_{2}
p3∗
p^{*}_{3}
p4∗
p^{*}_{4}
p...∗
p^{*}_{...}
pM∗
p^{*}_{M}
q2
q_{2}
q3
q_{3}
q4
q_{4}
qN
q_{N}
q...
q_{...}
4.1
\text{4.1}
2.6
\text{2.6}
11.3
\text{11.3}
89.7
\text{89.7}
14.2
\text{14.2}
90.8
\text{90.8}
64.2
\text{64.2}
7.8
\text{7.8}
16.9
\text{16.9}
70.8
\text{70.8}
16.7
\text{16.7}
32.4
\text{32.4}
9.7
\text{9.7}
15.4
\text{15.4}
8.3
\text{8.3}
20.1
\text{20.1}
11.6
\text{11.6}
14.9
\text{14.9}
18.1
\text{18.1}
3.7
\text{3.7}
24.1
\text{24.1}
36.4
\text{36.4}
12.2
\text{12.2}
9.5
\text{9.5}
14.2
\text{14.2}
17.5
\text{17.5}
23.45
\text{23.45}
6.7
\text{6.7}
19.9
\text{19.9}
22.1
\text{22.1}
14.8
\text{14.8}
16.4
\text{16.4}
63.7
\text{63.7}
21.5
\text{21.5}
81.1
\text{81.1}
9.6
\text{9.6}
q1
q_{1}
to obtain one-to-one matching
\textit{to obtain one-to-one matching }
Cij:=C(pi∗,qj)
C_{ij} := C(p^{*}_{i}, q_{j})
X^=Xarg min∑i=1M∑j=1NCijXij
\hat{X}= \underset{X}{\text{arg min}} \sum_{i=1}^{M} \sum_{j=1}^{N} C_{ij} X_{ij}
where Xij∈{0,1}
\text{ where } X_{ij} \in \{0, 1\}
s.t.∑k=1k=M1≤Xik≤2
\text{ s.t.} \sum_{k=1}^{k=M} 1 \leq X_{ik} \leq 2
∑k=1k=NXkj=1
\sum_{k=1}^{k=N} X_{kj} = 1
Apply Hungarian Algorithm
\textit{Apply Hungarian Algorithm}
IOU btw predicted kalman states and actual target states
\text{IOU btw predicted kalman states and actual target states}
Correct for mitosis - conservation of mass!
\text{Correct for mitosis - conservation of mass!}
...
\text{...}
...
\text{...}
p1∗
p^{*}_{1}
p2∗
p^{*}_{2}
p3∗
p^{*}_{3}
p4∗
p^{*}_{4}
p...∗
p^{*}_{...}
pM∗
p^{*}_{M}
q2
q_{2}
q3
q_{3}
q4
q_{4}
qN
q_{N}
q...
q_{...}
4.1
\text{4.1}
2.6
\text{2.6}
11.3
\text{11.3}
89.7
\text{89.7}
14.2
\text{14.2}
90.8
\text{90.8}
64.2
\text{64.2}
7.8
\text{7.8}
16.9
\text{16.9}
70.8
\text{70.8}
16.7
\text{16.7}
32.4
\text{32.4}
9.7
\text{9.7}
15.4
\text{15.4}
8.3
\text{8.3}
20.1
\text{20.1}
11.6
\text{11.6}
14.9
\text{14.9}
18.1
\text{18.1}
3.7
\text{3.7}
24.1
\text{24.1}
36.4
\text{36.4}
12.2
\text{12.2}
9.5
\text{9.5}
14.2
\text{14.2}
17.5
\text{17.5}
23.45
\text{23.45}
6.7
\text{6.7}
19.9
\text{19.9}
22.1
\text{22.1}
14.8
\text{14.8}
16.4
\text{16.4}
63.7
\text{63.7}
21.5
\text{21.5}
81.1
\text{81.1}
9.6
\text{9.6}
q1
q_{1}
to obtain one-to-one matching
\textit{to obtain one-to-one matching }
Cij:=C(pi∗,qj)
C_{ij} := C(p^{*}_{i}, q_{j})
X^=Xarg min∑i=1M∑j=1NCijXij
\hat{X}= \underset{X}{\text{arg min}} \sum_{i=1}^{M} \sum_{j=1}^{N} C_{ij} X_{ij}
where Xij∈{0,1}
\text{ where } X_{ij} \in \{0, 1\}
s.t.∑k=1k=M1≤Xik≤2
\text{ s.t.} \sum_{k=1}^{k=M} 1 \leq X_{ik} \leq 2
∑k=1k=NXkj=1
\sum_{k=1}^{k=N} X_{kj} = 1
Apply Hungarian Algorithm
\textit{Apply Hungarian Algorithm}
IOU btw predicted kalman states and actual target states
\text{IOU btw predicted kalman states and actual target states}
Future: incl apppearance vector from EmbedSeg
\text{Future: incl apppearance vector from EmbedSeg}
Fourier Shape Descriptor
\text{Fourier Shape Descriptor}
SORT
\text{SORT}
3d Visualization
\text{3d Visualization}
Graph Matching
\text{Graph Matching}
MitoEM
\text{MitoEM}
Graph Matching
\text{Graph Matching}
Graph Matching
\text{Graph Matching}
Jointly train EmbedSeg on multi modalities
\text{Jointly train EmbedSeg on multi modalities}
Same 12 landmarks are annotated in fixed and live embryo
\text{Same 12 landmarks are annotated in fixed and live embryo}
which means during training we can sample either two fixed embryos
\text{which means during training we can sample either two fixed embryos}
or two live embryos
\text{or two live embryos}
or a fixed and a live embryo
\text{or a fixed and a live embryo}
or the complete set of GT matches using fixed and simulated live
\text{or the complete set of GT matches using fixed and simulated live}
Graph Matching
\text{Graph Matching}
Jointly train EmbedSeg on multi modalities
\text{Jointly train EmbedSeg on multi modalities}
Same 12 landmarks are annotated in fixed and live embryo
\text{Same 12 landmarks are annotated in fixed and live embryo}
which means during training we can sample either two fixed embryos
\text{which means during training we can sample either two fixed embryos}
or two live embryos
\text{or two live embryos}
or a fixed and a live embryo
\text{or a fixed and a live embryo}
or the complete set of GT matches using fixed and simulated live
\text{or the complete set of GT matches using fixed and simulated live}
currently trying to consolidate and extend these 12 landmark pairs
\text{currently trying to consolidate and extend these 12 landmark pairs}
Fourier Shape Descriptor
\text{Fourier Shape Descriptor}
SORT
\text{SORT}
3d Visualization
\text{3d Visualization}
Graph Matching
\text{Graph Matching}
MitoEM
\text{MitoEM}
3d Visualization
\text{3d Visualization}
Create VTK models for all objects
\text{Create VTK models for all objects}
Figure out how to not introduce rendering artifacts in gray-scale image
\text{Figure out how to not introduce rendering artifacts in gray-scale image}
Provide link on project page for users to interact
\text{Provide link on project page for users to interact}
Fourier Shape Descriptor
\text{Fourier Shape Descriptor}
SORT
\text{SORT}
3d Visualization
\text{3d Visualization}
Graph Matching
\text{Graph Matching}
MitoEM
\text{MitoEM}
MitoEM
\text{MitoEM}
1024
\text{1024}
1024
\text{1024}
MitoEM
\text{MitoEM}
MitoEM
\text{MitoEM}
+
+
+
+
∗
*
∗
*
aff(
\text{aff(}
+
+
+
+
)
\text{)}
MitoEM
\text{MitoEM}
+
+
+
+
∗
*
aff(
\text{aff(}
+
+
+
+
)
\text{)}
+
+
+
+
∗
*
∗
*
aff(
\text{aff(}
+
+
)
\text{)}
+
+
MitoEM
\text{MitoEM}
+
+
+
+
∗
*
aff(
\text{aff(}
+
+
+
+
)
\text{)}
+
+
+
+
∗
*
∗
*
aff(
\text{aff(}
+
+
)
\text{)}
+
+
aff(
\text{aff(}
+
+
)
\text{)}
+
+
?
\text{?}
How doe we combine affinities
\text{How doe we combine affinities }
to get pixel-wise affinities for whole image
\text{to get pixel-wise affinities for whole image}
W e e k l y C h a t \text{Weekly Chat} M a r c h 2 , 2 0 2 1 \text{March 2, 2021}