Marker-based Inside-out Tracking for Medical Applications
Using a Single Optical Camera
Student: Chen-Hsuan, Shih (Iris)
Supervisor: Prof. Dr. Nassir Navab
Master of Biomedical Computing
Computer Aided Medical Procedure(CAMP),
Technical University of Munich, Germany
Advisor: Philipp Matthies
Motivation
In the context of computer-assisted interventions, object tracking is a key enable technology which allows the continuous localization of medical devices and patient anatomy.
It is a prerequisite for providing device guidance to subsurface anatomical structures.
Motivation
Outside-in Tracking
2
doctor
patient
nurse
line of sight problem ! -> Miss Tracking
Motivation
3
doctor
patient
doctor
patient
nurse
Motivation
4
Inside-out Tracking
-
State-of-The-Art
-
Tracking System
-
Multi-Tracker-App.
-
Experiment
-
Results & Evaluation
-
Discussion
-
Future Works
Road Map
5
Road Map
6
-
State-of-The-Art
-
Tracking System
-
Multi-Tracker-App.
-
Experiment
-
Results & Evaluation
-
Discussion
-
Future Works
State of the art tracking technology
Inside-out Tracking
Similar application for solving line-of-sight
State-of-the-Art: Marker-Based Tracking
Really Robustness in Occlusion ,Wrapping , Lens Distorsion
[1]Olson, E., AprilTag: A robust and flexible visual fiducial system, ICRA, 2011
[2] S. Garrido-Jurado, Automatic generation and detection of highly reliable fiducial markers under occlusion, Pattern Recognition, 2014
7
AprilTags [1]
ArUco [2]
Inside-out Tracking Application
8
ClearGuide [4]
Panorama ultrasound for
[4]Clear Guide Medical. http://clearguidemedical.com/ (accessed 3/6/2015).
[3] Hedyeh Rafii-Tari. Panorama ultrasound for guiding epidural anesthesia: a feasibility study. IPCAI’11 2011
guiding epidural anesthesia [3]
Marker-based tracking setup
Extract information from tracking
Multi-marker tracking model
Get the detector transformation matrix
Complete transformation for reconstruction
Road Map
9
-
State-of-The-Art
-
Tracking System
-
Multi-Tracker-App.
-
Experiment
-
Results & Evaluation
-
Discussion
-
Future Works
Technical Setup
GoPro Camera
Calibrator
3D Marker Cube
10
(Reference Target)
Gamma Camera
Marker-Based Tracking
Gamma Camera
Tracking System
?
Calibrator
patient
Image reconstruction
11
Cube Target
Tracking Transformation T1
camera --> marker cube
GoPro Camera
Gamma Camera
12
Tracking System
Cube Target
Tracking Transformation T2
camera --> calibrator
GoPro Camera
13
Tracking System
T1
Cube Target
Tracking Transformation T3 calibrator --> Detector
GoPro Camera
Gamma Camera
Tracking Transformation T3 calibrator --> Detector
T2
14
Tracking System
T1
Cube Target
GoPro Camera
Gamma Camera
T3
T1
T2
T3 x
T1
T2 x
Ttarget =
14
Tracking System
detector
-1
-1
Cube Target
GoPro Camera
Gamma Camera
T3
T1
T2
Cube Target
15
Tracking System
-1
patient
Volume of interest
Ttarget
detector
GoPro Camera
Gamma Camera
T3
T1
T2
T
Cube Target
16
Tracking System
-1
patient
Volume of interest
-1
Tdetector
voi
Multi-Marker Model + ArUco Tracking Framework
(voi)
Normal 2D Marker Tracking - Individual Coordinate
17
Tracking Model
Applied Multi-Marker Model
18
Tracking Model
s = 1/2 marker size
c = 1/2 cube size
Translation Extracted from Tracking Alg. with 3D Multi-Marker Model
19
3 markers were tracked:
red: #marker 1
green: #maker 2
blue: #marker 3
Tracking Model
20
Marker-Based Tracking
Calibrated Camera
Reference Points
(Intrinsic parameters)
Tcamera (Extrinsic parameters) :
Mean of all detected transformations (single markers)
Tcamera
target
target
ArUco
20
Marker-Based Tracking - ArUco
Input Image
Camera Captured
1
20
Marker-Based Tracking - ArUco
Input Image
Camera Captured
1
2
Gray-Scale Image
Image Segmentation
20
Marker-Based Tracking - ArUco
Input Image
Camera Captured
1
2
Gray-Scale Image
Image Segmentation
3
Contour extraction
20
Marker-Based Tracking - ArUco
Input Image
Camera Captured
1
2
3
Contour extraction
Filter Polygon
4
4-vertex Polygon
Gray-Scale Image
Image Segmentation
20
Marker-Based Tracking - ArUco
Input Image
Camera Captured
1
2
3
Contour extraction
Filter Polygon
4
4-vertex Polygon
Gray-Scale Image
Image Segmentation
5
Marker Identification
Perspective Reprojection
Binarized Image :
0: black, 1: white
Road Map
21
Inside-out Tracking & Outside-in Tracking
Reconstruction Tool
AR Visualization
-
State-of-The-Art
-
Tracking System
-
Multi-Tracker-App.
-
Experiment
-
Results & Evaluation
-
Discussion
-
Future Works
Multi-Tracker-App.
22
Multi-Tracker-App.
23
Threshold Setting
Reconstruction Tool
GammaProbe Acquisition
Augmented Reality Visualization
1
3
2
Switch View
Multi-Tracker-App.
23
Threshold Setting
Reconstruction Tool
GammaProbe Acquisition
Augmented Reality Visualization
1
3
2
Road Map
24
Lymph node phantom
Thyroid phantom
Breast cancer patient scan
-
State-of-The-Art
-
Tracking System
-
Multi-Tracker-App.
-
Experiment
-
Results & Evaluation
-
Discussion
-
Future Works
System Setup - Breast lymph node phantom scan
25
Inside-out Tracking
Outside-in Tracking (NDI Tracker)
System Setup - Breast lymph node phantom scan
25
Two hollow spheres in a water-bath:
-
volumes of 3:5ml & 0:6ml each
-
filled with a 3MBq solution of 99mTc
System Setup - Thyroid phantom scan
26
Inside-out Tracking
Outside-in Tracking (NDI Tracker)
System Setup - Thyroid phantom scan
26
-
simulate hot or cold nodules
Custom-printed thyroid phantom :
-
contains 4 chambers of different sizes
filled with radioactivity or water
Inside-out Workflow
27
Multi-Tracker-App.
28
-
System Setup
-
Gamma camera start acquisition
-
Switching view to make sure the tracking validation
Multi-Tracker-App.
28
-
System Setup
-
Gamma camera start acquisition
-
Switching view to make sure the tracking validation
-
Stop acquisition
-
Choose inside-out tracker info. for reconstruction
-
Thresholding the visualization
29
Road Map
-
State-of-The-Art
-
Tracking System
-
Multi-Tracker-App.
-
Experiment
-
Results & Evaluation
-
Discussion
-
Future Works
Inside-out Tracking vs. Outside-in Tracking
Reconstruction & AR
22
Video!!!!!
Phantom Experiment Result
30
Real distance: 5.5cm
Real distance: 3.3cm
Real distance: 5.5cm
Real distance: 3.3cm
Phantom Experiment Result
30
Mean 5.45 5.75
Mean 3.22 3.26
Patient Experiment Result
31
32
Tracking Accuracy
ArUco (Guppy) vs. Outside-in (NDI)
Distance between two sets of tracking points
Marker cube limitation
33
Multi-Marker Model Accuracy
34
35
Road Map
-
State-of-The-Art
-
Tracking System
-
Multi-Tracker-App.
-
Experiment
-
Results & Evaluation
-
Discussion
-
Future Works
Inside-out Tracking vs. Outside-in Tracking
Reconstruction & AR
Discussion
Reduced Line-of-sight issue !
Reference target (marker cube) can be easily positioned & tracked!
Simple, Easy, Low cost system setup with comparable accuracy!
Patient & surgeon friendly
36
Discussion
How to deal: operator's handshake that leads to bad tracking?
What are the constrains for the reference target? Is it necessary?
Can the system auto-detect the error if happened during real scan?
Is the system reliably applicable in real clinical practice?
37
38
Road Map
-
Literature research
-
Tracking System
-
Multi-Tracker-App.
-
Experiment
-
Evaluation & Results
-
Future Works
Tracking Accuracy & Stabilization
Mechanical Tracking as ground truth
Next Steps...
Marker-based Tracking:
-
Tracking Algorithm
-
Accuracy assurance & Standard deviation
-
Discard outliers : Error threshold
-
Compare to ground truth. (e.g. Mechanical tracking)
-
39
Next Steps...
Marker-based Tracking:
-
Tracking Algorithm
-
Accuracy assurance & Standard deviation
-
Discard outliers : Error threshold
-
Compare to ground truth. (e.g. Mechanical tracking)
-
-
Reference Marker Design
-
Size & Positioning
-
Cube vs. Plane
-
40
Next Steps...
Reference Marker Design : Cube vs. Plane
41
Next Steps...
Inside-out Tracking System:
-
Hardware setup
-
Dedicated combined optical-/gamma-camera
-
Custom printed the marker cube
-
Dedicated holder for marker cube
-
-
Clinical workflow
-
Address clinical integration issue
-
42
Next Steps...
Inside-out Tracking System:
-
Decomposed Transformation vs. Projection Matrix
-
Motion Tracking without Markers:
-
Project Tango from Google
-
Direct SLAM for RGB-D Cameras
-
43
42
Thank You for Your Attention!
Thank you Philipp & Thank you Prof. Navab
Thanks for a lot of people who support this project!
Master Thesis Final Presentation
By iriscshih
Master Thesis Final Presentation
- 91