Video Communication Demo

空拍影像偵測移動物體

  1. A. Hafiane, K. Palaniappan and G. Seetharaman, "UAV-Video Registration Using Block-Based Features," IGARSS 2008 - 2008 IEEE International Geoscience and Remote Sensing Symposium, Boston, MA, 2008, pp. II-1104-II-1107.
     
  2. Q. Wei, S. Lao and L. Bai, "Panorama Stitching, Moving Object Detection and Tracking in UAV Videos," 2017 International Conference on Vision, Image and Signal Processing (ICVISP), Osaka, 2017, pp. 46-50.

Referenced
Papers

  1. Problem definitions
    • UAV Streaming
    • Find homography
  2. Algorithms
    • Find feature points
    • Match feature points
    • Estimate homography
    • Detect targets
  3. Experimental results
    • PSNR plots
    • Moving object detections
  4. Discussions
    • True rejects from FLANN
    • False accepts from differences
    • Could be faster

Detailed
Slide MAps

Problem
Definitions

Situations:
UAV Streaming

SIMPLIFy:
Find Homography

homography

Algorithms

Steps

  1. Find feature points
    • SIFT
    • Beltrami
  2. Match feature points
    • FLANN
    • Motion vectors
  3. Estimate homography
    • RANSAC + DLT
  4. Detect targets
    • Difference methods

。

。

。

。

。

。

Step 1:
Find Feature Points

SIFT

Beltrami

R=\text{tr}(\mathrm{H})^2/\det(\mathrm{H}),\mathrm{H}=\begin{bmatrix} \frac{\partial^2I}{\partial x^2}& \frac{\partial^2I}{\partial x\partial y}\\ \frac{\partial^2I}{\partial x\partial y}& \frac{\partial^2I}{\partial y^2} \end{bmatrix}
R=tr(H)2/det(H),H=[∂2I∂x2∂2I∂x∂y∂2I∂x∂y∂2I∂y2]R=\text{tr}(\mathrm{H})^2/\det(\mathrm{H}),\mathrm{H}=\begin{bmatrix} \frac{\partial^2I}{\partial x^2}& \frac{\partial^2I}{\partial x\partial y}\\ \frac{\partial^2I}{\partial x\partial y}& \frac{\partial^2I}{\partial y^2} \end{bmatrix}
B=1 + \text{tr}(\mathrm{A})+\det(\mathrm{A}),\mathrm{A}=\begin{bmatrix} \big(\frac{\partial I}{\partial x}\big)^2& \frac{\partial I}{\partial x}\frac{\partial I}{\partial y}\\ \frac{\partial I}{\partial x}\frac{\partial I}{\partial y}& \big(\frac{\partial I}{\partial y}\big)^2 \end{bmatrix}
B=1+tr(A)+det(A),A=[(∂I∂x)2∂I∂x∂I∂y∂I∂x∂I∂y(∂I∂y)2]B=1 + \text{tr}(\mathrm{A})+\det(\mathrm{A}),\mathrm{A}=\begin{bmatrix} \big(\frac{\partial I}{\partial x}\big)^2& \frac{\partial I}{\partial x}\frac{\partial I}{\partial y}\\ \frac{\partial I}{\partial x}\frac{\partial I}{\partial y}& \big(\frac{\partial I}{\partial y}\big)^2 \end{bmatrix}

Step 1:
Find Feature Points

Beltrami + Prominent feature blocks

。

。

。

。

。

。

。

。

。

。

Step 2:
Match Feature Points

Motion vectors

FLANN

  1. priority search k-means tree
  2. multiple randomized k-d trees
  1. minimal SAD
     
  2. maximal NCC
\displaystyle\arg\max_{\Delta X}\bigg| \dfrac{\mathrm{W}_{X+\Delta X}\cdot\mathrm{W}_X} {\big\|\mathrm{W}_{X+\Delta X}\big\|\big\|\mathrm{W}_X\big\|}\bigg|
argmaxΔX∣WX+ΔX⋅WX∥WX+ΔX∥∥WX∥∣\displaystyle\arg\max_{\Delta X}\bigg| \dfrac{\mathrm{W}_{X+\Delta X}\cdot\mathrm{W}_X} {\big\|\mathrm{W}_{X+\Delta X}\big\|\big\|\mathrm{W}_X\big\|}\bigg|
\displaystyle\arg\min_{\Delta X}\sum|\mathrm{W}_{X+\Delta X}-\mathrm{W}_X|
argminΔX∑∣WX+ΔX−WX∣\displaystyle\arg\min_{\Delta X}\sum|\mathrm{W}_{X+\Delta X}-\mathrm{W}_X|

Step 3:
Estimate Homography

\displaystyle \mathrm{\hat{h}_\Omega}=\arg\min_\mathrm{h}\|\mathrm{A_\Omega h}\|^2\quad\text{w.r.t.}~\|\mathrm{h}\|=1
h^Ω=argminh∥AΩh∥2w.r.t. ∥h∥=1\displaystyle \mathrm{\hat{h}_\Omega}=\arg\min_\mathrm{h}\|\mathrm{A_\Omega h}\|^2\quad\text{w.r.t.}~\|\mathrm{h}\|=1

RANSAC + DLT

\mathrm{A}_\Omega=\begin{bmatrix} 0_{1\times3}&-\mathrm{a}_1^\intercal&y'_1\mathrm{a}_1^\intercal\\ \mathrm{a}_1^\intercal&0_{1\times3}&-x'_1\mathrm{a}_1^\intercal\\ \vdots&\vdots&\vdots \end{bmatrix}, \mathrm{a}_i=\begin{bmatrix} x_i\\y_i\\\mathrm{1} \end{bmatrix}, \mathrm{h}=\begin{bmatrix} \mathrm{H_1}\\\mathrm{H_2}\\\mathrm{H_3} \end{bmatrix}
AΩ=[01×3−a1⊺y1′a1⊺a1⊺01×3−x1′a1⊺⋮⋮⋮],ai=[xiyi1],h=[H1H2H3]\mathrm{A}_\Omega=\begin{bmatrix} 0_{1\times3}&-\mathrm{a}_1^\intercal&y'_1\mathrm{a}_1^\intercal\\ \mathrm{a}_1^\intercal&0_{1\times3}&-x'_1\mathrm{a}_1^\intercal\\ \vdots&\vdots&\vdots \end{bmatrix}, \mathrm{a}_i=\begin{bmatrix} x_i\\y_i\\\mathrm{1} \end{bmatrix}, \mathrm{h}=\begin{bmatrix} \mathrm{H_1}\\\mathrm{H_2}\\\mathrm{H_3} \end{bmatrix}
\displaystyle \hat{\Omega}=\arg_\Omega\dfrac{|\Omega|}{N}>\gamma \quad\text{w.r.t.}~\forall\mathrm{a}_i\in\Omega.~d(\mathrm{a}'_i,\mathrm{\hat{H}_\Omega\mathrm{a}}_i)<\epsilon
Ω^=argΩ∣Ω∣N>γw.r.t. ∀ai∈Ω. d(ai′,H^Ωai)<ϵ\displaystyle \hat{\Omega}=\arg_\Omega\dfrac{|\Omega|}{N}>\gamma \quad\text{w.r.t.}~\forall\mathrm{a}_i\in\Omega.~d(\mathrm{a}'_i,\mathrm{\hat{H}_\Omega\mathrm{a}}_i)<\epsilon

Step 3:
Estimate Homography

RANSAC + DLT

。

。

。

。

。

。

\quad,\quad\in\hat\Omega, \quad\notin\hat\Omega
,∈Ω^,∉Ω^\quad,\quad\in\hat\Omega, \quad\notin\hat\Omega
\mathrm{a}_1
a1\mathrm{a}_1
\mathrm{a}_2
a2\mathrm{a}_2
\mathrm{a}_3
a3\mathrm{a}_3
\quad=\mathrm{\hat{H}}\quad
=H^\quad=\mathrm{\hat{H}}\quad
\mathrm{a}'_2
a2′\mathrm{a}'_2
\mathrm{a}_2
a2\mathrm{a}_2
\quad=\mathrm{\hat{H}}\quad
=H^\quad=\mathrm{\hat{H}}\quad
\mathrm{a}'_3
a3′\mathrm{a}'_3
\mathrm{a}_3
a3\mathrm{a}_3

Step 4:
Detect Targets

Difference methods

\begin{aligned} =&\\\land& \end{aligned}
=∧\begin{aligned} =&\\\land& \end{aligned}
(D_{13}\lor D_{35})
(D13∨D35)(D_{13}\lor D_{35})
(D_{23}\lor D_{34})
(D23∨D34)(D_{23}\lor D_{34})
D
DD

Experimental
Results

Demo Video

PSNR &
Speeds

SIFT+FLANN Block-based with FFmpeg
19.22 dB 19.07 dB 19.47 dB
8.43 fps 1.20 fps 3.01 fps

Detections

1

2 

3

Discussions

True Rejects
From RANSAC

\displaystyle \hat{\Omega}=\arg_\Omega\dfrac{|\Omega|}{N}>\gamma \quad\text{w.r.t.}~\forall\mathrm{a}_i\in\Omega.
Ω^=argΩ∣Ω∣N>γw.r.t. ∀ai∈Ω.\displaystyle \hat{\Omega}=\arg_\Omega\dfrac{|\Omega|}{N}>\gamma \quad\text{w.r.t.}~\forall\mathrm{a}_i\in\Omega.
d(\mathrm{a}'_i,\mathrm{\hat{H}_\Omega\mathrm{a}}_i)<\epsilon
d(ai′,H^Ωai)<ϵd(\mathrm{a}'_i,\mathrm{\hat{H}_\Omega\mathrm{a}}_i)<\epsilon

False Accepts
From Differences

Solution to Errors

  1. As-projective-as-Possible



     
  2. Track from unreliable detections

Filter outliers

Nonlinear transform

p=0.9
p=0.9p=0.9
p=0.1
p=0.1p=0.1

Insufficient Speed

             ╔═════ main.py ═════╗
▶ frame ╾─┐  ║                   ║
          ├──╫───╼ parser.py     ║
▶  MVs  ╾─┘  ║         ┊         ║
 (1-thread)  ║  homographier.py  ║
             ║         ┊         ║
             ║    detector.py ╾──╫──╼ output ▶
             ║                   ║
             ╚═══════════════════╝

Insufficient Speed

  • Total time
    • [14%] Read frames and motion vectors
      • [48%] Parse motion vector info
    • [76%] Find homography
      • [78%] Find unknown motion vectors
    • [9%] Detect moving objects
 $ python3 -m cProfile main.py
Made with Slides.com