DeepTest:
Automated Testing of
Deep-Neural-Network-driven Autonomous Cars
Yuchi Tian, Kexin Pei, Suman Jana, Baishakhi Ray
Presentation by Breandan Considine
How do you test an arbitrary program?
\Huge{\mathcal{P}}
\Huge\mathcal{x_1}
\Huge{\mathcal{P(x_1)}}
\Huge{\mathcal{P}}
\Huge\mathcal{x_1}
\Huge{\mathcal{P(x_1)}}
\Huge{\mathcal{O(x_1)}}
\Huge{\stackrel{\checkmark}{=}}
\Huge{\mathcal{O}}
\Huge{\mathcal{P}}
\Huge{\mathcal{P(x_2)}}
\Huge{\mathcal{O(x_2)}}
\Huge{\stackrel{\checkmark}{=}}
\Huge\mathcal{x_2}
\Huge{\mathcal{O}}
\Huge{\mathcal{P}}
\Huge{\mathcal{P(x_3)}}
\Huge{\mathcal{O(x_3)}}
\Huge{\neq}
\Huge\mathcal{x_3}
\Huge{\mathcal{O}}
\Huge{\mathcal{O}}
\Huge{\mathcal{P}}
\Huge{\mathcal{P(x_3)}}
\Huge{\mathcal{O(x_3)}}
\Huge{\neq}
\Huge\mathcal{x_3}
Metamorphic Relations
\Huge{\mathcal{P}}
\huge{\mathcal{P(g(x_i))}}
\huge\mathcal{x_i}
\Huge{\mathcal{P}}
\huge\mathcal{g(x_i)}
\huge{\mathcal{P(x_i)}}
\mathcal{=,\neq, \subset, \subsetneq, ...}
Metamorphic Relations
S(A, B): \text{Returns shortest path from A to B}
\text{What are some necessary properties of} f?
\small{f(A, B) = f(B, A)}
\small{x \in S(A, B), S(A, x) + S(x, B) = S(A, B)}
\small{z \in G \setminus S(A, B), |S(A, B)| \leq |S(A, z) + S(z, B)|}
\small{x \in S(A, B), |S(A, x)| < |S(A, B)|}
\small{x \in S(A, B), S(A, x) \subset S(A, B)}
Metamorphic Relations
\Huge{\mathcal{S}}
\huge{S(B, A)}
\huge{A, B}
\Huge{\mathcal{S}}
\huge{B, A}
\huge{S(A, B)}
\Huge{\mathcal{=}}
Metamorphic Relations
\Huge{\mathcal{S}}
\huge{S(B, A)}
\huge{A, B}
\Huge{\mathcal{S}}
\huge{B, A}
\huge{S(A, B)}
\Huge{\mathcal{\neq}}
Neuron Coverage
\Huge{\frac{|\text{activated neurons}|}{|\text{total neurons}|}}
\Large{(\hat{\theta_i} - \theta_{ti})^2 \leq \lambda MSE_{origin}}
Metamorphic Relation
\Huge{\mathcal{N}}
\large{(\hat{\theta_i} - \mathcal{N}(T(i)))^2}
\large{i}
\Huge{\mathcal{N}}
\large{T(i)}
\Huge{<}
\large{(\hat{\theta_i} - \mathcal{N}(i))^2}
Metamorphic Relation
DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
By Breandan Considine
