LuisaMZintgraf, Taco S Cohen, Tameem Adel, MaxWelling
International Conference on Learning Representations 2017
University of Amsterdam, Canadian Institute of Advanced Research, Vrije Universiteit Brussel
15 Feb 2017
Visualize the response of a deep neural network to a specific input
Two approaches for understanding DCNNs through visualization:
Explaining Classifications for Individual Instances
Marko Robnik-Šikonja and Igor Kononenko,
Knowledge and Data Engineering, IEEE Transactions, 2008
Overview
Approach
Experiment
Conclusion & Future Work
Prediction Difference Analysis
1
0
1
1
1
0
1
Measure how the prediction changes if the feature is unknown
Define:
The probability of O=c given:
all input features
all input features except
Prediction Difference Analysis
Relevance of
is easy to compute
How to compute ?
⇒ Simulate the absence of a feature by marginalizing the feature:
Prediction Difference Analysis
How about ? It's infeasible with a large number of features
Assume that is independent of :
is usually approximated by the empirical distribution
(However, an image is not the case)
Prediction Difference Analysis
To compare with , we define
Weight of evidence:
is similar to the inverse of sigmoid function
Avoid 0 probabilities
⇒ Laplace correction:
N: number of training instances
K: number of classes
where
Prediction Difference Analysis
Positive value:
The feature contributes evidence for the class of interest
⇒ removing it would decrease the confidence of the classifier in the given class
Negative value:
The feature displays evidence against the class
⇒ removing it also removes potentially conflicting or irritating information and the classifier becomes more certain in the investigated class
The feature means of this specific input
know all the feature
know all the features except
Conditional Sampling
A pixel’s value is highly dependent on other pixels
Two observations:
Conditional Sampling
For a pixel , we can find a patch of size that contains , and condition on the remaining pixels in that patch:
For a feature to become relevant, it now has to satisfy two conditions:
Multivariate Analysis
We expect that a neural network is relatively robust to just one feature of a high-dimensional input being unknown
⇒ remove several features at once:
Take the average WE, which is accumulated in the pixel in the overlapping patches
Remove a patch and calculate WEs of pixels in this patch
Deep Visualization of Hidden Layers
We can adapt the method to see how the units of any layer of the network influence a node from a deeper layer
: the vector representation of the output values in a layer H
: the value of a node that depends on
We are not dealing with probabilities, so WE is not applicable
Instead, we use activation difference:
Pseudocode
How to implement it?
Implementing Conditional Sampling
Idea:
Assume pixels in the patch is a multivariate normal distribution
Implement :
Implementing Conditional Sampling
But we want
⇒ conditional distribution of multivariate normal distribution
proof (an error in the formula of covariance matrix): http://fourier.eng.hmc.edu/e161/lectures/gaussianprocess/node7.html
Draw samples from distribution
Two parts:
Sampling method
ImageNet: Understanding How a DCNN Makes Decisions
Use images from the ILSVRC challenge:
a large dataset of natural images from 1000 categories
Three DCNNs, use pre-trianed models, implemented by caffe
Analysis time:
Marginal v.s. Conditional Sampling
For the rest of our experiments, we show results using conditional sampling only
Multivariate Analysis
Comparison with Sensitivity Map
Comparison with Sensitivity Map
Deep Visualization of Hidden Network Layers
For each feature map in a convolutional layer:
Deep Visualization of Hidden Network Layers
Simple image filters
Deep Visualization of Hidden Network Layers
specialized to higher level features
Deep Visualization of Hidden Network Layers
units are highly specialized
Deep Visualization of Hidden Network Layers
Same layer, different input image
Penultimate v.s. Output Layer
Penultimate v.s. Output Layer
Network Comparison
MRI Data: Explaining Classifier Decisions in Medical Imaging
When classifying a patient, the practitioner weigh this information and incorporate it into the overall diagnosis process
MRI dataset of HIV and healthy patients
L2-regularized Logistic Regression classifier (model details unknown)
MRI Data: Explaining Classifier Decisions in Medical Imaging
weights of the logistic regression classifier
(noisier, scattered)
prediction difference
Relevance values are shown only for voxels with value above 15% of the absolute maximum value
MRI Data: Explaining Classifier Decisions in Medical Imaging
Different class
MRI Data: Explaining Classifier Decisions in Medical Imaging
Different patch size
Future work
Contributions