Inspecting Vision Models
Ahcène Boubekki
UCPH/Pioneer
Samuel G Matthiesen
Sebastian Mair
DTU
Linkøping
Motivation
Motivation
What did you see?
Where did you look?
Where were
the eyes?
How many eyes
were there?
How would you explain this image
using 5 concepts?
How would you decompose this image?
How to see what
a model sees?
How to see what a model sees?
RISE
XRAI
GradCAM
LRP
IG
Dingo or Lion
These do not answer directly
what does the model see?
Deep Dream?
What makes it more a tiger than a tiger?
Too slow, impossible to train, not really useful.
What is important for the prediction?
Inconsistent, difficult to read, objective unclear.
Saliency Maps?
How is the neighborhood in the embedding?
Inspection of the embedding, "biaised" justification..
Prototypes/Concepts?
Counterfactual?
What should I change to change class?
Tricky to compute, but nice!
How to see what a model sees?
Standard Image Classifier
Encoder
convolutions, pooling, non-linearity, skip-connections, attention, etc.
Classifier
Single linear layer... eventually a softmax
clustering
k=10
k=5
k=20
How to see what a model sees?
Seems like a déjà-vu?
One object at a time
Limited to 3 directions
For K=3,
PCA and k-means are similar
K-means provides
some hierarchy!
What can we
do with it?
What can we explain?
Connect Explanations and Semantics
Replace class-wise explanations
Concept Extraction
What can we explain?
Object Localization
NAS always improves MaxBox
but not all thresholded IoU
Always improves the most difficult IoU@70%
+
+
+
+
+
+
+
++
+
+
-
-
-
+
+
+
-
+
+
+
What can we explain?
ECG Explanation
What can we explain?
Annotation Masks
Model train for binary classif.
All lesions recovered
Can we use NAVE for medical annotations?
You need well performing model!
What can we explain?
Inspect Shortcuts Saturation
What else?
What else?
Repairing?
>>> Test set
>>> IMPUTATION: SWAP
>>> Accuracies
All LBL==0 LBL==1
Distribution 176 26.7 73.3
--------------------------------------
Nothing : 26.7 100.0 0.0
Watermark : 100.0 100.0 100.0
Wmk and Imput: 26.7 100.0 0.0
--------------------------------------What else?
Adversarial Attack?
Summary
Summary
Does it capture semantics?
Yes
Can we see what the model sees?
Yes
- Concept extraction
- Object localization
- Model inspection
- Annotations masks from labels
- Shortcut saturation
What can we do with it?
- Feature attribution?
- Causality?
- Concept injection?
- Any other idea?
What else?
Inspecting Vision Models
Ahcène Boubekki
UCPH
Samuel G Matthiesen
Sebastian Mair
DTU
Linkøping