Deep Learning

&

Content-Based Image Retrieval

presented by:

Saeid Balaneshinkordan

based on:

Wan, Ji, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, and Jintao Li.

"Deep learning for content-based image retrieval: A comprehensive study."

In Proceedings of the ACM International Conference on Multimedia, pp. 157-166. ACM, 2014.

Content-based Image Retrieval (CBIR) System

definition

ref: http://www.cse.unsw.edu.au/~jas/talks/curveix/notes.html

Content-based Image Retrieval (CBIR) System

definition

Image Retrieval

concept based

content based

use text-based image retrieval techniques

use computer vision techniques

use content of image

use metadata of image

Content-based Image Retrieval (CBIR) System

Learning effective feature representations and similarity measures

challenge

semantic gap

low-level image pixels captured by machines

high-level semantic concepts perceived by human

Deep Learning

for

Content-based Image Retrieval (CBIR) System

b. examining Convolutional Neural Networks

1- Is deep learning a solution for bridging the semantic gap in CBIR

2- How much improvements can be achieved by exploring the deep learning techniques?

3- Which framework of deep learning to apply?

a. an extensive set of empirical studies

Deep Learning

for

Content-based Image Retrieval (CBIR) System

Shallow CBIR

System

Deep CBIR

System

Features from DB

Features from Query

DB raw data

Query raw data

similar images

similar images

instead: learn features at multiple level of abstracts from data automatically

no human-crafted features 

Distance metric Learning

definition

what is "metric"?

 is a metric if it satisfies the following properties:

Distance metric is a family of metrics that compute
squared distances as:

linear transformation

ref: Weinberger, Kilian Q., and Lawrence K. Saul. "Distance metric learning for large margin nearest neighbor classification." The Journal of Machine Learning Research 10 (2009): 207-244.

 contains a similar pair and a dissimilar pair

Distance metric Learning

1- pairwise constraints

2- triplet constraints

 must-link constraints and cannot-link constraints are given

types of training data

 3- class labels

directly using the class labels, following a typical machine learning scheme

Distance metric Learning

1- pairwise constraints

 must-link constraints and cannot-link constraints are given

types of training data

 must-link

cannot-link

ref: Lu, Zhengdong. "Semi-supervised clustering with pairwise constraints: A discriminative approach." In International Conference on Artificial Intelligence and Statistics, pp. 299-306. 2007.

 contains a similar pair and a dissimilar pair

Distance metric Learning

2- triplet constraints

types of training data

relevant pairs

set

irrelevant pairs

set

triplet constraint set

overall loss:

loss for a triplet

Distance metric Learning

types of training data

directly using the class labels, following a typical machine learning scheme

example: using Linear Discriminant Analysis (LDA)

maximizes 

between-class variance 

within-class variance

(

)

ref: Weinberger, Kilian Q., and Lawrence K. Saul. "Distance metric learning for large margin nearest neighbor classification." The Journal of Machine Learning Research 10 (2009): 207-244.

3- Class Labels

learn a metric on the local sense by only satisfying the given local constraints from neighboring information

Distance metric Learning

global supervised approaches:

local supervised approaches:

learn a metric on a global setting by satisfying all the constraints 

learning approaches

1- data is usually locally “well-behaved,” simpler region-classifiers often suffice for controlling local empirical error. 

2- complex boundaries for partitions can be approximated by piecewise linear functions

ref: Wang, Joseph, and Venkatesh Saligrama. "Local supervised learning through space partitioning." InAdvances in Neural Information Processing Systems, pp. 91-99. 2012.

 learning is done in a sequential order

can handle large-scale data 

Distance metric Learning

batch learning methods:

online learning methods:

assume the whole collection of training data must be given before the learning task and train a model from scratch.

learning methodology

CNN implementation

(a) Training deep CNN models in an existing domain (ImageNet)

(b) Adopting trained model for CBIR in a new domain

CNN implementation

cuda-convnet:

- fast C++/CUDA implementation of convolutional neural networks

- Training is done using the back-propagation algorithm.

- Fermi-generation GPU (GTX 4xx, GTX 5xx, or Tesla equivalent) required

- training dataset: “ILSVRC-2012” (from ImageNet with 1,000 categories and more than 1 million training images.)

CNN implementation

training dataset

ILSVRC2012: ImageNet Large Scale Visual Recognition Challenge 2012

estimate the content of photographs for the purpose of retrieval and automatic annotation

using a subset of the large hand-labeled ImageNet dataset (10,000,000 labeled images depicting 10,000+ object categories) as training.

The goal of this competition:

CNN implementation

Training deep CNN models in an existing domain (ImageNet)

 deep convolutional network consists of two parts:

1) the convolution layers and maxpooling layers, and

2) the fully connection layers and the output layers.

five convolutional layers:

The first and the second convolution layers are following with a response normalization layers and a max pooling layers 

The third, fourth, and fifth convolution layers are connected to one another without any intervening pooling or normalization.

1- a nonlinear function: Rectified Linear units (ReLUs)

2- can reduce the training time of the deep convolutional neural networks several times than the equivalents with “tanh”’ units. 

neuron output function f:

ref: http://cs231n.github.io/neural-networks-1/

- "FC1" and "FC2": two fully connected layers with 4, 096 neurons.

- "FC3": 1000-way softmax layer which produces a distribution over the 1, 000 class labels in ImageNet.

- 60 million parameters in total in the whole deep convolutional neural network

- training set:  ImageNet’s ILSVRC-2012, with 1.2 million images.

- It takes about 200 hours to train a model

- error rate: 0.424 over the validation set (50, 000 images)

experimental setup:

• Linux server
• NVIDIA Tesla K20 GPUs.
• 13 active SMXes (Streaming Multiprocessor)
• 5 memory controllers
• 1.25MB of L2 cache,
• 5GB of GDDR5 (Double Data Rate type five synchronous Graphics random access memory).

CNN implementation

 Adopting trained model for CBIR in a new domain

feature generalization schemes

scheme I:

DF.FC3: the feature taken from the final output layer,

DF.FC2: the features taken from the final hidden layer,

DF.FC1: the activations of the layer before DF.FC2.

adopt one of the activation features DF.FC1, DF.FC2, and DF.FC3, directly.

To obtain the feature representation:

• directly feed the images in new datasets into the input layer of the pre-trained CNN model
• take the activation values from the last three layers.

feature generalization schemes

scheme II:

DF.FC3+SL: the feature taken from the final output layer,

DF.FC2+SL: the features taken from the final hidden layer,

DF.FC1+SL: the activations of the layer before DF.FC2+SL.

- Construct the triplets by simply considering relationships of instances in same class as relevant (positive), and relationships of instances belonging to different classes as irrelevant (negative).

- Classifier training: LogReg or SVM algorithm and by focusing on object recognition

feature generalization schemes

scheme III:

ReCLS.FC3 & ReDSL.FC3 : the feature taken from the final output layer,

ReCLS.FC2 & ReDSL.FC2: the features taken from the final hidden layer,

ReCLS.FC1 & ReDSL.FC1: the activations of the layer before ReCLS.FC2  & ReDSL.FC2.

Initializing the CNN model with the parameters of the ImageNet-trained models.

Two approaches to retrain the CNN model:

1) Refining with class labels.

2) Refining with side information.

Image Datasets

general image database: “ImageNet”

object image database: “Caltech256”

landmark image datasets: “Oxford” and “Paris”

facial image dataset: “Pubfig83LFW”.

Results

ReCLS: Refining with class labels  (Scheme III)

DF: Direct Representation (Scheme I)

H-L-G: Hog-LBP-Gabor: fuses three kinds of famous facial image representation features

Pubfig83LFW: is an open-universe web facial image dataset, which combines two widely used face databases: PubFig83 and LFW

Thank You!