2. To recognize the text in cropped word image.

सेल्फी

जिला

भारत

Lightening Conditions

Varying orientation

Perspective distortion

Complex background

To solve any machine learning problem, we need

Before going on, 

1. Data

2. Model

4. Learning Algorithm

3. Parameters

5. Loss Functions

Specifically in case of Indian languages(Hindi), 

1. Data

1. Synthetic Dataset for Text Detection (Shubham's thesis)

2. Synthetic Dataset for Text Recognition(Hindi) (in this work)

3. For test images, we have manually clicked the pictures and annotate them!

Cropped word dataset generation

Things required:

Hindi Wiki book database

Google fonts

1. Text

2. Font

3. Color of text

4. Background images

5. Rendering method

Discussed on next slide

Freely available on internet

 using dictionary of background and foreground colors

Rendering method:

Unlike English, Unicode level information won't be sufficient to render text on image

We need glyphs level information in this case. Before that, we select, text, font, background. With the help of a few rendering libraries of Python, we were able to render text on the image.

we store glyphs and their positions in buffer

Till the buffer did not get empty, we keep rendering the glyphs on the surface and finally rendered surface is converted to the  image.

कि

ि

क

will be rendered as 

कि

glyphs will be क, ि with ordering between them

कि

Also applied distortion and noise to randomly selected 80% images

To solve any machine learning problem, we need

Before going into literature, 

1. Data

2. Model

4. Learning Algorithm

3. Parameters

5. Loss Functions

Specifically in case of Indian languages, 

1. Data

2. Model (Detection /Recognition)

Solutions of Text Detection

Regression based solutions

Segmentation based solutions

Groundtruth bounding box

Anchor box

text ?

text ?

Regression-based Model

• EAST: An Efficient and Accurate Scene Text Detector

It tries to directly predict word or text line predictions, which are further sent to Non Maximum Suppression (NMS)

Network

Loss

\[ = L_{scoremap} + \lambda_g L_{gemoetry} \]

 \[L_{scoremap} \] 

= balanced cross entropy between predicted

and ground truth score map

\[L_{geometry} \]

= IoU loss + 1- cos(angle_p, angle_gt)

Results

Implementational details

• 5M synthetic images are used for training
• Batch size  = 8, trained for 99k steps
• 2 K80 GPUs with 12 GB memory
• learning rate = 1e-4 (exponential decay)
• Adam optimizer
• Images resized 512 x 512

• 428 images are used for inference
• IoU threshold = 0.4

Training

Testing

*with IoU threshold as 0.4

Results

*with IoU threshold as 0.4

• We also experimented with different scales

Solutions of Text Detection

Segmentation based solutions

text ?

text ?

Segmentation based model

• Pixel link

• Every pixel has 8 neighbours.

• If a pixel and one of its neighbours belong to same instance, the link between them will be predicted as positive, else negative.

Unlike regression method, which performs 2 types of tasks,​

1. Text/non-text prediction.
2. location regression.

Here only text/non-text prediction is required. But 2 type of predictions are done:

1. Pixel prediction.
2. Link prediction.

positive pixel

negative pixel

positive link

negative link

Network

Loss

\[ = L_{link} + \lambda L_{pixel} \]

\[L_{link} \]

\[L_{pixel} \]

\[ \lambda \]

 = instance balanced cross entropy 

= 2

 = Cross entropy on link x sum of W for +ve and -ve links

* With base network as VGG-16

Implementational details

Results

• 5M synthetic images are used for training
• Batch size  = 8, trained for 99k steps
• 2 K80 GPUs with 12 GB memory
• learning rate = 1e-4 with momentum 0.9
• Images resized 512 x 512

• 428 images are used for inference
• IoU threshold = 0.4

Training

Testing

IoU threshold=0.4, link and pixel threshold=0.5

After finetuning on 300 real images,

Effect on changing pixel threshold

Effect on changing link threshold

* link thres=0.5, IoU thres=0.4

* pixel thres=0.8, IoU thres=0.4

Text Recognition

भारत

Unlike the relation between object detection and text detection, there are significant differences between object recognition and text recognition,

In text recognition, we have to predict the series of labels, that why this problem is posed as a sequence recognition problem.

भ + ा + र + त

Model

Vgg-16 as feature extractor

Map to sequence 

BLSTM

BLSTM

Transcription

जिला

per-frame predictions

label sequence

height normalized feature maps

feature maps

input image

Transcription

- is to find the label sequence with the highest probability conditioned on the per-frame predictions

How to find this conditional probability?

we need a mapping function which can map from output sequence to label sequence.

P(label seq|per-frame prediction)* = sum of P(output seq |per-frame prediction)

Such that the output seq can be mapped to label sequence.

The mapping function will map

ि ि ि-----जजज---लल----ा

जिला

So, the conditional probability will be framed as,

While decoding, we will return the output sequence with highest probability

*In reality, it is difficult to compute so we use forward-backward algorthm

Results

Implementational Details

Training

Testing

• 3M synthetic word images for training, 1M for validation.
• batch size = 32, trained for 63k steps, took 1 day.
• 3 K80 GPUS with 12 GB memory
• lr = 0.001, momentum=0.9

• 1740 real images have been used for testing

Total number of labels = 129

[ 128 (according to Unicode representation) + 1 (unknown label) ]

Qualitative Results

अधिनियम

कालोनी

स्वल्पाहार

नित्यानंद

विभागाध्यक्ष

मंगलमय

Mistakes

इटास्पी

इलेक्टनिक्स

अभिवान

इबलिंग

डेन

सर्वशेष्ठ

विशाम

अघिनर्थ

कों

पथम

रकूल

ज

दं

द

सं

Its not only model's mistake!

Distorted images

Possible future work

• An end-to-end trainable framework for detection and recognition.
• An approach to solve multilingual text detection and recognition.
• A better evaluation metric for text detection, since most of the current metric, is borrowed from object detection and in the object case, if some part is neglected, it won't make much difference.
• A lightweight model, which can easily be incorporated in mobile phones.