Lip Reading using Deep Learning

Human's can lip-read when they cannot hear properly. 

We tried to figure out a logical correlation between how a person sounds and how his lips move.

We developed an Audio-Visual recognition system using coupled 3D ConvNets that is used for speech recognition when the audio is corrupted and not recognizable

The main task is to determine if a short duration of audio belongs to the same movement of lips.

Architechture

Audio feature extraction

Results

INPUT

OUTPUT