AssistiveU

Assistive Technology for All

Problem Statement

India has a population of approximately 63 million hearing and speech-impaired people; building effective channels of communication is critical.

Solution: AssistiveU

AssistiveU aims to eradicate this problem by providing a system that can help speech impaired people by giving them a voice to speak, and hearing impaired people by helping them intercept whatever someone is speaking in the form of text.

Building blocks

The app employs lip-reading, an upcoming area of research in machine learning.

The architecture used (called LipNet) is a state-of-the-art model with an accuracy of 96.4% compared to the professional lip-reading accuracy of 53.4%.

The model was initially trained on the GRID corpus, and fine-tuned further using self-prepared dataset videos using transfer learning.

Detailed Description

This app provides:

1. Lip-reading for speech-impaired people so their lip movements are converted to text, which is then converted to speech;

2. Speech-to-text facility for hearing-impaired people so they can intercept anything spoken to them, and access it through text.

Tech Stack

1. Android

2. Flask

3. Tensorflow

4. Google APIs

5. Firebase

Future Scope