AssistiveU

Assistive Technology for All

Problem Statement

India has a population of approximately 63 million hearing and speech-impaired people; building effective channels of communication is critical.

Solution: AssistiveU

AssistiveU aims to eradicate this problem by providing a system that can help speech impaired people by giving them a voice to speak, and hearing impaired people by helping them intercept whatever someone is speaking in the form of text.

Building blocks

The app employs lip-reading, an upcoming area of research in machine learning.

The architecture used (called LipNet) is a state-of-the-art model with an accuracy of 96.4% compared to the professional lip-reading accuracy of 53.4%.

The model was initially trained on the GRID corpus, and fine-tuned further using self-prepared dataset videos using transfer learning.

 

Detailed Description

This app provides:

1. Lip-reading for speech-impaired people so their lip movements are converted to text, which is then converted to speech;

2. Speech-to-text facility for hearing-impaired people so they can intercept anything spoken to them, and access it through text.

Tech Stack

1. Android

2. Flask

3. Tensorflow

4. Google APIs

5. Firebase

 

Future Scope

Made with Slides.com