Assistive Technology for All
India has a population of approximately 63 million hearing and speech-impaired people; building effective channels of communication is critical.
AssistiveU aims to eradicate this problem by providing a system that can help speech impaired people by giving them a voice to speak, and hearing impaired people by helping them intercept whatever someone is speaking in the form of text.
The app employs lip-reading, an upcoming area of research in machine learning.
The architecture used (called LipNet) is a state-of-the-art model with an accuracy of 96.4% compared to the professional lip-reading accuracy of 53.4%.
The model was initially trained on the GRID corpus, and fine-tuned further using self-prepared dataset videos using transfer learning.
This app provides:
1. Lip-reading for speech-impaired people so their lip movements are converted to text, which is then converted to speech;
2. Speech-to-text facility for hearing-impaired people so they can intercept anything spoken to them, and access it through text.
1. Android
2. Flask
3. Tensorflow
4. Google APIs
5. Firebase