A Full-Stack Multimodal Assistive Communication System

Abstract

MANUSCRIPT is a full-stack web-based assistive communication platform designed to bridge the communication gap between sign-language / visually impaired users and the hearing/ seeing population. The system enables real-time conversion of hand gestures and spoken language into readable text using computer vision and speech recognition technologies. It integrates MediaPipe Hands for gesture detection and the Web Speech API for speech-to-text conversion, supported by a Flask backend for secure data storage, authentication, and scalability. The primary problem addressed is the lack of accessible, scalable, and affordable communication tools for individuals with hearing, visual or speech impairments. Existing solutions are either limited to single modalities or rely heavily on server-based systems, leading to latency and privacy concerns. The objective of this project is to develop a hybrid system that combines real-time client-side processing with backend support to enhance usability, scalability, and data persistence. The system achieves an average response time of 0.17s for gesture recognition and 0.8s for speech-to-text conversion, enabling near real-time communication. The expected outcome is a robust, user-friendly platform capable of facilitating seamless communication using gestures and speech, with potential applications in education, healthcare, and everyday interactions.