Sign Voice: Real-Time Sign Language Recognition and Speech Translation System

Abstract

Communication between deaf or hearing-impaired people and the rest of society is still quite a social and technological challenge. The main reason for this is the fact that a very small number of people understand sign language. Although sign language is a very good and expressive way of communicating, the lack of sign language translation tools in real-time is one of the main barriers for inclusive communication in everyday environments. To solve this problem, we have developed SignVoice, a system that recognizes sign language in real-time and converts hands movements into spoken words automatically through computer vision and machine learning. The system we propose exploits MediaPipe to attain the most precise, efficient, and robust hand landmark detection under varied lighting conditions and different backgrounds. A hybrid classification schema has been implemented which allows the system to easily recognize both static and dynamic gestures. Static hand gestures are detected using a K Nearest Neighbors (KNN) classifier, dynamic gestures are analyzed by Long Short-Term Memory (LSTM) networks to understand sequential hand movements based on the change of time. This two-model system increases the recognition accuracy without requiring high computational power. Experimental results demonstrate that the proposed method performs extremely well in terms of recognition accuracy and at the same time demands very little time, thereby establishing the method as highly suitable for assistive communication applications. In summary, SignVoice is a step forward in addressing accessibility issues and promoting social inclusion by enabling communication between hearing-impaired individuals and the rest of the community.