Enhancing Environmental Awareness for the Visually Impaired with a Real-Time Mobile Ai Assistant
DOI:
https://doi.org/10.67440/ahj.vi.3316Keywords:
Assistive Technology, Object Detection, Fine-Grained Classification, Deep Learning, Mobile AI, YOLO.Abstract
This research focuses on the development of autonomous mobility solutions for visually impaired users in dynamic unstructured real-life environments. In that regard, the problem at hand is hindered due to insufficient semantic understanding associated with existing assistive devices. Indeed, while such solutions are highly effective for immediate obstacle avoidance, they lack ability to deliver precise and timely information necessary for more complicated spatial understanding of surroundings. For this reason, we have developed a two-stage deep learning solution for real-time audio description that will be delivered within a cloud-enabled mobile platform. The process starts with real-time camera frame classification performed via You Only Look Once (YOLO) detector, followed by the processing of selected image patches via second CNN, trained for fine-grained classification. Such a two-stage solution allows detecting of minute details such as open or closed status of a detected door that cannot be achieved with a one-stage model. The structured data obtained is then transferred to the mobile device for further conversion into voice synthesis via native engine. Evaluation of the proposed method on the specially curated dataset shows 96% descriptive accuracy

