An Effective Speech-To-Text Translation Using Attention Fusion of Multimodal Conformer and Transformer Approach
Keywords:
Pretrained models fusion, Transformer encoder, Decoder, Fusion, Metrics assessment, and Accuracy.Abstract
There are many applications such as education, online meetings, healthcare, and digital communication in which direct conversion of spoken language into required text is needed using Multilingual speech-to-text translation (MSTT), which focuses on eliminating intermediate steps to reduce complexity. In this process, there is always a challenge during translation in different languages, errors raised that reduce translation quality, and addressing acoustic variance. Tools such as mBART, Whisper, and wav2sec 2.0 illustrate Automatic speech recognition (ASR) and Machine Translation (MT) as complete pipeline models. In these approaches, existing translation methods and conventional models suffer from error propagation; ASR has issues with inaccuracies that degrade translation quality; and end-to-end mechanisms suffer from low-resource languages andlimited acoustic diversity. Other issues, such as limited adaptability, bias, and alignment issues between speech-to-text, were experienced by pretrained models. To eliminate these shortfalls, an integrated approach fuses XLS-R and WavLM pretrained models for speech recognition, integrates SpecAugment on Conformer, a gated cross-attention transformer decoder, hybrid connectionist temporal classification and cross-entropy loss, and learning-rate warm-up with the FedAdam optimizer. Evaluation results on the proposed model with existing models show superior BLEU score performance, WER reduction, and faster convergence under real-world communications.

