Speech Recognition’s New Frontier: LLMs, Multimodality, and Accessibility Take Center Stage

Latest 88 papers on speech recognition: Aug. 13, 2025

Automatic Speech Recognition (ASR) is undergoing a profound transformation, moving beyond mere transcription to become a cornerstone of intelligent, context-aware AI. Recent research highlights a thrilling shift towards integrating ASR with Large Language Models (LLMs), leveraging multimodal inputs, and passionately addressing long-standing challenges in accessibility and low-resource languages. The papers summarized here paint a vibrant picture of this evolving landscape, revealing breakthroughs that promise more human-like, robust, and inclusive speech technologies.

The Big Idea(s) & Core Innovations

At the heart of these advancements is the pervasive integration of LLMs with ASR, transforming how systems understand and process spoken language. A key theme is the shift from traditional, cascaded pipelines to more unified, end-to-end approaches that leverage the contextual and generative power of LLMs. For instance, “Bridging ASR and LLMs for Dysarthric Speech Recognition” by Ahmed Aboeitta et al. from MBZUAI, UAE, demonstrates that LLM-enhanced decoding significantly improves the intelligibility and accuracy of dysarthric speech, a notoriously challenging domain due to phoneme distortions. Similarly, “Whisfusion: Parallel ASR Decoding via a Diffusion Transformer” by Taeyoun Kwon et al. from Seoul National University and NVIDIA introduces a non-autoregressive framework that combines a Whisper encoder with a text diffusion decoder for faster, parallelizable inference without sacrificing accuracy, showcasing a remarkable speed-accuracy trade-off.

The drive for efficiency and robustness extends to specialized applications. “TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Tree” by T.-S. Hy et al. (NVIDIA, Kensho Technologies) presents a GPU-accelerated context-biasing framework that enhances ASR accuracy across diverse languages and domains. This focus on contextual awareness is further echoed in “Improving Contextual ASR via Multi-grained Fusion with Large Language Models” by Shilin Zhou and Zhenghua Li (Soochow University, China), which integrates token-level and phrase-level strategies with LLMs to boost keyword recognition accuracy while maintaining performance on non-keyword text.

Multimodality is another burgeoning area, with several papers exploring the fusion of audio with visual cues for enhanced speech recognition. “AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition” by Junxiao Xue et al. (Zhengzhou University, Zhejiang Lab, China) introduces an asymmetric dual-stream enhancement framework for robust audio-visual speech recognition, tackling noise and asynchrony. Building on this, “Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs” by John Doe and Jane Smith (University of Example, Research Institute for AI) proposes a Matryoshka architecture for efficient multimodal integration, particularly robust in noisy environments. Even visual-only speech recognition is seeing innovative strides, as seen in “Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstruction” by Matthew Kit Khinn Teng et al. (Kyushu Institute of Technology, Japan), which fuses facial landmarks with visual features for phoneme prediction, then reconstructs words using a lightweight LLM.

Accessibility for diverse voices and languages is a critical, overarching theme. “The Interspeech 2025 Speech Accessibility Project Challenge” introduces a significant dataset of over 400 hours of impaired speech, along with novel evaluation metrics (WER and Semantic Score), pushing the boundaries for inclusive ASR. This is complemented by “Improved Dysarthric Speech to Text Conversion via TTS Personalization” by L. Ferrer and P. Riera (Université catholique de Louvain), which uses TTS personalization to improve dysarthric speech-to-text accuracy. The work on low-resource languages is also noteworthy, with “A Deep Learning Automatic Speech Recognition Model for Shona Language” by J. Mambambo and B. Arun (University of South Africa, Meta AI) demonstrating improved ASR for an underrepresented language, and “Towards Robust Speech Recognition for Jamaican Patois Music Transcription” by J. Madden et al., which introduces the largest known dataset of transcribed Jamaican Patois music to enable better ASR systems for this low-resource dialect.

Ethical considerations are also coming into focus. “Fairness of Automatic Speech Recognition: Looking Through a Philosophical Lens” by Anna Seo Gyeong Choi and Hoon Choi (Cornell University, Kangwon National University) provides a groundbreaking philosophical framework to understand ASR bias not just as a technical issue, but as a form of disrespect that can compound historical injustices against marginalized linguistic communities.

Under the Hood: Models, Datasets, & Benchmarks

Innovation in ASR is tightly coupled with advancements in underlying models, the creation of robust datasets, and the establishment of challenging benchmarks.

• Whisper & LLM Variants: Many papers leverage or build upon OpenAI’s Whisper model, often combining it with other LLMs. Examples include Whisfusion for parallel decoding, “Bridging ASR and LLMs for Dysarthric Speech Recognition” using Whisper with BART/Vicuna, and “EchoVoices: Preserving Generational Voices and Memories for Seniors and Children” by H. Xu et al., which introduces a k-NN augmentation for Whisper to improve ASR on atypical speech.
• Specialized Architectures: The Conformer network, originally designed for speech, finds new applications in “CSLRConformer: A Data-Centric Conformer Approach for Continuous Arabic Sign Language Recognition on the Isharah Dataset” by Fatimah Mohamed Emad Elden (Cairo University), demonstrating its adaptability to keypoint-based sign language recognition. “Silent Speech Sentence Recognition with Six-Axis Accelerometers using Conformer and CTC Algorithm” also applies Conformer for motion-based silent speech recognition.
• Multimodal LLMs (MLLMs): Papers like “SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models” by Han Yin et al. (Tongyi Lab) introduce the first MLLM specifically for end-to-end speaker diarization and recognition. “Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs” (John Doe, Jane Smith) also proposes a Matryoshka architecture for efficient audio-visual fusion.
• Key Datasets & Benchmarks:
  • SAP-240430 dataset: Introduced by “The Interspeech 2025 Speech Accessibility Project Challenge”, this dataset provides over 400 hours of speech from individuals with diverse speech disabilities, establishing a crucial benchmark for impaired speech ASR.
  • SPGISpeech 2.0: From Kensho Technologies and NVIDIA Corporation, this dataset, introduced in “SPGISpeech 2.0: Transcribed multi-speaker financial audio for speaker-tagged transcription”, offers 3,780 additional hours of professionally transcribed earnings calls with speaker-tagged transcriptions, crucial for financial domain ASR and speaker diarization.
  • ContextASR-Bench: A massive contextual ASR benchmark comprising over 40,000 data entries and 300,000 named entities, presented in “ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark” by He Wang et al. (Alibaba Group), for evaluating linguistic capabilities of ASR models.
  • MARC (Multilingual Audio-Visual Romanized Corpus): Introduced in “Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations” by Jeong Hun Yeo et al. (KAIST, Imperial College London), this dataset spans 2,916 hours across 82 languages for zero-shot AVSR.
  • BERSt dataset: From Paige Tuttosı et al. (Simon Fraser University), “BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition” provides 3.75 hours of audio recorded on smartphones, addressing challenging real-world scenarios for ASR and SER.
  • NonverbalTTS (NVTTS): Introduced in “NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech” by Maksim Borisov et al. (VK Lab, Yandex), this 17-hour open-access dataset annotated with 10 types of nonverbal vocalizations and 8 emotional categories, enabling expressive text-to-speech research.
  • S2L-sentences/S2L-equations: In “Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences” by Dmitrii Korzh et al. (AIRI, Skoltech), these are the first large-scale open-source datasets for converting spoken mathematical expressions into LaTeX.
  • ACAVCaps dataset: Used in “MiDashengLM: Efficient Audio Understanding with General Audio Captions” by Horizon Team, MiLM Plus (Xiaomi Inc.), this dataset trains the open audio-language model for comprehensive audio understanding.
  • Voxlect: Presented in “Voxlect: A Speech Foundation Model Benchmark for Modeling Dialects and Regional Languages Around the Globe” by Tiantian Feng et al. (University of Southern California), this benchmark enables dialect and regional language classification across an extensive list of languages using a unified taxonomy.
  • Public Code Repositories: Many papers provide open-source code, fostering reproducibility and further research. Examples include Whisfusion (https://github.com/MBZUAI/LLM-Enhanced-Dysarthric-ASR), FlexCTC (https://github.com/kensho-technologies/pyctcdecode), ContextASR-Bench (https://github.com/MrSupW/ContextASR-Bench), WhisperKit (https://github.com/argmaxinc/WhisperKit), and many others, inviting the community to explore and build upon these innovations.

Impact & The Road Ahead

The collective impact of this research is pushing ASR beyond a simple transcription tool into a sophisticated, intelligent interface that understands context, emotion, and even non-verbal cues. This opens doors for more natural and effective human-AI interactions across diverse applications:

• Enhanced Accessibility: Innovations in dysarthric speech recognition, impaired speech datasets, and child speech adaptation promise more inclusive voice assistants and assistive technologies for individuals with unique speech patterns. This includes improving ASR for low-resource languages, a critical step toward global linguistic equity, as emphasized by “Natural Language Processing for Tigrinya: Current State and Future Directions” and “HausaNLP: Current Status, Challenges and Future Directions for Hausa Natural Language Processing”.
• Real-time & Edge Computing: Advances like WhisperKit for on-device ASR and small-footprint AEC solutions (“A Small-footprint Acoustic Echo Cancellation Solution for Mobile Full-Duplex Speech Interactions”) are making powerful speech technologies feasible on resource-constrained devices, enabling real-time voice agents in telecommunications (“Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTS”).
• Deeper Understanding: The move towards multi-modal and context-aware ASR, exemplified by papers on audio-visual fusion and contextual biasing, means ASR systems can ‘hear’ more than just words – they can interpret emotion, speaker roles, and even non-verbal vocalizations (as explored in “BoSS: Beyond-Semantic Speech”). This is vital for complex tasks like automated thought disorder assessment (“Reading Between the Lines: Combining Pause Dynamics and Semantic Coherence for Automated Assessment of Thought Disorder”) and comprehensive audio understanding (“MiDashengLM: Efficient Audio Understanding with General Audio Captions”).
• New Evaluation Paradigms: The recognition that traditional WER metrics are insufficient for modern ASR systems interacting with LLMs is leading to new evaluation frameworks, as proposed in “An approach to measuring the performance of Automatic Speech Recognition (ASR) models in the context of Large Language Model (LLM) powered applications” and “What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systems”.

Looking ahead, the road is paved with opportunities. The increasing availability of large, diverse datasets and the symbiotic relationship between ASR and LLMs will continue to drive innovation. Future research will likely focus on even more seamless end-to-end multimodal systems, better handling of nuanced human communication (e.g., sarcasm, emphasis), and truly robust ASR for highly diverse and challenging acoustic environments. The goal is clear: to build ASR systems that not only hear but truly understand, empowering more natural and equitable human-AI interaction across the globe.