Text-to-Speech’s New Era: From Emotional Control to Low-Resource Triumphs

Latest 15 papers on text-to-speech: Jul. 4, 2026

The world of AI-powered speech synthesis, or Text-to-Speech (TTS), is rapidly evolving, moving beyond mere naturalness to deliver highly expressive, context-aware, and adaptable voices. Recent breakthroughs are pushing the boundaries, tackling complex challenges from nuanced emotional steering to robust performance in low-resource settings and even handling the linguistic intricacies of specific languages. This digest dives into some of the most compelling advancements from recent research, showcasing a vibrant field brimming with innovation.

The Big Idea(s) & Core Innovations

At the heart of these advancements is a concerted effort to imbue TTS systems with greater control, realism, and accessibility. A major theme is disentanglement and fine-grained control over various speech attributes. For instance, researchers at South China University of Technology, Huya Inc., Alibaba Group, and Foshan University in their paper HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech address the fundamental challenges of information conflict and scale gap in emotional TTS. They introduce the HD-Emo codec to structurally isolate emotional optimization from semantic content, ensuring emotional expressiveness doesn’t degrade linguistic intelligibility. Similarly, Sony Research India’s CrossAccent-TTS: Cross-Lingual Accent-Intensity Controllable Text-to-Speech via Disentangled Speaker and Accent Representations enables precise control over accent intensity in cross-lingual settings while meticulously preserving speaker identity through adversarial learning and weighted language embeddings. Their work showcases how disentangled representations can lead to more versatile and controllable voice generation. Further exploring control, University of Melbourne, Monash University, and University of Melbourne’s A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models geometrically analyzes Speech Language Model (SLM) and Conditional Flow-Matching (CFM) modules, finding that SLM offers clean, low-dimensional emotion subspaces ideal for proportional control, paving the way for more nuanced mixed-emotion synthesis.

Another critical area of innovation focuses on optimizing performance in low-resource scenarios and tackling linguistic complexities. The University of Illinois Urbana-Champaign and National Center for Supercomputing Applications’ SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings introduces a speaker-aware grapheme model that significantly reduces word error rates in extremely low-resource TTS settings (as little as 10 minutes of training data) by aligning graphemes with acoustic representations through contrastive learning. This offers a promising alternative to traditional G2P systems. For specific language challenges, SB Intuitions’ Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis addresses the notorious problem of kanji polyphony in Japanese. They employ massive data scaling and targeted synthetic data augmentation to achieve state-of-the-art kanji-level reading accuracy, even introducing a new metric, Kana-CER, for accurate pronunciation evaluation. Meanwhile, the Chungbuk National University, Institute of Digital Research and Innovation (Cambodia), and BigData Labs Co., Ltd.’s Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean demonstrates that LoRA fine-tuning can dramatically improve TTS quality for low-resource languages like Khmer with minimal parameter training, showing that PEFT is most effective where the base model is weakest. Furthermore, The Hong Kong University of Science and Technology (Guangzhou) and Tencent’s VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation introduces a novel reinforcement learning framework that adapts zero-shot TTS models at test time using lightweight learnable prefixes, achieving superior style similarity and intelligibility for uncommon speaking styles (e.g., accented, slurred speech) with just seconds of reference audio. This is a game-changer for personalized TTS without extensive retraining.

Beyond generation, evaluation and integration are also seeing significant innovation. Iconic, Technische Universität München, KTH Royal Institute of Technology, and Imperial College London challenge the conventional wisdom in Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation. Their perceptual study reveals that naturalness and appropriateness are distinct, domain-dependent dimensions, suggesting that for AI assistants, a slightly robotic voice might even be more appropriate than a highly human-like one. This calls for context-aware evaluation beyond universal MOS scores. Addressing end-to-end systems, Microsoft’s Preserving Speech-to-Text LLM Capabilities in Speech-to-Speech Generation introduces PRIME-Speech, a framework that converts frozen speech-to-text LLMs into speech-to-speech models by training only speech-generation modules. This cleverly preserves the LLM’s original reasoning capabilities while adding speech output, solving the critical problem of catastrophic forgetting. And for a truly integrated low-resource solution, ATM Mobilis, Saad Dahlab Blida 1 University, and Algiers, Algeria present Dziri Voicebot: An End-to-End Low-Resource Speech-to-Speech Conversational System for Algerian Dialect, the first full speech-to-speech conversational system for Algerian Darija, demonstrating the power of fine-tuned Whisper-medium and LoRA adaptation for highly underserved dialects. Lastly, NetEase Cloud Music highlights a crucial evaluation gap with CN-NewsTTS Bench: a target-level automatic benchmark for raw-input Chinese news TTS pronunciation, revealing that commercial Chinese TTS systems often mispronounce compact written forms (e.g., sports scores) from raw text, emphasizing the need for domain-specific, target-level evaluation metrics.

Under the Hood: Models, Datasets, & Benchmarks

These innovations are powered by sophisticated models, diverse datasets, and rigorous benchmarks:

• SPARCLE leverages LibriSpeech-960h for pre-training, VCTK v0.92 for evaluation, and integrates Wav2Vec2 and FaCodec timbre embeddings for robust speaker-aware grapheme representations. Notably, it doesn’t offer a public code repository yet.
• HPRO utilizes LibriSpeech-960h, LSSED (206h), and EmoVoice-DB (40h), integrating Whisper-medium and emotion2vec-plus-large for its HD-Emo codec and hierarchical reward optimization. A demo is available at https://xxh333.github.io/hpro-demo/.
• CrossAccent-TTS uses an LLM-based TTS architecture (Qwen 2.5) with Neucodec speech tokenizer and is evaluated on Indic Multilingual and L2-ARCTIC datasets. A demo can be explored at https://research.sri-media-analysis.com/interspeech26-cross-accent-tts/.
• Sarashina2.2-TTS is trained on a massive 361k hours of multilingual speech (53.7% Japanese), introduces the Joyo Kanji Yomi Benchmark (https://github.com/sbintuitions/JoyoKanji-Yomi-Benchmark), and a Kana-ASR model (https://huggingface.co/sbintuitions/kana-whisper). Their system code is at https://github.com/sbintuitions/sarashina2.2-tts.
• Dziri Voicebot relies on fine-tuned Whisper-medium for ASR, DziriBERT for NLU, Llama 3.2 for RAG, and XTTS-v2 with LoRA for TTS. It also built dedicated ASR (2.68h), NLU (15,891 examples), and TTS (50.7 minutes) corpora for Algerian Darija.
• CN-NewsTTS Bench provides an open benchmark (https://github.com/Jayden-X-L/cn-news-tts-bench) with 992 auto-evaluable targets and a three-ASR automatic scoring protocol using MiMo API ASR, SenseVoiceSmall (https://github.com/FunAudioLLM/SenseVoice), and Paraformer-zh (https://github.com/modelscope/FunASR).
• PRIME-Speech uses Phi-4-MM-7B as a frozen backbone, CosyVoice2 tokenizer, and is trained on LibriHeavy (46k hours), evaluated on CoVoST-2 X2EN, VoiceAssistant-400K, TriviaQA, and benchmarks like FLEURS, UltraEval-Audio, BigBench-Audio, VocalBench.
• VoiceTTA improves zero-shot TTS models using RL-based test-time adaptation for flow matching-based TTS, evaluated on KeSpeech and an internal dataset of uncommon speech prompts. Demo available at https://voicetta.pages.dev/.
• LoRA Fine-Tuning of VoxCPM2 uses the VoxCPM2 model and adapts it to Khmer (IDRI corpus) and Korean (KSS corpus), also leveraging FLEURS and Common Voice.
• Geometric Perspective on Composable Emotion Steering uses CosyVoice2 as its backbone TTS, and evaluates on ESD, CREMA-D, RAVDESS, IEMOCAP datasets, with evaluation using Emotion2Vec and WavLM speaker embeddings.
• Adaptive Oscillatory Inductive Bias introduces Oscilla activation into the StyleTTS2 architecture, creating OscillaTTS. It’s evaluated on LJSpeech and Emotional Speech Dataset (ESD), with a demo at https://research.sri-media-analysis.com/interspeech26-oscilla-tts/.
• Joint Residual Reweighting proposes a new CFG decomposition for Flow-Matching Zero-Shot TTS, validated on F5-TTS (https://github.com/SWivid/F5-TTS) and CosyVoice2 (https://github.com/modelscope/CosyVoice).

Impact & The Road Ahead

These research efforts collectively paint a picture of a TTS landscape that is becoming incredibly sophisticated and impactful. From making voice assistants truly understand and speak in nuanced emotional tones to democratizing high-quality speech synthesis for low-resource languages, the implications are vast. Imagine AI interfaces that can adjust their vocal style to match user context and preference, or personalized language learning tools that replicate specific accents on demand.

The ability to integrate speech output seamlessly into frozen LLM backbones, as shown by PRIME-Speech, means that advanced reasoning capabilities can now be paired with natural, multi-turn spoken dialogue without compromising the core intelligence. The shift towards domain-aware evaluation, highlighted by the “Natural Always Appropriate?” paper and CN-NewsTTS Bench, is crucial for developing truly fit-for-purpose TTS systems. We’re moving away from a one-size-fits-all approach towards highly specialized and adaptable voice AI.

The future of TTS is not just about generating human-like speech, but about generating appropriately human-like (or even machine-like, if that’s the context) speech with fine-grained control over every aspect, from emotion and accent to prosody and pronunciation accuracy. The open questions now revolve around achieving even more granular and intuitive control, robustly handling extreme low-resource situations for all languages, and bridging the gap between perceptual quality and task-specific appropriateness across a wider array of applications. The journey towards truly intelligent and empathetic conversational AI is accelerating, and TTS is at the forefront of this exciting revolution!

Please leave this field empty

Hi there 👋

Get a roundup of the latest AI paper digests in a quick, clean weekly email.

Check your inbox or spam folder to confirm your subscription.