Transformers & Mamba: Charting New Horizons in Efficiency, Security, and Understanding

Latest 82 papers on transformer models: Aug. 17, 2025

The world of AI/ML continues its relentless march forward, driven by innovations that push the boundaries of what’s possible. At the forefront of this revolution are transformer models, which have reshaped natural language processing and computer vision, and more recently, the emerging Mamba architecture, promising efficiency in sequence modeling. However, their increasing complexity brings new challenges: how to make them more efficient, more robust, and more interpretable. Recent research dives deep into these pressing issues, offering breakthroughs that promise to democratize access to powerful AI and enhance its reliability across diverse applications.

The Big Idea(s) & Core Innovations

One central theme in recent work is achieving efficiency without sacrificing performance. In a groundbreaking move, “Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization” by John Doe and Jane Smith demonstrates the feasibility of training large transformer models directly on FPGA hardware with minimal memory footprint through tensor compression. This is echoed in hardware acceleration for inference, with “An ultra-low-power CGRA for accelerating Transformers at the edge” proposing a coarse-grained reconfigurable array optimized for energy-efficient edge deployment.

Beyond hardware, architectural innovations are key. “Speed Always Wins: A Survey on Efficient Architectures for Large Language Models” by Weigao from Stanford University comprehensively surveys techniques like Sparse Mixture-of-Experts (MoE) and Linear Sequence Modeling. Complementing this, “The New LLM Bottleneck: A Systems Perspective on Latent Attention and Mixture-of-Experts” offers crucial insights into how MoE and latent attention affect inference efficiency, while “Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition” proposes a new routing strategy for MoE models to boost efficiency in speech recognition.

Robustness and security are also paramount. “Pruning and Malicious Injection: A Retraining-Free Backdoor Attack on Transformer Models” by Taibiao Zhao et al. from Louisiana State University introduces HPMI, the first retraining-free backdoor attack on transformers, highlighting a critical vulnerability. Addressing the broader reliability, “FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention” by Huangliang Dai et al. from the University of California, Riverside, pioneers an end-to-end fault tolerance framework protecting against soft errors. Security is further explored in “Energon: Unveiling Transformers from GPU Power and Thermal Side-Channels”, revealing how sensitive model information can be inferred from hardware signals.

Interpretablity and understanding of complex models are crucial for trust and improvement. “Entropy-Lens: The Information Signature of Transformer Computations” by Riccardo Ali et al. from the University of Cambridge introduces a model-agnostic framework using entropy profiles to interpret transformer computations. “Model Internal Sleuthing: Finding Lexical Identity and Inflectional Morphology in Modern Language Models” by Michael Li and Nishant Subramani from Carnegie Mellon University provides fascinating insights into how lexical and morphological information is represented across transformer layers.

New architectures like Mamba are also making waves. “Keyword Mamba: Spoken Keyword Spotting with State Space Models” by Hanyu Ding et al. from Jiangsu University introduces the first state-space model for keyword spotting, offering strong performance with fewer parameters. In computer vision, “Mamba-X: An End-to-End Vision Mamba Accelerator for Edge Computing Devices” by Dongho Yoon et al. from KAIST improves Vision Mamba efficiency on edge devices, and “AtrousMamaba: An Atrous-Window Scanning Visual State Space Model for Remote Sensing Change Detection” introduces a novel Mamba-based model for remote sensing, effectively balancing local detail and global context.

Under the Hood: Models, Datasets, & Benchmarks

Recent advancements are underpinned by novel models, strategic dataset utilization, and rigorous benchmarking:

• STream3R: A causal Transformer for sequential 3D reconstruction, leveraging KVCache for real-time updates. Project website: https://nirvanalan.github.io/projects/stream3r.
• Erwin NSA model: Integrates Native Sparse Attention into hierarchical transformers for point cloud data, evaluated on cosmology, molecular dynamics, and air pressure datasets. Code: https://github.com/fla-org/native-sparse-attention.
• Transformer-based DDoS Detection: Evaluated on fused UNSW-NB15 and BoT-IoT datasets, achieving 99.79% accuracy with low CPU utilization. Paper: A Transformer-Based Approach for DDoS Attack Detection in IoT Networks.
• eMamba: An end-to-end framework accelerating Mamba models on edge devices, validated on the MARS dataset, achieving up to 48.6× lower energy consumption. Paper: eMamba: Efficient Acceleration Framework for Mamba Models in Edge Computing.
• HPMI: A retraining-free backdoor attack on transformers. Paper: Pruning and Malicious Injection: A Retraining-Free Backdoor Attack on Transformer Models.
• LLMCARE: Uses transformer models and LLM-generated synthetic data (MedAlpaca-7B) for Alzheimer’s detection, utilizing DementiaBank and TalkBank datasets. Code: GitHub (LLMCARE codes).
• Synaptic Pruning: A regularization method for deep learning, demonstrated on time series forecasting with up to 52% error reduction. Code: https://github.com/xalentis/SynapticPruning.
• FT-Transformer: End-to-end fault tolerance for transformers, optimized for Tensor Cores. Paper: FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention.
• WaveTS-B and WaveTS-M: Lightweight models for time series forecasting, combining wavelet transforms and Mixture of Experts (MoE). Paper: Wavelet Mixture of Experts for Time Series Forecasting.
• RealisMotion: A decomposed human motion control and video generation framework, integrating text-to-video diffusion models. Code: https://jingyunliang.github.io/RealisMotion.
• DeepKoopFormer: Enhances transformers with the Koopman operator for nonlinear time series forecasting. Code: https://github.com/Ali-Forootani/deepkoopformer.
• MammoFormer: A transformer-based framework for breast cancer detection in mammography, integrating multi-feature enhancement and XAI. Paper: Transformer-Based Explainable Deep Learning for Breast Cancer Detection in Mammography: The MammoFormer Framework.
• Crisp Attention: A regularization technique for transformers using structured sparsity. Paper: Crisp Attention: Regularizing Transformers via Structured Sparsity.
• MetaHate Dataset: Integrates multiple hate speech datasets, used to evaluate transformer models like BERT, ELECTRA, and RoBERTa for hate speech detection. Code: https://github.com/chapagaisa/hate_speech_detection.
• ADAPTOR: A runtime-adaptive FPGA accelerator for transformer neural networks. Code: Source code available (see paper: A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs).
• Lightweight Transformers (T5-Small, BART-Small, GPT-2): Evaluated on the Spider dataset for text-to-SQL tasks. Code: https://github.com/chiragseth/lightweight-transformers-text-to-sql.
• RoBERTa for Crash Data: Fine-tuned transformer models outperforming zero-shot LLMs for secondary crash identification in Kentucky. Paper: Improving Crash Data Quality with Large Language Models: Evidence from Secondary Crash Narratives in Kentucky.
• Modular Transformer for Precision Agriculture: Optimized for UAV imagery, enhancing image analysis. Paper: Modular Transformer Architecture for Precision Agriculture Imaging.
• xDeepServe: LLM serving system optimized for Huawei CloudMatrix384 SuperPod, with Transformerless architecture and XCCL communication library. Code: https://github.com/HuaweiModelZoo/xDeepServe.
• Interference Matrix: Quantifies cross-lingual interference in encoder-only transformer models across 83 languages. Paper: Interference Matrix: Quantifying Cross-Lingual Interference in Transformer Encoders.
• Generative Attention Models (VAEs, DMs): Applied to sequential recommendation to generate attention weight distributions. Paper: Why Generate When You Can Transform? Unleashing Generative Attention for Dynamic Recommendation.
• OpenMed NER: Open-source transformer models with DAPT and LoRA for biomedical NER across 12 public datasets. Code: https://huggingface.co/OpenMed.
• Transformers in PRNGs: Decoder-only transformers simulating LCG and Mersenne Twister PRNGs. Paper: Transformers in Pseudo-Random Number Generation: A Dual Perspective on Theory and Practice.
• Length Generalization Transfer: Investigates how transformers generalize to longer inputs through task association. Code: https://github.com/meta-llama/llama3/.
• RACE-IT: Analog CAM-crossbar engine for in-memory transformer acceleration. Paper: RACE-IT: A Reconfigurable Analog CAM-Crossbar Engine for In-Memory Transformer Acceleration.
• Scaling and Distilling Transformers for sEMG: Achieves improved cross-user performance and effective knowledge distillation for surface electromyography. Code: https://github.com/facebookresearch/fairemg.
• Hybrid U-Net with Transformer for MRI Tumor Segmentation: Demonstrates competitive performance on local hospital datasets. Code: https://github.com/qubvel/segmentation.
• Detection Transformers Under the Knife: Neuroscience-inspired ablation studies on DETR, DDETR, and DINO models. Code: https://github.com/deepdissect/DeepDissect.
• Spanish Morphomic Patterns: Compares transformer models’ generalization abilities to human responses using L-shaped verbs. Code: https://anonymous.4open.science/r/cognitive_modeling_aaacl-2C78/.
• Bangla BERT for Hyperpartisan News Detection: Semi-supervised and explainable AI approach leveraging limited labeled data. Code: https://github.com/AngelFelipeMP/Arabic-Hate-Speech-Covid-19.
• Reveal2Revise Framework: Interpretability-driven bias detection and mitigation in medical AI, applied to VGG16, ResNet50, and ViT. Code: https://github.com/frederikpahde/medical-ai-safety.
• Cluster Purge Loss: Novel Deep Metric Learning loss for fine-tuning transformer models in equivalent mutant detection. Code: https://github.com/tianzhaotju/EMD.
• Transformer vs. LSTM for Mental Disorder Detection: Evaluates performance on social media data. Code: https://github.com/KhaledHasan/MentalHealthNLP.
• Frozen Visual Unicode Representations: Semantic understanding in Transformers emerges without trainable input embeddings. Code: https://github.com/AVBochkov/Embeddings.
• Number Theory with Deep Learning: Small transformers learn Möbius and squarefree indicator functions using CRT-based encoding. Code: https://github.com/davidlowryduda/mobius_case_study.
• LoRA Adapters for Clinical Text Classification: Reduces computational overhead in LLMs for biomedical NLP tasks. Paper: The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints.
• MAELRE: Modality-Agnostic Efficient Long Range Encoder, reducing computational and memory costs for multi-modal processing. Paper: Modality Agnostic Efficient Long Range Encoder.
• ADE Detection in Dutch Clinical Text: Benchmarks transformer models like MedRoBERTa.nl. Paper: Detection of Adverse Drug Events in Dutch clinical free text documents using Transformer Models: benchmark study.
• LLM-based Embedders for Prior Case Retrieval: Outperforms BM25 in legal case retrieval. Code: https://github.com/DamithDR/case-retrieval.git.
• Convergence of Gradient Descent on Transformers: Investigates the role of residual connections in training stability. Paper: On the Convergence of Gradient Descent on Learning Transformers with Residual Connections.
• Mammo-Mamba: Hybrid state-space and transformer architecture with sequential Mixture of Experts for multi-view mammography. Paper: Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography.
• Ironman: Accelerator for Oblivious Transfer Extension in privacy-preserving AI with near-memory processing. Paper: Ironman: Accelerating Oblivious Transfer Extension for Privacy-Preserving AI with Near-Memory Processing.
• ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference. Code: https://github.com/luo3300612/.
• Scaling Recommender Transformers: Deployed to one billion parameters for improved music recommendations. Paper: Scaling Recommender Transformers to One Billion Parameters.
• StackTrans: Transformer variant integrating hidden state stacks, inspired by pushdown automata, for formal language and NLU tasks. Paper: StackTrans: From Large Language Model to Large Pushdown Automata Model.
• DNA Sequence Modeling with Transformers: Evaluation of tokenization (BPE vs. k-mer) and positional encoding (RoPE, AliBi, sinusoidal). Code: https://github.com/synlp/DNA-coding.
• Partial Symmetry Enforced Attention Decomposition (PSEAD): A group-theoretic framework for equivariant transformers in biological systems. Code: https://github.com/DanielAyomide-git/psead.
• Interpretable Vision Transformers Under Attack: Framework for evaluating and attacking interpretable vision transformers using interpretability features. Code: https://github.com/InfoLab-SKKU/AdViT.
• Contextual Embeddings for Bipolar Disorder Detection: Compares RoBERTa and LSTM for social media analysis. Paper: Beyond Architectures: Evaluating the Role of Contextual Embeddings in Detecting Bipolar Disorder on Social Media.
• PTSD Detection in Clinical Interviews: Evaluates NLP methods and LLM prompting strategies (SentenceBERT, LLaMA) on clinical interview transcripts. Paper: Detecting PTSD in Clinical Interviews: A Comparative Analysis of NLP Methods and Large Language Models.
• Transformers and Ensemble for Arabic Hate Speech: Evaluates six transformer models and two ensemble techniques (Majority Vote, Highest Sum) on the CERIST NLP Challenge dataset. Code: https://github.com/AngelFelipeMP/Arabic-Hate-Speech-Covid-19.
• Political Leaning and Politicalness Classification: Combines multiple datasets and trains transformer models for political text analysis. Code: https://github.com/matous-volf/political-leaning-prediction.
• Training Transformers with Enforced Lipschitz Constants: Introduces spectral soft cap and spectral hammer for robust training. Code: https://github.com/Arongil/lipschitz-transformers.
• Best Practices for Crop Mapping: Identifies optimal preprocessing and transformer models for pixel-wise crop mapping. Code: “All code is publicly available to encourage reproducibility practice.”
• ROSE: Transformer-based model for refactoring recommendations to address architectural smells. Paper: ROSE: Transformer-Based Refactoring Recommendation for Architectural Smells.
• DVFL-Net: Lightweight distilled video focal modulation network for spatio-temporal action recognition. Code: https://github.com/iscaas/DVFL-Net.
• SystolicAttention: Fuses FlashAttention within a single systolic array for hardware acceleration. Code: https://github.com/VCA-EPFL/FSA.
• Language Models for Adult Service Website Text Analysis: Custom transformer models for combating sex trafficking using ASW data. Paper: Language Models for Adult Service Website Text Analysis.
• Universal Approximation Theorem for Single-Layer Transformer: Formal proof of universal approximation capabilities. Paper: Universal Approximation Theorem for a Single-Layer Transformer.
• GMLN-BTS in EdgeIMLocSys: Graph-based Multi-Modal Interaction Lightweight Network for brain tumor segmentation, with Continuous Learning from Human Feedback. Paper: Graph-based Multi-Modal Interaction Lightweight Network for Brain Tumor Segmentation (GMLN-BTS) in Edge Iterative MRI Lesion Localization System (EdgeIMLocSys).

Impact & The Road Ahead

These advancements have profound implications across numerous domains. In healthcare, transformer-based models are proving vital for early disease detection (Alzheimer’s with LLMCARE, breast cancer with MammoFormer, mental disorders from social media, PTSD from clinical interviews) and ensuring AI safety by mitigating spurious correlations (Reveal2Revise). Their ability to handle complex clinical text and imaging data, combined with growing interpretability features, pushes us closer to reliable, clinically adoptable AI.

Efficiency and edge deployment are critical for democratizing AI. The work on FPGA acceleration, lightweight transformers, and specialized Mamba accelerators (eMamba, Mamba-X) means powerful AI can move from data centers to personal devices, enabling real-time applications in autonomous systems, IoT security (DDoS detection), and even precision agriculture.

Security and robustness remain key concerns. The identification of backdoor attacks (HPMI) and side-channel vulnerabilities (Energon) underscores the need for continuous research into making these models resilient. Conversely, advancements in fault tolerance (FT-Transformer) and theoretical understanding of adversarial robustness (Understanding In-Context Learning) offer pathways to more secure AI.

From scaling recommender systems to billions of parameters, to understanding the fundamental nature of language representation, the research presented here paints a vibrant picture of an AI field rapidly evolving. The interplay between theoretical insights, novel architectures, and hardware-aware optimizations is driving a future where AI is not just powerful, but also efficient, transparent, and trustworthy across an ever-expanding array of real-world applications. The journey is far from over, and these papers are crucial signposts on the road to next-generation AI.