Model Compression: Shrinking AI’s Footprint and Boosting Performance

Latest 50 papers on model compression: Oct. 6, 2025

The world of AI and machine learning is rapidly evolving, with models growing ever larger and more powerful. Yet, this power comes at a cost: immense computational resources, significant energy consumption, and slower inference times, especially for deployment on edge devices. This challenge has fueled intense research into model compression, a critical area focused on making these advanced AI systems smaller, faster, and more efficient without sacrificing performance. Recent breakthroughs, as highlighted by a collection of innovative papers, are pushing the boundaries of what’s possible, tackling everything from large language models (LLMs) to vision transformers and distributed learning.

The Big Idea(s) & Core Innovations

At the heart of these advancements is a shared ambition: to achieve substantial model reduction while preserving, or even enhancing, performance. Several recurring themes and novel solutions emerge across the research:

• Intelligent Pruning & Low-Rank Approximations: Traditional pruning often removes weights indiscriminately. However, papers like “Interpret, Prune and Distill Donut: towards lightweight VLMs for VQA on document” by A. Ben Mansour et al. from Universitat Autònoma de Barcelona and Microsoft Research introduce interpretability-guided pruning. This enables the creation of lightweight models like Donut-MINT, which achieve competitive performance on document VQA by focusing on essential computational patterns. Similarly, “GAPrune: Gradient-Alignment Pruning for Domain-Aware Embeddings” from Yixuan Tang and Yi Yang at The Hong Kong University of Science and Technology leverages gradient alignment and Fisher Information to prune domain-specific embeddings, often improving domain capabilities. For LLMs, MUCHAMMAD DANIYAL KAUTSAR et al.’s “CALR: Corrective Adaptive Low-Rank Decomposition for Efficient Large Language Model Layer Compression” from IEEE Transactions on Artificial Intelligence and tech giants like Meta and Google Research, introduces an adaptive low-rank decomposition to effectively compress layers while maintaining performance. “Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models” by Jialin Zhao et al. from Tsinghua University proposes PIFA, a lossless meta low-rank representation and error-minimization reconstruction for efficient LLM inference, demonstrating significant memory savings and speedups.

• Advanced Quantization Strategies: Quantization reduces the precision of model weights and activations, but it’s a delicate balance. The Shanghai Jiao Tong University team, led by Kai Liu, introduces “CLQ: Cross-Layer Guided Orthogonal-based Quantization for Diffusion Transformers”, a post-training method that achieves ultra-low bit-width compression for Diffusion Transformers (DiTs) by mitigating quantization errors through cross-block calibration and orthogonal smoothing. For LLMs, “LLM Compression: How Far Can We Go in Balancing Size and Performance?” by Sahil Sk et al. at Odia Generative AI and AMD Silo AI empirically evaluates 4-bit quantization techniques like GSQ and GPTQ, showing minimal impact on latency and throughput, making them viable for production. Weilun Feng et al. (Chinese Academy of Sciences, ETH Zürich) present “S²Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation”, a technique to quantize video diffusion models with minimal quality loss using Hessian-aware salient data selection and attention-guided sparse token distillation.

• Knowledge Distillation & Architectural Refinements: Transferring knowledge from a large teacher model to a smaller student is a powerful compression strategy. “An Efficient GNNs-to-KANs Distillation via Self-Attention Dynamic Sampling with Potential for Consumer Electronics Edge Deployment” by Can Cui et al. from Dalian Jiaotong University presents SA-DSD, a framework for distilling GNNs into more efficient Kolmogorov-Arnold Networks (KANs) for edge deployment. “Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method” by Suleyman O. Polat et al. at the University of North Texas dynamically generates synthetic data in high-loss regions of the embedding space, significantly boosting student performance. “IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method” from Northeastern University introduces a Transformer variant that uses iterative implicit Euler methods, combined with Iteration Influence-Aware Distillation (IIAD), to balance accuracy and speed.

• Hybrid & Holistic Approaches: Many papers advocate for combining techniques. “Integrating Pruning with Quantization for Efficient Deep Neural Networks Compression” by Author A et al. from the University of Example explicitly highlights how integrating pruning and quantization yields superior efficiency. Kai Yi (King Abdullah University of Science and Technology), in “Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization”, presents a unified framework for biased and unbiased compression operators with convergence guarantees, vital for distributed systems. The Red Hat AI Innovation team in “Hopscotch: Discovering and Skipping Redundancies in Language Models” shows how selectively skipping attention blocks, combined with trainable scaling parameters, can reduce computational costs without significant performance loss.

Under the Hood: Models, Datasets, & Benchmarks

This wave of research relies on and introduces a variety of essential resources:

• Models Utilized & Advanced:
  • Donut-MINT: A lightweight Visual Language Model (VLM) for document VQA, derived from Donut through interpretability-guided pruning (Interpret, Prune and Distill Donut).
  • Diffusion Transformers (DiTs): The core architecture for visual generation tasks, optimized by methods like CLQ (CLQ: Cross-Layer Guided Orthogonal-based Quantization).
  • Large Language Models (LLMs) (e.g., LLaMA, Qwen, PHI, CodeBERT, CodeGPT, PLBART): Heavily featured across quantization, pruning, and distillation studies (e.g., LLM Compression, CALR, MoBE, Model Compression vs. Adversarial Robustness).
  • whisperM2M: A modified Whisper model fine-tuned for multilingual speech translation, achieving SOTA performance in efficiency (Novel Parasitic Dual-Scale Modeling).
  • MoBE-based LLMs: Mixture-of-Experts (MoE) LLMs like DeepSeek-V3 and Kimi-K2-Instruct are targeted for parameter-efficient compression (MoBE: Mixture-of-Basis-Experts).
  • FR-KAN+: An enhanced Kolmogorov-Arnold Network model for improved computational efficiency in GNN distillation (An Efficient GNNs-to-KANs Distillation).
• Key Datasets & Benchmarks:
  • DocVQA: A standard dataset for document Visual Question Answering, used for evaluating Donut-MINT (Interpret, Prune and Distill Donut).
  • FinMTEB, ChemTEB: Domain-specific benchmarks for evaluating domain-aware embeddings and pruning methods like GAPrune (GAPrune: Gradient-Alignment Pruning).
  • MS MARCO, BoolQ, GSM8K, GLUE benchmarks: Widely used NLP benchmarks for evaluating LLM compression techniques (LLM Compression, Synthetic Adaptive Guided Embeddings (SAGE)).
  • LLMC+: A new comprehensive benchmarking framework and toolkit specifically designed for Vision-Language Model (VLM) compression, addressing multi-modal and multi-turn dialogue tasks (LLMC+: Benchmarking Vision-Language Model Compression).
• Code Repositories for Exploration:
  • CLQ: https://github.com/Kai-Liu001/CLQ
  • SUBSPEC: https://github.com/NYCU-EDgeAi/subspec
  • MaRVIn: https://github.com/alexmr09/Mixed-precision-Neural-Networks-on-RISC-V-Cores
  • GAPrune: https://github.com/yixuantt/GAPrune
  • Hopscotch: https://github.com/redhat-labs/hopscotch
  • SymWanda, Scafflix, Cohort-Squeeze: https://github.com/kaiyi-me/symwanda, https://github.com/kaiyi-me/scafflix, https://github.com/kaiyi-me/cohort-squeeze
  • SLiM: https://github.com/Mohammad-Mozaffari/slim
  • S²Q-VDiT: https://github.com/wlfeng0509/s2q-vdit
  • FAIR-Pruner: https://github.com/Chenqing-Lin/FAIR-Pruner
  • MoBE: https://github.com/inclusionAI/MoBE
  • Pivoting Factorization: https://github.com/biomedical-cybernetics/pivoting-factorization
  • Model Folding: https://github.com/nanguoyu/model-folding-universal
  • OWLed: https://github.com/JiaxiLi1/OWLed

Impact & The Road Ahead

The impact of this research is profound. These advancements are not merely academic; they are enabling a future where sophisticated AI models are ubiquitous, running efficiently on everything from smartphones to autonomous vehicles and embedded systems. This means faster, more responsive AI applications, reduced carbon footprints, and broader accessibility to advanced AI capabilities. For instance, Intel Corporation’s work on “Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity” showcases up to 149x lower energy consumption on neuromorphic hardware, paving the way for truly intelligent edge devices.

However, challenges remain. The paper “Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code” by Md. Abdul Awal et al. from the University of Saskatchewan highlights a crucial trade-off: compressed models, especially those using knowledge distillation, can be more vulnerable to adversarial attacks. “Silent Until Sparse: Backdoor Attacks on Semi-Structured Sparsity” by Wei Guo et al. from the University of Cagliari further exposes a new type of stealthy backdoor attack that becomes active only after sparsification, emphasizing the need for robust security evaluations in compressed models. Furthermore, “The Hidden Costs of Translation Accuracy: Distillation, Quantization, and Environmental Impact” from University of California, Santa Cruz and Research Spark Hub Inc. warns that low-resource languages are more susceptible to performance degradation under compression, urging careful consideration in multilingual contexts.

The integration of model compression with emerging paradigms like federated learning (as surveyed in “Strategies for Improving Communication Efficiency in Distributed and Federated Learning” and “Towards Adapting Federated & Quantum Machine Learning for Network Intrusion Detection” by Author A et al. from Institute of Cybersecurity, University X) promises a future of privacy-preserving, decentralized AI. Even quantum computing is entering the fray, with “Is Quantum Optimization Ready? An Effort Towards Neural Network Compression using Adiabatic Quantum Computing” from **A*STAR, Singapore** exploring its potential for fine-grained pruning-quantization. These studies collectively chart a course towards a future where AI’s immense capabilities are delivered with unprecedented efficiency, driving innovation across every domain while being mindful of resource constraints and ethical implications.