LLM Compression: The Quest for Efficiency, Robustness, and Fairness

Latest 50 papers on model compression: Oct. 20, 2025

The relentless growth of Large Language Models (LLMs) and Vision-Language Models (VLMs) has brought unprecedented capabilities to AI, but it comes at a significant cost: massive computational demands, extensive memory footprints, and substantial energy consumption. This makes their deployment on resource-constrained devices, or even efficient scaling in the cloud, a persistent challenge. Fortunately, recent research is pushing the boundaries of model compression, striking a delicate balance between efficiency, performance, and crucial ethical considerations like fairness and robustness.

The Big Idea(s) & Core Innovations

The latest wave of research in model compression isn’t just about making models smaller; it’s about making them smarter, more resilient, and more accountable. A recurring theme is the move beyond simple weight pruning or quantization to more sophisticated, context-aware, and theoretically grounded approaches.

For instance, the paper Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning by Minsik Choi, Hyegang Son, Changhoon Kim, and Young Geun Kim from Korea University and Soongsil University introduces HIES (Head Importance-Entropy Score). This novel pruning criterion unifies gradient-based head importance with attention entropy, leading to more stable and efficient transformer pruning, improving model quality by up to 15.2% and stability by 2.04x. This focus on attention entropy highlights the importance of understanding the internal dynamics of transformers for effective compression.

Building on the idea of importance-aware compression, the University of California-Santa Barbara and Amazon team, including Ryan Solgi, Kai Zhen, and Zheng Zhang, presents Saten in their paper Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models. Saten combines sparse error approximation with tensor networks for post-training compression of LLMs, achieving state-of-the-art results in accuracy and compression ratio. This emphasizes integrating structured and unstructured sparsity for higher compression without performance sacrifice.

Further pushing the boundaries of low-rank compression, Ryan Solgi et al. from University of California-Santa Barbara and Amazon also introduce PGSVD in Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM. This framework leverages Pareto-guided rank selection and activation-aware insights for both LLMs and VLMs, providing theoretical foundations for layer-wise compression and achieving over 30% accuracy gains at the same memory usage. This aligns with the findings in IMPACT: Importance-Aware Activation Space Reconstruction by Md Mokarram Chowdhury et al. from University of Washington and Microsoft Research, which argues that traditional low-rank weight compression is insufficient, and LLMs exhibit stronger low-rank structure in activations, leading to significant size reduction with minimal accuracy loss.

Crucially, the ethical implications of compression are gaining prominence. The paper Fewer Weights, More Problems: A Practical Attack on LLM Pruning by Kazuki Egashira et al. from ETH Zurich reveals a chilling vulnerability: malicious behaviors can be triggered only after pruning, exploiting common inference engines like vLLM. Similarly, Downsized and Compromised?: Assessing the Faithfulness of Model Compression by Moumita Kamal and Douglas A. Talbert from Tennessee Tech University demonstrates that high accuracy doesn’t guarantee faithfulness or fairness in compressed models, introducing metrics to detect subtle shifts in predictive patterns. These works underscore that compression isn’t just an engineering problem, but a security and ethics challenge.

Other innovations include SUBSPEC: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding from National Yang Ming Chiao Tung University and Cornell University, which achieves over 10x speedups for offloaded LLMs on consumer GPUs via low-bit quantized layers and probability sharpening. The Red Hat AI Innovation team in Hopscotch: Discovering and Skipping Redundancies in Language Models shows how trainable scaling parameters can enable skipping redundant attention blocks without significant performance drops. Meanwhile, Kai Yi from King Abdullah University of Science and Technology (KAUST), in Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization, presents SymWanda for robust post-training pruning without retraining, alongside a unified theoretical framework for biased and unbiased compression operators.

Under the Hood: Models, Datasets, & Benchmarks

These advancements are underpinned by robust experimentation on widely-used models and datasets, often alongside new tools and benchmarks designed to rigorously evaluate compression techniques.

• LLaMA, Qwen, and PHI Models: Heavily utilized in studies like LLM Compression: How Far Can We Go in Balancing Size and Performance?, these models serve as benchmarks for evaluating the impact of low-bit quantization (GSQ, GPTQ) on accuracy, latency, and throughput across NLP tasks (e.g., MS MARCO, BoolQ, GSM8K).
• BERT-Base and LlaMA Models: Used by Saten (https://github.com/rmsolgi/saten.git) to demonstrate superior performance over existing tensor-network compression methods.
• DocVQA Dataset: The target for A. Ben Mansour et al. from Universitat Autònoma de Barcelona, Microsoft Research, and Google Research in Interpret, Prune and Distill Donut: towards lightweight VLMs for VQA on document, leading to the lightweight Donut-MINT model.
• HuggingFace Accelerate, Cerebras SlimPajama-627B: Resources supporting ARMOR in ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization, which focuses on 2:4 sparsity for LLMs.
• LLMC+ Toolkit: Introduced in LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit by Chengtao Lv et al. from Nanyang Technological University and SenseTime Research, this toolkit offers a comprehensive benchmark for VLM compression, addressing the need for modular comparisons and diverse task evaluation. (Code: https://github.com/ModelTC/LightCompress)
• Intel Loihi 2 Neuromorphic Hardware: Explored in Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity by Alessandro Pierro et al. from Intel Corporation, demonstrating significant latency and energy efficiency gains for sparse linear RNNs. (Code: https://github.com/IntelLabs/SparseRNNs)
• Pythia Models: Used in Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory by Einar Urdshals et al. from Timaeus and UK AI Security Institute, to empirically demonstrate the relationship between local learning coefficient (LLC) and quantization-based compressibility metrics. (Code: https://github.com/neelnanda-io/TransformerLens)
• DaMoC Framework: Introduced in DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression by Wei Huang et al. from Ant Group, China, it combines data filtering (distribution-aware, quality-aware, hybrid) and model compression (token compression, layer pruning) for efficient LLM fine-tuning across domain-specific datasets (e.g., medical Q&A, financial Q&A).
• GitHub Repositories: Numerous papers provide open-source code, fostering reproducibility and further research, such as D-com (https://github.com/faraztahmasebi/d-com), Saten (https://github.com/rmsolgi/saten.git), PGSVD (https://github.com/UCSB-LLM-Research/PGSVD), GAPrune (https://github.com/yixuantt/GAPrune), CPSC-DFKD (https://github.com/RoryShao/CPSC-DFKD.git), SLIM (https://github.com/Mohammad-Mozaffari/slim), and Pivoting Factorization (https://github.com/biomedical-cybernetics/pivoting-factorization).

Impact & The Road Ahead

These breakthroughs hold immense potential for the democratization of advanced AI. By making models smaller, faster, and more energy-efficient, we can deploy sophisticated LLMs and VLMs on edge devices, enabling real-time, privacy-preserving AI in areas like industrial IoT (as explored in Federated Split Learning for Resource-Constrained Robots in Industrial IoT), smart manufacturing, and autonomous systems. Frameworks like InstaGeo (https://github.com/instadeepai/InstaGeo-E2E-Geospatial-ML.git) by I.S. Yusuf et al. from InstaDeep streamline the entire geospatial ML workflow, demonstrating the impact of compute-efficient solutions from data to deployment.

The critical emphasis on fairness and robustness, highlighted by papers like “Fewer Weights, More Problems” and “Downsized and Compromised?”, is a wake-up call for responsible AI development. As we compress models, we must actively guard against introducing or exacerbating biases and vulnerabilities. The development of methods like HGLA pruning (https://github.com/amberhuang01/HGLA) by Nannan Huang et al. from RMIT University shows a promising path forward for preserving fairness during compression.

The integration of model compression with federated learning, as discussed in Strategies for Improving Communication Efficiency in Distributed and Federated Learning and Federated Split Learning for Resource-Constrained Robots in Industrial IoT, offers a powerful paradigm for privacy-preserving, collaborative AI at scale. Moreover, the emergence of quantum optimization techniques for neural network compression, as explored in Is Quantum Optimization Ready?, hints at a future where quantum computing could unlock entirely new levels of efficiency.

The road ahead involves continued innovation in multi-modal compression (LLMC+), exploring the theoretical underpinnings of complexity and compressibility (Compressibility Measures Complexity), and developing adaptive, principled approximation methods (Principled Approximation Methods for Efficient and Scalable Deep Learning). It’s clear that the future of AI hinges not just on bigger models, but on smarter, more efficient, and more trustworthy ones. The research community is rapidly advancing towards this exciting vision, making powerful AI accessible to more users and applications than ever before.