Energy Efficiency in AI/ML: From Cloud to Edge and Beyond
Latest 29 papers on energy efficiency: Aug. 1, 2026
The relentless march of AI and Machine Learning, especially with the proliferation of Large Language Models (LLMs), has brought unprecedented computational power and transformative applications. However, this progress comes at a significant environmental cost, with energy consumption emerging as a critical challenge. From power-hungry data centers to battery-constrained edge devices, achieving high performance without draining resources is paramount. This blog post dives into recent breakthroughs, gleaned from a collection of cutting-edge research papers, that tackle this energy efficiency imperative across various facets of AI/ML systems.
The Big Ideas & Core Innovations
At the heart of many recent innovations is the dual focus on algorithmic optimization and hardware-software co-design. Researchers are rethinking how AI models are structured and how they interact with underlying silicon to squeeze out every joule of efficiency.
For instance, the challenge of energy-hungry LLM inference is being aggressively addressed. In LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference, authors from KAIST (Korea Advanced Institute of Science and Technology) introduce Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA). This co-designed approach not only mitigates the high energy overhead of rotation-based quantization but also resolves accuracy issues caused by extreme outliers, achieving an impressive 27.4 TOPS/W. Building on this, GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference, also from KAIST, tackles the fundamental conflict between global rotation and fine-grained quantization. Their Coarse Rotation Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP) allow for synergistic operation, enabling state-of-the-art 4-bit LLM inference with fully integer dequantization—a critical step for hardware efficiency.
The broader theme of hardware-software co-design extends to novel computing paradigms. The New York University Abu Dhabi (NYU Abu Dhabi) team, in MDTransformer: A Hardware-Software Co-Design of Mode-Division Photonic Transformer Accelerator with Inverse-Designed Coherent Crossbar, proposes a photonic transformer accelerator that leverages mode-division multiplexing and inverse-designed coherent crossbars. This eliminates the need for expensive multi-wavelength light generation, achieving four-fold parallelism-per-waveguide and resulting in 40.6% energy savings over state-of-the-art photonic accelerators. Similarly, BRIM: Workload-Balanced Dual-Sided Bit-Serial Sparse Inference Accelerator from University of Southern California and Yale University tackles a common bottleneck in sparse accelerators: workload imbalance. By combining Cyclic-Balanced Pruning (CBP) and runtime Pairwise Slot Donation, BRIM boosts processing element (PE) utilization to over 90%, yielding up to 1.63x energy efficiency improvement.
Energy efficiency isn’t just about compute; memory plays a huge role. LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving by researchers from Georgia Institute of Technology and Samsung Semiconductor Inc. introduces a simulation framework to evaluate monolithic 3D (M3D) memories. They show that expanding L2 cache with M3D technology can reduce chip energy by up to 44% for LLM prefill tasks, a significant gain for large models. The team at Duke University and Hewlett Packard Labs, in Multi-primitive in-memory computing for Monte Carlo tree search, further pushes the boundaries of in-memory computing (IMC), achieving up to 2,059x energy efficiency over GPUs for Monte Carlo tree search by mapping each phase to specialized IMC primitives.
Beyond direct hardware and algorithmic enhancements, strategic management of computing resources is key. In Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising, King’s College London introduces N-MDLMs, which combine block-parallel diffusion decoding with spike-based sparse computation. This event-driven, neuromorphic approach improves LLM inference efficiency by addressing both compute and memory bottlenecks. This aligns with platforms like the SpiNNaker2 chip: a many-core platform for flexible and scalable brain-inspired computing from Technische Universität Dresden and University of Manchester, which achieves 2.7 TOPS/W and offers 18x energy reduction for certain event-based inference algorithms compared to GPUs. Furthermore, the University of Twente demonstrates significant instruction-fetch energy reductions (up to 35.5%) in RISC-V processors for embedded AI via dynamic and static loop caching in Reducing Instruction-Fetch Energy in RISC-V for Embedded AI Processing via Dynamic and Static Loop Caching.
Even how we prompt LLMs can impact energy. Researchers from Georgia State University, in Keyword Matters: Unveiling the Energy Sensitivity of On-Device LLM Prompting, empirically show that careful keyword choices in prompts can save up to 62.6% energy during on-device LLM inference.
For communication systems, energy efficiency is equally vital. Secrecy Energy Efficiency for IRS-Assisted Low-Altitude Communications: A D3QN-PER Based Approach by Luoyang Normal University and Xidian University optimizes UAV-IRS networks for secure and energy-efficient communication using a novel D3QN-PER reinforcement learning algorithm. In a similar vein, TRUAV: Distributed Multi-Agent Reinforcement Learning for Trajectory Planning and Routing Enhancement in UAV-Aided IoT-Enabled VANETs from Beihang University and Southeast University demonstrates that distributed Q-learning can reduce energy consumption by ~19% for UAV swarms compared to centralized baselines. The challenge of future 6G networks is addressed by KTH Royal Institute of Technology in Energy-Efficient Dual-Band Communication: How to Allocate Traffic to Sub-THz Carriers?, which provides analytical thresholds for optimally activating sub-THz bands, potentially cutting power consumption.
Finally, monitoring and managing energy consumption in complex systems is crucial. The CONNECT Centre at Trinity College Dublin’s work in ML-based Predictive Models for Power Consumption in Virtualised O-RANs shows hybrid DNN-XGBoost models predicting O-RAN power with <0.5% error, enabling energy-aware orchestration. NUMA balancing hampering performance of spiking network simulations by Jülich Research Centre and RWTH Aachen University found that simply disabling automatic NUMA balancing on supercomputers can reduce energy consumption by 30% for spiking neural network simulations—a subtle but profound insight into system-level optimization.
Under the Hood: Models, Datasets, & Benchmarks
Many of these advancements are propelled by new frameworks, methodologies, and rigorous evaluations:
- LLMET Framework: A cross-layer simulation framework for LLM serving supporting emerging M3D memory technologies, validated against NVIDIA A100 data. It uses resources like NS-Cache for memory evaluation and Timeloop for DNN accelerator evaluation. (LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving)
- LightRot/GyRot Hardware Accelerators: Dedicated hardware featuring hierarchical Fast Hadamard Transform (FHT) units and specialized Gathering Units for outlier handling, implemented in 28nm CMOS, and validated on LLaMA2-13B and LLaMA3-8B using real-world conversational benchmarks like MT-Bench. (LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference, GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference)
- IMC-MCTS Accelerator: A novel in-memory computing accelerator with co-located primitives (CAM, combinational logic, RRAM crossbar, SRAM) capable of running diverse grid-based tasks including Go, Connect Four, and protein folding. Code and simulator will be made available on Zenodo. (Multi-primitive in-memory computing for Monte Carlo tree search)
- SpiNNaker2 Chip: A many-core neuromorphic platform with 152 ARM M4F processors and dedicated accelerators, supporting over 150,000 spiking neurons and DNNs up to 4.5 TOPS. (The SpiNNaker2 chip: a many-core platform for flexible and scalable brain-inspired computing)
- NEORV32 RISC-V with Loop Caches: Open-source implementations of dynamic and static loop caches for the NEORV32 processor, evaluated on LeNet-5 CNN inference synthesized in GlobalFoundries 22nm FDX+. Code is available at https://github.com/wwiebren/NEORV32 loopcache for SparkRV. (Reducing Instruction-Fetch Energy in RISC-V for Embedded AI Processing via Dynamic and Static Loop Caching)
- MDTransformer: Utilizes Tidy3D electromagnetic simulator and a PTA hardware simulator from the Lightening-Transformer (LT) accelerator for verification. (MDTransformer: A Hardware-Software Co-Design of Mode-Division Photonic Transformer Accelerator with Inverse-Designed Coherent Crossbar)
- CIS-Fuse: A spiking neural network evaluated on RoadScene, MSRS, M3FD, and FMB datasets for image fusion tasks. Code will be released at https://github.com/. (Current Injection Spiking Neural Network for Infrared and Visible Image Fusion)
- SparHiXcel-v2: An FPGA accelerator with column-wise kernel compression, evaluated on AMD Kintex UltraScale+ FPGAs for VGG16 and ResNet18. Code available at https://github.com/INRS-ECCoLe/SparHiXcel-Optimized. (A Flexible Sparsity-Aware FPGA Accelerator with Column-Wise Compression for Efficient CNN Inference)
Impact & The Road Ahead
These advancements herald a new era of Green AI, where powerful models can be deployed more sustainably, extending their reach to resource-constrained environments from edge devices to deep space. The ability to perform complex AI tasks with significantly less energy will accelerate the adoption of AI in critical applications like autonomous systems, smart cities, and healthcare.
However, the challenge extends beyond raw technical efficiency. As highlighted in The Fallacy of Sustainable Generative AI: Limitations in EU Environmental Regulation of Data Centres and Paths Forward by Hasso-Plattner-Institute and Oxford Internet Institute, current metrics like PUE and WUE can create an “efficiency paradox,
Share this content:
Discover more from SciPapermill
Subscribe to get the latest posts sent to your email.
Post Comment