O(N) & O(N log log N) Breakthroughs: The Latest in AI/ML Efficiency and Scalability

Latest 50 papers on computational complexity: Nov. 16, 2025

The quest for greater efficiency and scalability is a perennial challenge in AI/ML, especially as models and datasets explode in size. Computational complexity often acts as a bottleneck, hindering widespread adoption and real-time performance. However, recent research is pushing the boundaries, offering groundbreaking solutions that dramatically reduce complexity, from quadratic (or even quartic) to linear, or even a remarkable O(N log log N) in specific domains. This digest dives into some of the most exciting advancements, highlighting how researchers are rethinking algorithms, architectures, and theoretical foundations to unlock unprecedented efficiency.

The Big Idea(s) & Core Innovations

Many of the papers explore novel approaches to tackle the inherent complexities of large-scale AI/ML tasks. A major theme is the strategic reduction of computational scaling, often from quadratic to linear, enabling operations that were once intractable. For instance, in language models, the traditional quadratic scaling of attention mechanisms is a significant hurdle. Mingkuan Zhao et al. from Xi’an Jiaotong University and Tsinghua University, in their paper “Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off” introduce SPAttention. This novel sparse attention mechanism reorganizes computations through Principled Structural Sparsity, achieving O(N²) computational complexity (a quadratic improvement over the common O(H·N²) for multi-head attention) without sacrificing performance, by partitioning workload into non-overlapping bands for each head.

Similarly, Gimun Bae and Seung Jun Shin from Korea University, in “Scaling Up ROC-Optimizing Support Vector Machines”, tackle the O(N²) complexity of ROC-optimizing SVMs. Their method leverages incomplete U-statistics and low-rank kernel approximations to reduce it to O(N), making ROC-SVM viable for large datasets. This theme of linear scaling is echoed in graph learning, where Xiang Chen et al. from Yunnan University, in “Dual-Kernel Graph Community Contrastive Learning” present DKGCCL. This framework drastically cuts GCL training complexity from quadratic to linear time by employing a dual-kernel contrastive loss and knowledge distillation.

Beyond linear scaling, some breakthroughs achieve even better. Atsuki Sato and Yusuke Matsui from The University of Tokyo present “PCF Learned Sort: a Learning Augmented Sort Algorithm with O(n log log n) Expected Complexity”. This groundbreaking algorithm uses machine learning augmented with Piecewise Constant Functions (PCF) to achieve an expected complexity of O(n log log n), a significant leap over traditional O(n log n) sorting methods, complete with theoretical guarantees. For variational inference, Joohwan Ko et al. from KAIST and University of Pennsylvania, in “Provably Scalable Black-Box Variational Inference with Structured Variational Families”, prove that structured scale matrices can reduce BBVI’s iteration complexity from 𝒪(𝑁²) to 𝒪(𝑁), bridging the gap between mean-field and full-rank approximations.

Innovative architectural designs also play a crucial role. Noam Koren et al. from Technion and EPFL, in “SVD-NO: Learning PDE Solution Operators with SVD Integral Kernels”, introduce a neural operator that uses Singular Value Decomposition (SVD) to parameterize PDE solution operators, achieving high expressivity while outperforming Fourier- and graph-based methods. For 3D human pose estimation, Hu Cui et al. from Nagaoka University of Technology, with “SasMamba: A Lightweight Structure-Aware Stride State Space Model for 3D Human Pose Estimation”, leverage a novel Structure-Aware Stride SSM (SAS-SSM) module in their SasMamba model. This provides linear computational complexity and competitive performance by preserving spatial topology and capturing multi-scale dependencies without expensive attention mechanisms.

Under the Hood: Models, Datasets, & Benchmarks

These innovations are often underpinned by novel model architectures, specialized datasets, or new benchmarks that enable their development and validation. Many papers also provide open-source code, fostering reproducibility and further research.

• SPAttention (for LLMs): Introduces a principled structural sparsity paradigm. Implemented without explicit code repository mentioned, but builds on transformer architectures. The underlying idea is a restructuring of multi-head attention to achieve O(N²) from O(H·N²). (Paper Link)
• PCF Learned Sort: A machine learning-augmented sorting algorithm with O(n log log n) expected complexity. Code available at https://github.com/anikristo/LearnedSort. Tested on real-world datasets like NYC Taxi Trip Record Data and Chicago taxi data.
• SVD-NO (Neural Operators for PDEs): Leverages Singular Value Decomposition for parameterizing PDE solution operators. Code available at https://github.com/2noamk/SVDNO.git. Outperforms Fourier- and graph-based neural operators on diverse PDEs.
• AFCF (Scalable Fair Clustering): A general anchor-based framework reducing complexity from quadratic to linear for fair clustering. Code available at https://github.com/smcsurvey/AFCF.
• SasMamba (3D Human Pose Estimation): A lightweight State Space Model-based architecture. Code and project page at https://hucui2022.github.io/sasmamba_proj/. Evaluated on standard benchmarks: Human3.6M and MPI-INF-3DHP.
• DKGCCL (Graph Contrastive Learning): Employs a dual-kernel graph community contrastive loss for scalable GNN training. Code available at https://github.com/chenx-hi/DKGCCL. Evaluated on 16 real-world datasets.
• BOKE (Bayesian Optimization): Reduces computational complexity from O(T^4) to O(T^2) using kernel regression. (Paper Link)
• LoKO (Low-Rank Kalman Optimizer): A Kalman-based optimizer for online fine-tuning of large models, using low-rank adaptation. Code available at https://github.com/abdi-hossein/Loko.
• RefiDiff (Missing Data Imputation): Combines predictive and generative methods with a Mamba-based denoising model. Code available at https://github.com/Atik-Ahamed/RefiDiff.
• 2S-AVTSE (Audio-Visual Target Speaker Extraction): A two-stage system for real-time processing on edge devices, leveraging a simplified VVAD network and 3D talking portrait generation. Code available at https://github.com/cslzx/2S-AVTSE. Utilizes LRS2-2mix, LRS3-2mix, and VoxCeleb2-2mix datasets.
• SharpV (VideoLLMs): A two-stage pruning framework for efficient visual token processing. Code available at https://github.com/JalenQin/SharpV.
• FractalCloud: A fractal-inspired architecture for efficient large-scale point cloud processing. Code available at https://github.com/fractalcloud-team/fractalcloud.
• EALA (Efficient Linear Attention): A linear attention mechanism for multivariate time series modeling based on entropy equality. Code available at https://github.com/MingtaoZhang/EALA.
• Efficient Dynamic MaxFlow: GPU-based Push-Relabel algorithms for dynamic graphs. Code available at https://github.com/ShruthiKannappan/dyn_maxflow.
• S4F Standpoint Logic: A novel formalism unifying non-monotonic reasoning with multi-viewpoint semantics. (Paper Link)
• FAQNAS: A FLOPs-aware hybrid quantum neural architecture search using genetic algorithms for NISQ devices. (Paper Link)
• 4KDehazeFlow: Ultra-high-definition image dehazing via flow matching with a learnable 3D LUT. (Paper Link)
• Efficient Distributed Exact Subgraph Matching via GNN-PE: A framework with load balancing, caching optimization, and query plan ranking. (Paper Link)
• Dense Cross-Scale Image Alignment: An unsupervised method with fully spatial correlation and JND guidance. (Paper Link)
• Multi-Level Damage-Aware Graph Learning: Enhances UAV swarm network resilience. Code available at https://github.com/lytxzt/Damage-Attentive-Graph-Learning.
• DOA Estimation with Lightweight Network on LLM-Aided Simulated Acoustic Scenes: Uses LLM-generated acoustic scenes for DOA estimation. Utilizes the BEWO-1M dataset. (Paper Link)
• CometNet: Contextual Motif-guided Long-term Time Series Forecasting. (Paper Link)
• Information Capacity: A metric for LLM efficiency via text compression. (Paper Link)
• MirrorMamba: Mamba-based video mirror detection. (Paper Link)
• Variable-order fractional wave equation: Fast divide-and-conquer algorithm reduces complexity to O(MN log² N). (Paper Link)
• Random Construction of Quantum LDPC Codes: ILP-based repair for scalable quantum code design. (Paper Link)
• Efficient and rate-optimal list-decoding: Achieves optimal rates with minimal feedback in adversarial channels. (Paper Link
• Federated Learning with Gramian Angular Fields: Privacy-preserving ECG classification on IoT devices. Code available at https://github.com/your-organization/federated-ecg.
• LaMoS: SRAM-based CiM acceleration for large number modular multiplication. (Paper Link)
• VecComp: Vector Computing via MIMO Digital Over-the-Air Computation. (Paper Link)

Impact & The Road Ahead

The collective impact of this research is profound. By tackling computational complexity head-on, these advancements pave the way for more scalable, efficient, and robust AI/ML systems. From making fair clustering practical for massive datasets to enabling real-time audio-visual processing on edge devices, the implications are far-reaching. Quantum computing is also seeing significant strides, with methods for efficient quantum LDPC code construction and quantum Monte Carlo algorithms for finance, hinting at a future where quantum advantage tackles classically intractable problems.

The ability to integrate multiple viewpoints in logical reasoning, as explored in “Non-Monotonic S4F Standpoint Logic”, or the development of lightweight 3D human pose estimation with “SasMamba: A Lightweight Structure-Aware Stride State Space Model for 3D Human Pose Estimation”, showcases how efficiency doesn’t have to come at the cost of sophistication or accuracy. We’re seeing a clear trend towards algorithms and architectures that are not only powerful but also designed with resource constraints and real-world deployment in mind.

The road ahead will likely involve further exploration of hybrid approaches, combining the best of classical and quantum computing, and continuing to refine approximate algorithms that offer strong theoretical guarantees with practical scalability. As researchers continue to innovate, we can expect to see AI/ML permeate more domains, from energy management to critical communication networks, unlocking new capabilities and pushing the boundaries of what’s possible.