Interpretability Unleashed: Navigating the Complexities of AI with New Lenses
Latest 54 papers on interpretability: Nov. 22, 2026
The quest for interpretability in AI/ML continues to accelerate, driven by the need for trust, accountability, and scientific discovery. As models grow in complexity, understanding their internal workings moves from a desirable trait to an absolute necessity across domains like healthcare, autonomous systems, and scientific research. Recent breakthroughs, synthesized from a collection of cutting-edge papers, offer diverse approaches to demystifying AI, from dissecting internal circuits to making black-box models more transparent and efficient.
The Big Idea(s) & Core Innovations
At the heart of these advancements lies a common thread: transforming opaque AI systems into glass-box models or at least equipping us with powerful “lenses” to peer inside. One significant challenge addressed is the trade-off between model performance and interpretability. For instance, “Separating Interpretability and Reconstruction in Sparse Autoencoders” by Anh Tien Nguyen, Dimitris Samaras, and Mahdi S. Hosseini from Stony Brook University and Concordia University, introduces DualSAE. This modular sparse autoencoder improves both reconstruction fidelity and monosemanticity by dedicating a Sparse Branch to interpretable features and a Residual Branch to non-semantic information, effectively decoupling these objectives rather than forcing a compromise. This insight directly tackles the notion that interpretability must come at the cost of performance.
Building on the power of sparse autoencoders, SCALE (Scalable Counterfactuals through Attention-guided Latent Editing in Pathology), presented in “SCALE: Towards Whole-Slide Counterfactual Explanations in Cancer Histopathology” by a team including Weiyi Qin and Hao Wang from the University of Illinois Urbana-Champaign, demonstrates a groundbreaking application in digital pathology. SCALE generates whole-slide counterfactual images by discovering reusable morphological concepts with a sparse autoencoder, then steers features towards target-associated concepts using diffusion models. This innovation shifts explanations from where a model looks to what plausible morphological changes would flip a decision, a crucial step for clinical relevance.
For Large Language Models (LLMs), a key finding from “Gradients for Interventions and Activation Values for Detection: Targeted Feature Learning in Language Models” by Jonathan Drechsel and Steffen Herbold from the University of Passau reveals a systematic trade-off: activation values excel at feature detection, while gradients are superior for causal intervention. This means different interpretability techniques are suited for different goals – a nuanced understanding vital for effective LLM steering and editing.
Addressing the practicalities of LLM deployment, “GateWatch: Low-Overhead LLM Monitoring with Boolean Circuits” by Firas Spanioli and Yaniv David from Technion proposes GateWatch. It converts LLM monitoring probes into compact, human-readable Boolean circuits over binarized SAE features. This innovation significantly reduces memory and serving overhead, making real-time, interpretable LLM monitoring feasible in production environments. Their finding that such probes achieve near-full-SAE accuracy with vastly reduced resources is a game-changer for AI safety and efficient deployment.
Another significant development, “DV-Lens: Revealing the Functional Organization of Language Model Parameters” by Chenhang Cui et al. from the National University of Singapore, introduces DV-Lens. This framework propagates native parameter columns through prompt-averaged downstream Jacobians, revealing their signed effects in vocabulary space. Intriguingly, their DV-Complexity metric, derived from DV-Lens, shows a strong correlation (0.904 Spearman) with LLM benchmark capabilities, suggesting a direct link between the functional organization of parameters and overall model performance.
From a foundational perspective, “The Transformer Paradigm: Dynamic Processing and a New Principle of Inference” by Marco Giunti and Fabrizia Giulia Garavaglia from the University of Cagliari argues that Transformers embody a new principle of inference. Beyond mere scale, they dynamically construct input-dependent transformations during inference, a concept formalized as Sequence-level Interactive Dynamic Parallel Processing (SIDPP). This dynamic nature helps explain strong prompt sensitivity and opens new avenues for interpretability research, moving beyond static weight analysis.
Finally, the theoretical underpinning of interpretability itself is explored in “Mechanistic Interpretability Needs a Logic of Inference” by Zhimin Hu et al. from Georgia Tech. They propose five principles for rigorously moving from observing internal structures to establishing mechanistic claims, emphasizing that interpretability is a recursive process connecting local findings, causal roles, and testable hypotheses. This paper serves as a crucial guide for conducting robust mechanistic interpretability research.
Under the Hood: Models, Datasets, & Benchmarks
These papers leverage and contribute to a rich ecosystem of models and datasets, pushing the boundaries of what’s possible in interpretability:
- DualSAE demonstrated its versatility across various Sparse Autoencoder variants, vision encoders (CLIP, SigLIP2, ViT-L/16), and language models (Pythia), using ImageNet-1K, OpenImagesV7, OpenWebText, and GitHub code datasets. The code is publicly available at https://github.com/AtlasAnalyticsLab/DualSAE.
- SCALE was trained on 141.1 million UNI2-h tile representations from 10,484 TCGA whole-slide images, showcasing its pan-cancer applicability. It uses the UNI2-h pathology foundation model from Hugging Face.
- The study on Gradients for Interventions and Activation Values for Detection extensively benchmarked six targeted methods against pretrained sparse autoencoders across 15 tasks and ten diverse language models, including Pythia, Mistral-7B, Llama 3, Gemma 3, DeepSeek-R1, and Qwen 3.5. It utilizes resources like SAELens and the RAVEL interpretability benchmark.
- GateWatch was evaluated for accuracy, memory, and throughput across 24 classification tasks using Gemma-2-9B and Gemma Scope SAEs, showing compatibility with the vLLM serving system. It utilizes Neuronpedia for feature descriptions.
- The DV-Lens framework was validated against 21 language models and its DV-Complexity metric correlated with capabilities across a fixed cohort of 48 LLMs from the Open LLM Leaderboard v2. Models include Falcon, Hermes, Llama, Mistral, Phi, Qwen, and others from Hugging Face.
- The Transformer Paradigm paper is theoretical, but references the reported GPT-4.5 three-party Turing-test results as a behavioral illustration.
- LMLineage: Content Lineage and Context Governance for LLM Agents offers an out-of-band system for LLM-agent sessions, using a 71-sentence Chinese web-QA attribution benchmark. It’s implemented as an OpenClaw plugin and uses sentence-transformer-based embeddings.
- VRLLM: Turning LLMs into Powerful Tabular Classifiers adapted a Llama 3.2 1B decoder-only LLM for tabular classification, outperforming specialized models on Visa’s proprietary industrial-scale benchmark. It uses Llama 3 and other tabular foundation models for comparison.
- ReMoE: An Interpretable Mixture-of-Experts Framework for Human Mobility Recovery Forecasting Across Disasters leverages over 219 million visit records across 18 disasters, integrating various public datasets like CDC SVI, EPA SLD, and NOAA weather data. Code is at https://github.com/lvyl9909/ReMoEKDD27.
- RLDE: A Reinforcement Learning Framework for Interpretable Iterative Document Editing utilizes BART-base for infilling, Sentence-BERT for semantic preservation, and GPT-2 for perplexity evaluation on datasets like BEA-2019 and CoNLL-2014. It formulates editing as a reference-free Markov Decision Process.
- Disambiguating Moral Circuits and Moral Features in Language Models used 227 human-validated minimal pairs and the ETHICS benchmark across 14 Llama, Gemma, and Qwen models. Code is at https://github.com/daviligade/normative-mechanistic-interpretability.
- ReShap: Interpretable Reinforcement Learning for High-Dimensional Multi-Action Settings evaluates on Atari (MsPacman, RoadRunner) and Box2D (LunarLander, CartPole) environments, distilling DQN, A2C, and PPO policies. Code is at https://anonymous.4open.science/r/ReShap-E14E/.
- ICA Lens: Non-Gaussianity Reveals Interpretable Directions in Language Models applies its FastICA workflow to GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base. The
icalensPython package is available viapip install icalensand at https://liusida.github.io/ica-lens-paper/. - Understanding Contextual Bias in Agricultural CNNs uses five agricultural datasets (PlantVillage, PlantDoc, DeepWeeds, Cassava, WeedCrop) and four CNN architectures, integrating UDAT descriptors and OWL knowledge graphs.
- Can Mechanistic Interpretability Help Explain LLM Agent Failures? is a case study on web element hallucination using Qwen2.5-7B-Instruct on live Online-Mind2Web environments.
- Towards Explainable Sleep Stage Classification using Single-Channel Electroencephalogram uses Sleep-EDF and DREAMT datasets to compare XAI methods on a CNN-CRF architecture. Code is at https://github.com/GeetSandhu/xai-2d–eeg-sleep-staging.
- Causal AI-Driven Secure mmWave Beam Alignment uses the DeepMIMO dataset and Boston5G ray-tracing scenarios, applying DirectLiNGAM for causal feature selection. The DirectLiNGAM implementation is open-source at https://github.com/ashishb/DirectLiNGAM.
- Interpretable weight-space programming builds transparent transformers and brain encoding models utilizes LAION-fMRI, Natural Scenes Dataset, TREC, SST-2, MRPC, and SNLI, with models like GPT-2 XL, Whisper-large-v3, and AlexNet. Code is at https://github.com/csinva/weight-space-programming-autoresearch.
- Learning nonlinear interactions and cross-feature couplings with product-unit residual networks evaluates CV-PURe on Friedman synthetic benchmarks and real-world datasets like Concrete Compressive Strength, California Housing, and Bike Sharing. Code is at https://github.com/ziyuan-li/1d_pure.
- DECODE: Diagnosing Interpretability Failures in Automated Interpretation Process uses Gemma-2-9B layer-31 residual-stream SAE features and LLM explainers (Llama 3.1 70B, GPT-4o mini, Gemini 2.5 Flash-Lite) from Neuronpedia.
- Activation Steering the Nan-Shan in Typhoon uses Llama-3.2-3B-Instruct and a 529-paragraph Beaufort-scale intensity dataset derived from Joseph Conrad’s Typhoon. The audiovisual installation is at https://stormhelm.net/.
- A Tumour-Aware Multimodal Framework for Explainable Prediction of IDH Mutation in Gliomas uses the RSNA–ASNR–MICCAI BraTS 2021 glioma dataset and PyRadiomics.
- Enhancing Quality of Experience in Future Telecommunication Networks is a survey referencing ITU-T P.1203 and G.1051 standards and frameworks like QoE-DASH. Code for P.1203 is at https://github.com/itu-p1203/itu-p1203.
- Exploring Post-Hoc Disentanglement of Audio Foundation Models adapts CLAP, MuQ, and MERT on MTG-Jamendo and MagnaTagATune datasets.
- SIGMO: Sigmoid Gated Interaction-Aware Motion Planning uses MPPI on MetaDrive scenarios initialized from the Waymo Open Dataset, validated on a 1/10-scale F1Tenth vehicle. Simulation videos are at https://sites.google.com/view/sigmoid-polygate.
- Causal Concept Explanations for Deep Neural Models evaluates CCW-Net on aircraft control, CarRacing-v1, and CUB-200-2011 bird classification.
- Reliable and explainable analysis of AVS/RS by Statistical Model Checking and Process Mining uses a new open-source MATLAB DES simulator for multi-deep AVS/RS, integrated with MultiVeStA and Disco, on a 72-hour horizon. The simulator is available on Zenodo: https://doi.org/10.5281/zenodo.18629711.
- Are Causal Neuron Circuits Unique and Transferable? evaluates all 4,096 subsets of 12-neuron groups in Qwen3, Llama 3.2, and Gemma 3 models. Code at https://github.com/omtekyav/causal-neuron-circuits.
- Mechanistic Interpretability of Deep Neural Networks: Methods, Frameworks, and Perspectives is a survey discussing various techniques, with examples using ViT-B/16 on ESC-50 and convolutional autoencoders on CIFAR-10/ImageNet-1K.
- Evidential Deep Learning with conditional entropy for glaucoma detection uses MobileNetV3 on EyePACS-AIROGS-light-v2, RIM-ONE-DL, CRFO, and ACRIMA datasets.
- The Shape of a Program: Path Signatures for Trace-to-Program Induction compares log-signatures against LSTM, Transformer, and Fourier encoders on Karel, List, and RASP tasks. Project page: https://oschart.github.io/program-signature.
- GraphToolbox: A Configurable Framework for Controlled Benchmarking of Graph Neural Network Forecasting is an open-source PyTorch Geometric framework, using RTE French electricity data and Météo-France SYNOP weather observations.
- Challenges, Opportunities, and Misconceptions in Neurosymbolic AI is a perspective paper, referencing various neurosymbolic methods and applications.
- Robust Deep Prototype Classifiers via Lipschitz Training experiments with Lipschitz-GLVQ on MNIST, CIFAR-10, and Imagenette-160.
- Interpretable Context-Aware Multi-Label Retinal Classification uses a dual-hub GNN on 16,266 BRSET fundus photographs. Code at https://github.com/proshanto-c/ca-mlc-refactor.
- Cross-Layer TopK: Dynamically Allocating Sparsity Across Layers in Transcoders utilizes Qwen3-8B-Base and the Olmo 3 corpus. Released weights: https://huggingface.co/ByteDance-Seed/Cross-Layer-TopK-CLT.
- RAG-Enhanced LLMs for Interactive Explainability in Clinical Decision Support Systems uses a Random Forest classifier, LIME, MedCPT, and Mixtral-8x7B-Instruct-v0.1 on the Pima Indians Diabetes Database, grounded in the StatPearls medical corpus. MedRAG toolkit: https://github.com/Teddy-XiongGZ/MedRAG.
- Calibrated concept profiles: making the reading of a model’s neurons a measurement applies to Python constructs in OpenCoder-1.5B across 61 checkpoints. Battery and code: https://huggingface.co/datasets/piotrwilam/AtlasOpenCoder and https://github.com/piotrwilam/AtlasOpenCoder.
- Beyond the Grid: Continuous Dictionary Pursuit for Interpretable Signal Decomposition introduces CDP, evaluated on custom synthetic and 13 LTSF datasets (ETT, Electricity, Exchange, Traffic, Solar, Weather, PEMS). Code at https://github.com/nick-janssen/ContinuousDictionaryPursuit.
- Knowledge Distillation from Neural Networks to Differentiable Logic Networks evaluates KD for DLNs on 20 public tabular datasets, including UCI AdultIncome.
- Glyph: An Interpretability Engine for AI-Driven Scientific Discovery in Clinical Data uses Merlin CT, Qwen3-Embedding-8B text, and Labs foundation models on 41,628 patients. Code at https://github.com/StanfordMIMI/Glyph.
- MEDALLION: Manifold Embedding Distillation and Attributions for Low-dimensional Latent InterpretatiON interprets embeddings from PCA, t-SNE, UMAP on MNIST-64, melanoma immune-cell, and mouse neocortical transcriptomics. Project page: https://arxiv.org/pdf/2611.08767.
- Random Symbolic Boosting: A Geometric Approach to Symbolic Regression is evaluated on SRBench and TALENT benchmarks.
- Higher Order Component Attribution via Learned Surrogates in High Energy Physics uses KAN surrogates for transformer-based jet taggers on the JETCLASS dataset.
- Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks introduces Vis-Lens, using CLIP embeddings on the Natural Scenes Dataset, SimCLR, and Vision Transformers. Code at https://github.com/NeuroAIHub/Vis-Lens.
Impact & The Road Ahead
The implications of these advancements are profound. In healthcare, systems like SCALE and Glyph offer not just better predictions but clinically meaningful, traceable explanations, moving AI closer to real-world diagnostic and prognostic applications. The ability to identify high-risk concepts and confounder-adjusted associations, as seen with Glyph, unlocks true AI-driven scientific discovery, going beyond mere correlation.
For LLMs, the distinction between detection- and intervention-focused interpretability, the efficiency gains of GateWatch, and the parameter-level insights from DV-Lens will be crucial for building more reliable, controllable, and deployable agents. The nuanced understanding of reasoning traces, as explored in “Tracking Representations in LLM Reasoning Traces for Iterated Games” by Chaipat Tirapongprasert et al. from Columbia University, paves the way for preventing “belief lock-in” and other strategic failures in multi-agent systems. Similarly, “Can Mechanistic Interpretability Help Explain LLM Agent Failures?” highlights the need for agent-specific interpretability methods that account for distributed, temporal, and stateful computation, acknowledging the limitations of current techniques and proposing a roadmap for future research.
More broadly, the drive towards interpretable-by-construction models, as exemplified by iWSP, RSB, CDP, and DLNs, promises more transparent and auditable AI from the ground up, reducing the need for post-hoc explanations. The theoretical work on a “Logic of Inference” and the “Transformer Paradigm” challenges us to think more deeply about what constitutes a mechanistic explanation and how we characterize new AI architectures. These efforts will not only enhance the reliability and safety of AI but also accelerate scientific discovery by allowing us to formulate and test hypotheses about complex systems with unprecedented clarity.
The road ahead will involve developing more unified frameworks that seamlessly integrate these diverse interpretability techniques, scaling them to even larger and more complex models, and rigorously validating them in real-world, high-stakes scenarios. As we gain a deeper understanding of how AI thinks, we move closer to building truly intelligent systems that are not only powerful but also trustworthy and aligned with human values. The era of inscrutable black boxes is slowly but surely giving way to a future of transparent and explainable AI.
Share this content:
Discover more from SciPapermill
Subscribe to get the latest posts sent to your email.
Post Comment