In-Context Learning: Revolutionizing AI Across Language, Vision, and Tabular Data
Latest 44 papers on in-context learning: Feb. 14, 2026
In-context learning (ICL) has emerged as a transformative paradigm in AI, allowing models to adapt to new tasks and learn from examples provided directly in their input, without requiring explicit fine-tuning. This ability to ‘learn on the fly’ is rapidly reshaping how we approach complex problems across diverse domains, from natural language understanding to computer vision and tabular data analysis. Recent research showcases not only the burgeoning power of ICL but also innovative strategies to optimize its efficiency, reliability, and reach.
The Big Idea(s) & Core Innovations
The overarching theme uniting recent advancements in ICL is the pursuit of more adaptable, efficient, and robust AI systems. Researchers are tackling key challenges such as demonstration selection, model interpretability, and the inherent biases of large models. For instance, Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learning by Xubin Wang and Weijia Jia (BNU-BNBU Institute of Artificial Intelligence and Future Networks, Beijing Normal-Hong Kong Baptist University) introduces a supervised meta-learning framework that efficiently selects optimal demonstrations for ICL. Their approach uses lightweight features like TF-IDF similarity, making demonstration selection scalable and interpretable, particularly benefiting smaller models. This directly addresses the computational overhead often associated with ICL.
Simultaneously, understanding the fundamental mechanisms of ICL is crucial. Elif Akata and her colleagues from Helmholtz Munich in In-Context Function Learning in Large Language Models formalize ICL as non-parametric regression, analyzing how LLMs learn continuous functions. Their work reveals that LLMs tend to predict functions under less smooth Gaussian Process kernels, but post-training with reinforcement learning can shift these inductive biases towards smoother, more data-driven functions, improving sample efficiency.
Extending ICL beyond traditional NLP, TabICLv2: A better, faster, scalable, and open tabular foundation model from Jingang Qu, David Holzmüller, Gael Varoquaux, and Marine Le Morvan (SODA Team, INRIA Saclay) showcases a state-of-the-art tabular foundation model that leverages ICL for superior performance without fine-tuning. This, alongside TFMLinker: Universal Link Predictor by Graph In-Context Learning with Tabular Foundation Models by Tianyin Liao et al. (Nankai University, Tsinghua University, Beihang University), which integrates tabular foundation models (TFMs) for universal link prediction across diverse graphs, highlights ICL’s power in generalizing across structured data domains. TFMLinker, in particular, uses a prototype-augmented local-global context module and topology-aware link encoder to capture both graph-specific and transferable patterns.
In the realm of multimodal applications, VIRAL: Visual In-Context Reasoning via Analogy in Diffusion Transformers by Zhiwen Li et al. (East China Normal University, Alibaba Group) introduces a generative formulation of visual ICL, using visual analogy to enable pre-trained image editing models to perform diverse tasks like perception and open-domain editing. Similarly, HOICraft: In-Situ VLM-based Authoring Tool for Part-Level Hand-Object Interaction Design in VR by Dohui Lee et al. (KAIST, New York University) integrates Vision-Language Models (VLMs) with ICL for intuitive part-level hand-object interaction design in VR, significantly reducing manual effort. This shows how ICL is pushing the boundaries of interactive AI. For challenging tasks like anomaly detection, Enhancing Weakly Supervised Multimodal Video Anomaly Detection through Text Guidance by Shengyang Sun et al. (Nanjing University) leverages text guidance and ICL to achieve state-of-the-art results in multimodal video anomaly detection.
The challenge of model reliability and robustness is addressed by Beyond Confidence: The Rhythms of Reasoning in Generative Models by Deyuan Liu et al. (Harbin Institute of Technology, Wechat AI), which introduces δTCB (Token Constraint Bound) to measure the local robustness of LLM predictions against internal state perturbations. This metric effectively assesses prompt quality and refines prompt engineering and ICL, revealing how effective context leads to more stable internal states. This quest for reliability extends to cybersecurity with Hallucination-Resistant Security Planning with a Large Language Model by Kim Hammar and Rudolf Stadler (Chalmers University of Technology, Sweden), which fine-tunes LLMs and uses structured prompts to reduce hallucinations in security planning, enhancing the reliability of automated incident response.
Addressing the critical issue of catastrophic forgetting in continual learning, Djohan Bonnet et al. (Forschungszentrum Jülich, RWTH Aachen) propose Palimpsa in Learning to Remember, Learn, and Forget in Attention-Based Models. This self-attention model uses Bayesian metaplasticity to dynamically adjust memory states, preserving crucial information while allowing for efficient forgetting of outdated knowledge, thus improving performance on complex reasoning tasks with fixed-size memories. For time series forecasting, Jiecheng Lu et al. (Georgia Institute of Technology) in In-context Time Series Predictor introduce ICTSP, a framework that reformulates TSF tasks using (lookback, future) pairs as input tokens, allowing for efficient, parameter-free prediction without relying on pre-trained LLM parameters.
Under the Hood: Models, Datasets, & Benchmarks
Innovation in ICL is deeply intertwined with advancements in foundational models, novel datasets, and rigorous benchmarks:
- TabICLv2: Introduces architectural innovations like a new scalable softmax in attention and optimized pretraining protocols using the Muon optimizer. It also features a novel synthetic data generation engine for high pretraining diversity and is evaluated on TabArena and TALENT benchmarks. Code is available at https://github.com/soda-inria/nanotabicl.
- CL-bench: A groundbreaking benchmark for context learning, consisting of 500 complex real-world contexts, 1,899 tasks, and 31,607 verification rubrics. It highlights the significant performance gaps in current LMs, showing models solve only ~17.2% of tasks on average. Access at https://clbench.com/.
- Demo-ICL-Bench: Proposed in Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition, this challenging benchmark evaluates ICL capabilities for learning from instructional videos. The associated Demo-ICL method uses MLLMs enhanced with two-stage training. Code is available at https://github.com/dongyh20/Demo-ICL.
- ArtifactLens: Utilizes Vision-Language Models (VLMs) with a multi-component architecture and black-box optimization methods (counterfactual demonstrations, full-spectrum prompting) to achieve state-of-the-art artifact detection with minimal labeled data. Code at http://jmhb0.github.io/ArtifactLens.
- OUTFORMER: The first foundation model for zero-shot tabular outlier detection, leveraging mixed synthetic priors (GMMs, SCMs, Copulas) and a self-evolving curriculum training strategy. It is evaluated on ADBench and new benchmarks. Code: https://github.com/psorus/Outformer.git.
- TabularMath: A benchmark for evaluating computational generalization in tabular learning, featuring 233k labeled rows from GSM8K and AIME, generated via program-verified synthesis. Code: https://github.com/Marco-Cheng/TabularMath.
- HoliAntiSpoof: An Audio LLM (ALLM) framework for holistic speech anti-spoofing, introducing the DailyTalkEdit dataset for semantic influence analysis in conversational spoofing scenarios. Code: https://github.com/wsntxxn/HoliAntiSpoof.
- Private PoEtry: Employs a Product-of-Experts (PoE) model for private ICL, outperforming DP-ICL across text, math, and vision-language tasks. Code: https://github.com/robromijnders/private-poe.
- ORBIT: A multi-task, multi-episode meta-reinforcement learning framework that allows LLMs (e.g., Qwen3-14B) to perform in-context online learning and adapt without weight updates. Code: https://github.com/XiaofengLin7/ORBIT.
- LUCID Attention: Introduced in LUCID: Attention with Preconditioned Representations by Sai Surya Duvvuri et al. (The University of Texas at Austin, Google), it improves standard softmax attention for long contexts by applying a preconditioner based on key-key similarities. Code: https://zenodo.org/records/12608602.
Impact & The Road Ahead
These advancements signify a paradigm shift towards more flexible, adaptable, and computationally efficient AI. The ability of LLMs to dynamically learn from context without explicit weight updates promises to revolutionize real-world applications in several areas. For instance, in control systems, Learning Nonlinear Systems In-Context: From Synthetic Data to Real-World Motor Control by Tong Jian et al. (Analog Devices, Inc.) demonstrates how ICL can generalize from synthetic data to real-world physical systems with minimal examples, offering a data-efficient framework for adapting control strategies. In scientific discovery, In-Context System Identification for Nonlinear Dynamics Using Large Language Models by Author A and Author B (University of Example, Institute of Advanced Research) shows LLMs discovering symbolic governing equations from data, an unprecedented leap for physics-informed AI. Moreover, Computing Conditional Shapley Values Using Tabular Foundation Models by Lars H. B. Olsen and Danniel Christensen (University of Bergen, Norway) shows TabPFN’s efficacy in computing conditional Shapley values, unlocking new possibilities for model interpretability in Explainable AI (XAI). Further, GAMformer: Bridging Tabular Foundation Models and Interpretable Machine Learning by Andreas Mueller et al. (Microsoft Research, University of Freiburg) offers the first tabular foundation model for Generalized Additive Models (GAMs), blending ICL with interpretability for transparent, high-performance tabular modeling.
However, challenges remain. When Does Context Help? Error Dynamics of Contextual Information in Large Language Models by Dingzirui Wang et al. (Harbin Institute of Technology, Singapore Management University) explores the theoretical role of context, showing that adaptive retrieval is often a signal of uncertainty, and models scale its use with problem difficulty—suggesting that the decision to use context is as important as the context itself. This metacognitive aspect points to the need for LLMs to understand their own limitations. Similarly, Relational reasoning and inductive bias in transformers and large language models by Jesse P. Geerts et al. (Imperial College London, Google DeepMind) indicates that while in-weights learning (IWL) naturally induces transitive inductive bias, in-context learning (ICL) requires specific pre-training to achieve similar results, emphasizing the nuanced interaction between pre-training and ICL.
The future of ICL is bright, pushing towards systems that are not just intelligent, but also self-aware, robust, and ethical. From automating security responses and improving multi-domain machine translation, as seen in Consensus-Aligned Neuron Efficient Fine-Tuning Large Language Models for Multi-Domain Machine Translation by Shuting Jiang et al. (Kunming University of Science and Technology), to enabling efficient reinforcement learning without explicit rewards (Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers by Juncheng Dong et al. from Duke University), ICL is set to redefine what’s possible in AI, making models more capable, versatile, and seamlessly integrated into real-world challenges.
Share this content:
Post Comment