In-Context Learning: Unlocking New Frontiers from Transformers to Real-World AI

Latest 50 papers on in-context learning: Nov. 30, 2025

In-context learning (ICL) has emerged as a cornerstone of modern AI, allowing large language models (LLMs) to adapt to new tasks with minimal or no fine-tuning, simply by providing a few examples within the prompt. This remarkable capability is propelling advancements across diverse fields, from natural language processing to computer vision and even complex scientific modeling. Recent research delves deep into both the theoretical underpinnings and practical applications of ICL, pushing its boundaries and addressing critical challenges.

The Big Idea(s) & Core Innovations

At its heart, ICL leverages pre-trained knowledge, enabling models to generalize and perform new tasks efficiently. A key theme emerging from recent papers is the continuous effort to refine how models learn in-context, improve their robustness, and extend their applicability to more complex, real-world scenarios.

For instance, the paper “Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels” by Anantha Padmanaban Krishna Kumar from Boston University highlights a fundamental limitation: smaller LLMs struggle to override pre-trained label semantics, even with inverted demonstrations. This suggests that ICL primarily adjusts how inputs map to stable semantic directions rather than redefining core meanings, a concept termed ‘semantic anchors’. Complementing this, Warren Li et al. from UC San Diego in “Order Matters: Rethinking Prompt Construction in In-Context Learning” challenge the conventional wisdom that example selection is paramount, demonstrating that example ordering can have a comparable impact on ICL performance, often in a dataset-dependent and non-transferable manner. This underscores the subtle yet powerful influence of prompt design.

Innovations also extend to specialized domains. In “TSFM in-context learning for time-series classification of bearing-health status” by C. Feng et al., Time Series Foundation Models (TSFMs) are adapted for industrial predictive maintenance, achieving high accuracy in bearing health classification with few-shot prompting, a testament to ICL’s efficiency in data-scarce environments. Similarly, Chin-Chia Michael Yeh et al. from Visa Research, in “TiCT: A Synthetically Pre-Trained Foundation Model for Time Series Classification”, introduce a foundation model for time series classification that uses synthetic data pre-training and novel architectures for robust ICL, significantly reducing reliance on extensive labeled data.

The push for multimodal and ethical AI is also evident. Dawei Li et al. (Arizona State University, University of Rochester, and others) address fairness in multimodal medical diagnosis with their “Fairness in Multi-modal Medical Diagnosis with Demonstration Selection” paper, proposing FADS, a fairness-aware demonstration selection method that reduces demographic disparities. For vision, Shao-Jun Xia et al. from Duke University and Texas A&M University introduce “T2T-VICL: Unlocking the Boundaries of Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs”, enabling cross-task visual ICL without additional training by leveraging implicit textual descriptions. This shows how ICL can unlock complex reasoning across different visual tasks.

Under the Hood: Models, Datasets, & Benchmarks

These advancements are often powered by novel architectures, specialized datasets, and rigorous benchmarks:

• G²VLM: Introduced by Wenbo Hu et al. (Shanghai AI Lab, UCLA, and others) in “G²VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning”, this is a unified vision-language model that bridges 3D reconstruction and high-level spatial understanding using dedicated geometric and semantic perception experts. Code is available at https://github.com/ShanghaiAI/G2VLM.
• TiCT: From Chin-Chia Michael Yeh et al. (Visa Research), this foundation model for time series classification features scalable bit-based label encoding and an output attention mechanism, pre-trained on synthetic data. Explore more at https://sites.google.com/view/tsicl.
• ExDDV: The first dataset and benchmark for explainable deepfake detection in video, introduced by Vlad Hondru et al. (University of Bucharest, West University of Timisoara). It comprises ~5.4K videos with manual text and click annotations. Code: https://github.com/vladhondru25/ExDDV.
• KDR-Agent: Proposed by Wenxuan Mu et al. (Dalian Maritime University, Dalian Minzu University), this multi-agent LLM framework enhances low-resource in-context Named Entity Recognition (NER) by integrating knowledge retrieval, disambiguation, and reflective analysis. Code is at https://github.com/MWXGOD/KDR-Agent.
• PRISM: Developed by Chun Chet Ng et al. (AI Lens, Kuala Lumpur, Malaysia), PRISM is a training-free framework for financial information retrieval leveraging prompt-refined system modeling and multi-agent systems, evaluated on the FinAgentBench dataset. Code: https://bit.ly/prism-ailens.
• VRD-UQA: A new benchmark by Davide Napolitano et al. (Politecnico di Torino) for evaluating Visual LLMs’ resilience to unanswerable questions on multi-page visually rich documents, with code at https://github.com/DavideNapolitano/VRD-UQA.
• LG-DUMAP: Presented by Sai Puppala et al. (University of Texas at El Paso, Southern Illinois University Carbondale), this LLM-guided framework enhances personalized federated graph learning through cross-modal alignment and privacy-preserving aggregation. See the paper at https://arxiv.org/pdf/2511.09438.

Impact & The Road Ahead

These collective advancements significantly deepen our understanding of ICL and its capabilities. From refining prompt engineering for Arabic Text-to-SQL (as seen in S. Almohaimeed et al. from King Abdulaziz University, Saudi Arabia, in “Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic”) to formalizing privacy auditing for DP-ICL (by Zhengyuan Liu et al. from Columbia University in “Tight and Practical Privacy Auditing for Differentially Private In-Context Learning”), researchers are tackling both performance and ethical considerations.

The implications are far-reaching. Imagine more accurate and fairer medical diagnostic AI, resilient autonomous driving systems (as explored by P. Wang et al. from UC Berkeley and others in “Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RL”), or even LLMs that can truly ‘understand’ and form theories about their environment through curiosity-driven exploration, as suggested by Guillaume Levy et al. (Inria, Univ. of Bordeaux, MIT, Hugging Face) in “WorldLLM: Improving LLMs’ World Modeling Using Curiosity-Driven Theory-Making”.

The ability of transformers to implement learning-to-optimize algorithms for sparse recovery, as demonstrated by Renpu Liu et al. (University of Virginia, UCLA) in “On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery”, reveals deeper computational capacities. Meanwhile, theoretical work on tabular ICL, such as that by Amir Rezaei Balef et al. (University of Tübingen, TU Dortmund University) in “Towards Understanding Layer Contributions in Tabular In-Context Learning Models”, seeks to demystify how these models function layer by layer.

The future of ICL promises more robust, interpretable, and adaptable AI systems that can seamlessly integrate into complex tasks, making AI more accessible and trustworthy across industries.