In-Context Learning: Revolutionizing AI with Adaptability and Efficiency

Latest 50 papers on in-context learning: Nov. 2, 2025

In-context learning (ICL) has rapidly emerged as a cornerstone of modern AI, allowing large models to adapt to new tasks with minimal or no explicit fine-tuning. This paradigm shift, where models learn from examples provided directly in the input prompt, is transforming how we approach everything from complex reasoning to robotic control. Recent research showcases not only remarkable advancements in ICL’s capabilities but also a deeper theoretical understanding of its underlying mechanisms and limitations.### The Big Idea(s) & Core Innovationsoverarching theme in recent ICL research is about enhancing adaptability, robustness, and efficiency across diverse domains. A significant thrust is to move beyond mere pattern matching towards truly intelligent, adaptable systems. For instance, Peking University’s and Tencent’s work on Think Outside the Policy: In-Context Steered Policy Optimization introduces ICPO, a novel reinforcement learning framework for Large Reasoning Models (LRMs). This framework leverages few-shot ICL rollouts to provide high-quality expert signals, improving exploration and training stability without relying on external expert models—a critical step towards more scalable LRM optimization. Similarly, the Empowering RepoQA-Agent based on Reinforcement Learning Driven by Monte-carlo Tree Search paper introduces RepoSearch-R1, combining Monte Carlo Tree Search (MCTS) with Group Relative Policy Optimization (GRPO) for repository-level reasoning. This self-training approach eliminates the need for external model distillation, demonstrating remarkable efficiency gains.is also proving transformative in specialized applications. For medical imaging, the University of Alberta’s FlexICL: A Flexible Visual In-context Learning Framework for Elbow and Wrist Ultrasound Segmentation dramatically improves segmentation accuracy with minimal labeled data, a boon for resource-limited settings. In robotics, OmniVIC, presented in OmniVIC: A Self-Improving Variable Impedance Controller with Vision-Language In-Context Learning for Safe Robotic Manipulation, integrates vision-language ICL with variable impedance control to significantly boost task success rates by incorporating language-based reasoning into robotic decision-making. Researchers from Keio University and NVIDIA, in their paper Towards Predicting Any Human Trajectory In Context, introduce TrajICL, an ICL framework for pedestrian trajectory prediction that adapts to new scenarios without fine-tuning, leveraging spatio-temporal similarity for enhanced generalization.the fundamental mechanics of ICL is another crucial area. The paper How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPs from Koç University reveals how nonlinear MLPs in Transformers, combined with structured data mixing, enhance ICL performance. Princeton University’s research in Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers provides a scalable method to interpret the functional roles of attention heads, showing how LLMs utilize sparse sub-circuits for task completion. Furthermore, the work from Nanjing University and University of Technology Sydney, Mixture-of-Experts Meets In-Context Reinforcement Learning, introduces T2MIR, integrating Mixture-of-Experts (MoE) into ICRL to handle multi-modal inputs and task diversity more efficiently.### Under the Hood: Models, Datasets, & Benchmarksresearch is pushing the boundaries of models, datasets, and benchmarks to fully realize ICL’s potential:ARC-Encoder: Introduced in ARC-Encoder: learning compressed text representations for large language models by Kyutai, this method compresses text inputs into continuous representations, reducing input sequence length for decoder LLMs without modifying the model. The associated code is available at https://github.com/kyutai-labs/ARC-Encoder.Memory Mosaics v2: Developed by New York University and FAIR, Meta Inc., and detailed in Memory Mosaics at scale, these architectural modifications scale associative memory networks to Llama-8B size, outperforming traditional Transformers in new-task learning. Code is available at https://github.com/facebookresearch/MemoryMosaics.LoCoMo Benchmark: From the University of Edinburgh, this synthetic benchmark (as per Evaluating Long-Term Memory for Long-Context Question Answering) evaluates memory-augmented methods in long-context dialogues, helping develop LLMs with better long-term memory. Code can be found at https://github.com/.MIR-Bench: Proposed by ByteDance Seed and University of Illinois Urbana-Champaign in MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?, this is the first many-shot ICL benchmark for complex pattern recognition, available at https://github.com/KaiYan289/MIR-Bench.MultiVerse Benchmark: KAIST and other institutions introduce MultiVerse in MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models, a multi-turn conversation benchmark for VLMs with checklist-based evaluation. The project page is https://passing2961.github.io/multiverse-project-page/.SIG and SIGBench: Johns Hopkins University’s work in Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study introduces Spatial Intelligence Grid (SIG) for Visual-Spatial Intelligence (VSI) and SIGBench, a benchmark with 1.4K driving frames annotated with human gaze traces, available at https://guanlinwu123.github.io/sigbench.MITRA: From Amazon and CMU, this tabular foundation model, detailed in Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models, uses a mixture of synthetic priors for state-of-the-art performance in tabular tasks. Hugging Face models are available for classification and regression.### Impact & The Road Aheadadvancements highlight a powerful future for AI, where models are not only intelligent but also highly adaptive, efficient, and transparent. The ability to perform complex tasks with few-shot or even one-shot examples drastically reduces the need for extensive labeled datasets and costly fine-tuning, making AI more accessible and scalable. This is evident in medical imaging, robotic manipulation, and even code intelligence with systems like R2ComSync from Shandong University, Zhejiang University, and City University of Hong Kong in R2ComSync: Improving Code-Comment Synchronization with In-Context Learning and Reranking., challenges remain. The “many-shot paradox” identified in When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation from Intellica Business Intelligence and Yildiz Technical University shows that simply adding more examples isn’t always beneficial, underscoring the need for smarter context construction. Furthermore, the paper Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context from Northwestern University points to inherent efficiency limitations in ICL with longer contexts. Addressing these limitations will require deeper theoretical understanding, as explored in papers like Provable test-time adaptivity and distributional robustness of in-context learning by the University of Cambridge and London School of Economics and Political Science, which offers strong optimality guarantees for pretrained Transformers.future of ICL promises more robust and human-aligned AI. Researchers are now exploring metacognitive abilities, as seen in Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations by UC San Diego, which reveals LLMs’ capacity to monitor internal activations—critical for AI safety. The integration of biologically inspired plasticity mechanisms, as demonstrated by St Paul’s School in Enabling Robust In-Context Memory and Rapid Task Adaptation in Transformers with Hebbian and Gradient-Based Plasticity, further pushes the boundaries of adaptable AI. As we continue to refine our understanding and methods, in-context learning is set to unlock unprecedented levels of AI performance and utility, making AI systems more versatile, efficient, and ultimately, more intelligent.