Few-Shot Learning: Navigating the Data Desert with LLMs, Vision, and Robotics

Latest 50 papers on few-shot learning: Oct. 12, 2025

Few-Shot Learning: Navigating the Data Desert with LLMs, Vision, and Robotics

In the rapidly evolving landscape of AI/ML, the ability of models to learn from minimal data – a paradigm known as few-shot learning (FSL) – is no longer a luxury, but a necessity. As we push the boundaries of AI, from deploying intelligent agents in complex robotic systems to diagnosing diseases with limited medical images, FSL offers a compelling solution to the perennial challenge of data scarcity. Recent research showcases a vibrant ecosystem of innovation, where large language models (LLMs), sophisticated vision systems, and robotics are converging to unlock unprecedented capabilities.

The Big Idea(s) & Core Innovations

At the heart of these breakthroughs lies the fundamental challenge of generalizing from a handful of examples. Many papers are tackling this by rethinking how models acquire, process, and apply contextual information. For instance, the paper, “Understanding In-context Learning of Addition via Activation Subspaces” by Xinyan Hu and colleagues from UC Berkeley and INRIA, reveals that LLMs localize task-specific information in in-context learning (ICL) to a few attention heads and low-dimensional subspaces. They highlight self-correction mechanisms and a fascinating encoding of arithmetic using trigonometric functions, suggesting deep internal structures that can be exploited for more efficient learning.

Further refining how LLMs leverage context, “Submodular Context Partitioning and Compression for In-Context Learning-short paper” by Shaoyi Zheng et al. from New York University and other institutions, introduces Sub-CP. This framework uses submodular optimization to control diversity and semantic structure within context blocks, significantly improving ICL performance across diverse setups. This idea of intelligent context construction is echoed by “GRAD: Generative Retrieval-Aligned Demonstration Sampler for Efficient Few-Shot Reasoning” by Oussama Gabouj and his team from EPFL. GRAD dynamically generates task-specific, concise demonstrations under strict token budgets, outperforming traditional RAG methods in out-of-distribution tasks and demonstrating that smaller models can effectively guide larger ones.

The notion of dynamic, adaptive context generation extends to enhancing robustness. “RAGs to Riches: RAG-like Few-shot Learning for Large Language Model Role-playing” by Timothy Rupprechta et al. from Northeastern University, proposes a novel framework that uses curated reference demonstrations to boost authenticity and consistency in LLM role-playing, making models more resilient to ‘jail-breaking’ attempts. Meanwhile, “Mechanism of Task-oriented Information Removal in In-context Learning” by Hakaze Cho et al. from JAIST, presents a compelling theory that ICL isn’t about learning new tasks, but rather removing irrelevant information through “Denoising Heads,” a crucial insight for making LLMs more precise.

Bridging modalities, “VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning” by Wenhao Li et al. from Shandong University, introduces VT-FSL. This framework integrates LLMs to generate cross-modal prompts, combining class names and support images to produce semantically consistent descriptions and synthetic images, achieving state-of-the-art results through geometry-aware alignment. For specialized domains, “Crossing Domains without Labels: Distant Supervision for Term Extraction” by Elena Senger et al. from LMU Munich, introduces DiSTER, a framework that leverages distant supervision and LLMs to generalize Automatic Term Extraction (ATE) across various domains using pseudo-labels from black-box models. Similarly, “Semantic-Aware Fuzzing: An Empirical Framework for LLM-Guided, Reasoning-Driven Input Mutation” by Meng Lu et al. from Queen’s University, integrates LLMs into fuzzing to generate meaningful mutations without fine-tuning, enhancing code coverage and bug discovery.

In the realm of vision and robotics, few-shot learning is proving transformative. “ANROT-HELANet: Adverserially and Naturally Robust Attention-Based Aggregation Network via The Hellinger Distance for Few-Shot Classification” by Gao Yu Lee et al. from Nanyang Technological University, leverages Hellinger distance to enhance robustness against adversarial and natural noise. For medical applications, “MetaChest: Generalized few-shot learning of patologies from chest X-rays” and “Expert-Guided Explainable Few-Shot Learning for Medical Image Diagnosis” are significant. The latter, by Uddin et al., from MICCAI Workshop, integrates radiologist annotations into few-shot learning to improve both accuracy and interpretability in low-data medical imaging. In robotics, “O³Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulation” by Zhiyuan Li et al. from MIT, uses a one-shot framework to enable robots to infer object-to-object affordances in 3D, integrating LLMs for enhanced spatial understanding.

Under the Hood: Models, Datasets, & Benchmarks

These advancements are often catalyzed by novel models, carefully curated datasets, and robust benchmarking. Here are some key resources and their significance:

• Models & Frameworks:
  • Sub-CP (https://github.com/deepseek-ai/): A block-aware context selection framework using submodular optimization for efficient ICL.
  • GRAD (https://github.com/charafkamel/GRAD-demonstration-sampler): An RL-trained generative model for task-specific, token-constrained demonstrations.
  • DiSTER (https://huggingface.co/ElenaSenger/DiSTER-Llama-3-8B-Instruct): A scalable ATE framework combining synthetic data generation and LLM fine-tuning.
  • VT-FSL (https://github.com/peacelwh/VT-FSL): Integrates LLMs to generate cross-modal prompts for enhanced FSL performance.
  • CLIP-SVD (https://github.com/HealthX-Lab/CLIP-SVD): A parameter-efficient adaptation technique for vision-language models using SVD.
  • ANROT-HELANet (https://github.com/GreedYLearner1146/ANROT-HELANet/tree/main): Leverages Hellinger distance and attention for robust few-shot classification.
  • DAC-FCF (https://github.com/sunshengke/DAC-FCF): Combines data augmentation, contrastive learning, and Fourier convolution for bearing fault diagnosis under limited data.
  • MOMEMTO (https://arxiv.org/pdf/2509.18751): A time series foundation model with a patch-based memory gate for anomaly detection.
  • Attn-Adapter (https://arxiv.org/pdf/2509.03895): An online few-shot learner enhancing CLIP features through dual attention mechanisms.
  • MLSD (https://github.com/parushgera/mlsd-few-shot): A metric learning approach for cross-target and cross-domain stance detection.
  • BATR-FST (https://github.com/yourusername/BATR-FST): A framework for bi-level adaptive token refinement in few-shot transformers.
  • StepSPT (https://github.com/xuhuali-mxj/StepSPT): A source-free CDFSL approach using style prompt tuning and step-wise distribution alignment.
  • ARFM (https://github.com/huggingface/lerobot): An adaptive offline reinforcement learning method for fine-tuning VLA flow models.
• Datasets & Benchmarks:
  • SynTerm dataset (https://huggingface.co/datasets/ElenaSenger/SynTerm): For cross-domain generalization of term extraction tasks.
  • MetaChest dataset (https://arxiv.org/pdf/2509.25590): A large-scale dataset for chest X-ray pathology classification with 479,215 images.
  • U-DIADS-TL dataset (for FEST competition, https://ai4ch.uniud.it/FESTcompICDAR25/): Diverse, high-quality dataset for few-shot text line segmentation in historical documents.
  • RRDataset (https://zenodo.org/records/14963880): A comprehensive benchmark for evaluating AI-generated image detection under real-world conditions.
  • MOLE dataset (https://huggingface.co/datasets/IVUL-KAUST/MOLE): A new benchmark dataset for metadata extraction from scientific papers.
  • MultiClinSUM Shared Task (https://temu.bsc.es/multiclinsum/): A dataset for clinical document summarization, used in “MaLei at MultiClinSUM: Summarisation of Clinical Documents using Perspective-Aware Iterative Self-Prompting with LLMs”.
  • MimicDroid simulation benchmark (https://ut-austin-rpl.github.io/MimicDroid): 8 hours of human play data for evaluating few-shot learning for humanoids.
  • RewardBench (https://github.com/Qwen-Labs/RewardBench): Benchmark tasks (Chat, Chat Hard, Safety, Reasoning) used in “Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness”.
  • MetaAudio benchmark (used in “Prototypical Contrastive Learning For Improved Few-Shot Audio Classification”): For few-shot audio classification.
  • ICRL project website (https://github.com/ztlmememe/LLMxFM_ICRL): Code for In-Context Representation Learning, allowing LLMs to integrate representations from foundation models for multi-modal inference.

Impact & The Road Ahead

The collective impact of this research is profound, touching upon nearly every corner of AI application. From personalized education systems that adapt to student histories, as shown by “Personalized Auto-Grading and Feedback System for Constructive Geometry Tasks Using Large Language Models on an Online Math Platform” from Hongik University, to critical industrial applications like intelligent reservoir management explored in “Intelligent Reservoir Decision Support: An Integrated Framework Combining Large Language Models, Advanced Prompt Engineering, and Multimodal Data Fusion for Real-Time Petroleum Operations”, few-shot learning is enabling robust, data-efficient solutions. We’re seeing a shift from models that demand vast quantities of labeled data to agile systems that can quickly adapt with minimal examples.

Several papers, including “From Physics to Machine Learning and Back: Part II – Learning and Observational Bias in PHM” by Olga Fink et al. from EPFL, highlight the integration of domain knowledge and physical consistency, leading to more generalizable and trustworthy models. This is crucial for high-stakes areas like Prognostics and Health Management (PHM). The identification of “channel bias” and “feature redundancy” in pre-trained models by Ji Zhang et al. in “From Channel Bias to Feature Redundancy: Uncovering the ”Less is More” Principle in Few-Shot Learning” (UESTC) challenges conventional wisdom, suggesting that sometimes, less is more when it comes to feature utilization in FSL. This signals a future where models are not just powerful, but also elegantly parsimonious.

Looking ahead, the emphasis will be on increasing robustness, interpretability, and ethical deployment across diverse domains. Research like “Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches” underscores the critical need to address algorithmic bias as few-shot methods become more pervasive in sensitive applications. As models grow increasingly capable, frameworks like “MetaMiDA” for scalable meta-learning and “Empowering Time Series Analysis with Foundation Models: A Comprehensive Survey” will pave the way for more robust and universally applicable foundation models. The exciting journey toward truly intelligent and adaptive AI, capable of learning efficiently from the world around it, is clearly well underway.