From Foundation Models to Agile Agents: Bridging Perception, Prediction, and Purpose-Driven Action
Latest 100 papers on foundation models: Nov. 22, 2026
The landscape of AI/ML is being rapidly reshaped by Foundation Models (FMs), colossal neural networks pre-trained on vast datasets that exhibit remarkable generalization capabilities. Yet, transforming these powerful models into truly agile, reliable, and purpose-driven agents capable of interacting with complex, unpredictable real-world environments remains a significant challenge. Recent research has focused on distilling, adapting, and orchestrating these FMs to move beyond static prediction towards dynamic, context-aware decision-making. This blog post dives into some of the latest breakthroughs, highlighting how FMs are being enhanced to tackle everything from robotic manipulation and medical diagnosis to urban planning and scientific discovery.
The Big Idea(s) & Core Innovations
The central theme across these papers is the strategic adaptation and orchestration of Foundation Models to address specific, often dynamic, real-world problems. Researchers are moving beyond simply applying FMs off-the-shelf, instead developing frameworks that enable them to reason, adapt, and even “self-improve” within complex scenarios.
One significant area of innovation is making FMs responsive and adaptable to changing conditions or individual instances. For example, PhysDial: Adapting Pretrained Video Models for PDE Simulation under Changing Physical Conditions from The University of Alabama demonstrates that models trained on fixed physical conditions often ignore explicit condition inputs. By introducing “switch trajectories” (where conditions change mid-rollout), they halve prediction error post-change, showing the importance of interventional training data for true condition-responsiveness. Similarly, SWIFT: Sample-Wise Instance Fine-Tuning of Time-Series Foundation Models for Forecasting by Nanjing University and Siemens Data and AI Research enables a Time-Series FM to adapt to each individual forecasting window using only its historical data, reducing MASE and WQL by over 4%. This “learning from self” approach highlights a shift from global adaptation to hyper-localized, on-the-fly refinement. In a similar vein, PACT: Recurrent Domain Incremental Adaptation of Time-Series Foundation Models from Nanyang Technological University focuses on continuous adaptation to recurrent domain shifts by learning local, chart-specific residual transports, achieving strong adaptation-retention trade-offs.
Another crucial innovation is injecting domain-specific priors and structures to enhance FM performance and interpretability. In robotics, GraSLA: A Lightweight Spatial-Language-Action Model with Grasp Hypotheses as Geometric Priors by National University of Defense Technology integrates 6-DoF grasp hypotheses as geometric priors into a continuous-control VLA model, achieving state-of-the-art success rates on manipulation benchmarks. The “soft prior” approach allows the policy to leverage geometric knowledge without being rigidly constrained. For medical imaging, Improving Zero-shot 3D Semantic Mapping in Offroad with Geometric Priors by Aalto University uses class-specific elevation-sphericity histograms from LiDAR data to correct vision FM errors in offroad 3D semantic mapping, notably improving SAM3’s performance. The geometric priors serve as a lightweight, training-free way to ground visual FMs in physical reality. In biomedicine, Glyph: An Interpretability Engine for AI-Driven Scientific Discovery in Clinical Data by Stanford University and Mayo Clinic, decomposes complex multimodal FM embeddings into sparse, clinically interpretable concepts using sparse autoencoders, enabling reproducible prognostic marker discovery that goes beyond existing literature-derived markers. This work exemplifies how FMs can be made more transparent and useful for scientific discovery.
Reliability, safety, and efficiency under real-world constraints are also paramount. LEASH: Stay Safe in Agentic Robotics from Xi’an Jiaotong-Liverpool University introduces a model-agnostic safety harness for FM-controlled robots, using visual-context-grounded task, action, and commit gates to reduce harm rates while preserving task completion. This addresses the critical need for guardrails in embodied AI. In a similar vein, ERA: Error-Resistant Autonomous Object Navigation with Predicted Depth by Shanghai Jiao Tong University tackles the inherent unreliability of predicted depth in robot navigation. Its “Move with Error” paradigm allows robots to revise their occupancy maps as they move, coupling revisable perception with active exploration to dramatically reduce collisions. The theoretical foundation is strengthened by Prediction with Foundation Models under Deployment Budgets from the National University of Singapore, which introduces ACES-T and ACES-I, frameworks for combining multiple FM predictions under query-cost budgets. This work provides minimax theory for cost-aware prediction, crucial for practical, resource-constrained deployments.
Finally, the integration of multimodality and complex reasoning is pushing FMs into new domains. TimeOmni-o1: Bidirectionally Coupling Time Series Foundation Models and LLMs for Multimodal Reasoning and Forecasting from Tsinghua University and Shanghai Artificial Intelligence Laboratory showcases a novel approach where a Time-Series FM compresses representations for an LLM’s reasoning, and that reasoning is fed back to the TSFM for conditioned forecasting. This bidirectional coupling enables state-of-the-art multimodal understanding and forecasting. Darmok: Championship Cryptic Crossword Solving via Constraint Tightening and Resilient Orchestration by The University of Texas at Austin exemplifies multi-model reasoning and resilient orchestration. This system uses multiple LLMs with synchronous constraint tightening and fallback stages to solve complex cryptic crosswords, highlighting that redundancy and robustness are key for real-world agentic performance, especially under API degradation.
Under the Hood: Models, Datasets, & Benchmarks
These advancements are driven by innovative architectural designs, specialized datasets, and rigorous benchmarking strategies:
- Architectures: Many papers leverage established foundation models as backbones, then introduce lightweight, trainable modules for adaptation. Examples include:
- DINOv3: Heavily utilized across vision tasks, like in Improving Zero-shot 3D Semantic Mapping in Offroad with Geometric Priors and Medical Image Segmentation: Foundation Model Adaptation versus Task-Specific Training.
- TabPFN V3: A key component in tabular data applications, as seen in MediPFN: Amortized In-Context Nuisance Learning and Inference for Causal Mediation Analysis and Causal graph fusion across incomplete cohorts and causal steering of tabular foundation models.
- Wan2.1 (video diffusion transformer): Adapted in PhysDial: Adapting Pretrained Video Models for PDE Simulation under Changing Physical Conditions for PDE simulation and in PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Imagined Subgoals and Policy Priors for robotic planning.
- FLUX 3 Image Klein: Adapted with LoRA in GenRF: Generative Registration-free Image Fusion for generative image fusion.
- Mamba-3: Used in Time Flies When You Parallelise: Pareto-efficient Time Series Foundation Models with Linear Recurrences for efficient time-series forecasting.
- TiRex: A Time-Series FM explored in The Unused Input: The Initial State as a Conditioning Interface for Recurrent TSFMs for parameter-efficient conditioning.
- Qwen3-VL: A vision-language backbone used in agentic embodied intelligence like Toward the Era of Agentic Embodied Intelligence.
- Bonbon: A novel 1.6B-parameter transformer FM specifically for molecular interactions, demonstrating emergent biological capabilities in Emergent Biological Capabilities in a Foundation Model for Molecular Interactions.
- Lightweight adapters (LoRA, BitFit): Frequently employed for parameter-efficient fine-tuning, as exemplified in SARVIA: Fast-Adapting the Strong SAR Foundation Models That Are Already in Your MLLMs and Medical Image Segmentation: Foundation Model Adaptation versus Task-Specific Training.
- Datasets & Benchmarks: The papers introduce or heavily utilize specialized benchmarks for rigorous evaluation:
- EgoPass-Bench: A new benchmark introduced by University of California, Santa Barbara in EgoPass-Bench: Identifying Objects That May Affect the Wearer’s Passage in Egocentric Video for evaluating multimodal models’ understanding of safe passage in egocentric video.EHR-WM: Utilizes MIMIC-IV, eICU-CRD, and NWICU datasets for longitudinal EHR forecasting in Forecasting Patient Trajectories from the Electronic Health Records with Action-Conditioned World Models.
- RoboRisk & LEASH project website: Introduced in LEASH: Stay Safe in Agentic Robotics for evaluating semantic and physical safety in agentic robotics.
- BioXArena-Pro: A research-grade benchmark for autonomous biomedical ML agents with 64 tasks across eight domains, detailed in BioXArena-Pro: a research-grade benchmark for autonomous biomedical machine learning agents.
- INTENT: A seven-cell benchmark for next-item prediction on user event sequences, auditing 59 public dataset families, proposed by KTH AI Society in INTENT: Out-of-Sample, Out-of-Time, and Cold-Start Evaluation on User Event Sequences.
- MMTR: A 573,653-sample multimodal reasoning dataset for time series, introduced in TimeOmni-o1: Bidirectionally Coupling Time Series Foundation Models and LLMs for Multimodal Reasoning and Forecasting.
- OTBENCH: A unified benchmark for tabular generation, imputation, and prediction across 56 datasets and 20 semantic capability axes, introduced in One Table, Many Queries: Benchmarking Generalist Tabular Models Across Generation, Imputation, and Prediction.
- WoW: An Educational Knowledge Graph-driven Chinese-English corpus for LLM training, spanning 9 domains and 76,364 KnowledgePoints, detailed in The WoW: A Large-Scale World Knowledge Corpus for Full-Lifecycle LLM Training.
- SurgCBVR: The first public, reproducible, multi-domain benchmark for content-based surgical video-to-video retrieval, unifying eight public surgical datasets, presented in SurgCBVR: A Multi-Domain Benchmark for Content-Based Surgical Video-to-Video Retrieval.
- ABHBench: A multimodal benchmark for evaluating foundation models as first-person moral agents in human-android dilemmas, proposed by Shanghai Jiao Tong University in ABHBench: Evaluating Moral Decision-Making of Foundation Models from an Agentic Perspective.
- TabBioMed/TabBench-Bio: Comprehensive biomedical tabular learning benchmarks (TabBioMed: A Large-Scale Benchmark for Biomedical Tabular Learning, TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tables) for evaluating TFMs across diverse biomedical domains and data regimes.
- SARVIA Project dataset and code repository: For adapting visual encoders in MLLMs to Synthetic Aperture Radar (SAR) perception, available at https://github.com/zcablii/SARVIA.
Impact & The Road Ahead
The collective impact of this research is profound, pushing Foundation Models from mere prediction engines to proactive, adaptive, and responsible agents. The advancements hold immense promise for:
- Robotics: Enabling robots to operate safely and effectively in unstructured environments, perform long-horizon manipulation, and adapt to unforeseen circumstances with robust error handling.
- Healthcare: Revolutionizing patient trajectory forecasting, refining medical image diagnosis, accelerating drug discovery with emergent biological capabilities, and enabling privacy-preserving multi-site collaboration in neuroimaging.
- Scientific Discovery: Providing AI weather models with deeper physical consistency, enabling astrophysicists to discover stellar parameters more easily from multi-survey spectra, and enhancing marine intelligence through geospatial FMs for critical environmental monitoring.
- Business & Finance: Offering more accurate and cost-efficient retail sales forecasting, robust financial predictions under deployment budgets, and sophisticated analysis of market shocks through self-supervised learning.
The road ahead is rich with challenges and opportunities. Future work will likely focus on strengthening multimodal grounding, improving reasoning capabilities under uncertainty, and developing more robust and interpretable adaptation strategies. For instance, Multimodal Graph Learning: A Survey of Models, Benchmarks, and Frontiers emphasizes the need for graph-aware cross-modal alignment for AI agents. The theoretical underpinnings are also rapidly evolving, with works like Deep Learning Theory through the Lens of Statistical Physics offering a unified framework to understand generalization and learning dynamics, guiding the design of more effective pipelines. We’re witnessing an exciting transition: FMs are no longer just powerful black boxes, but increasingly flexible, coachable, and accountable partners, ready to tackle the most complex challenges across science and industry. The journey towards truly intelligent and embodied AI agents is well underway, and these papers mark crucial steps on that path.
Share this content:
Discover more from SciPapermill
Subscribe to get the latest posts sent to your email.
Post Comment