Multimodal Large Language Models: Navigating New Frontiers from Embodied AI to Medical Diagnostics

Latest 50 papers on multimodal large language models: Nov. 2, 2025

Multimodal Large Language Models (MLLMs) are rapidly evolving, pushing the boundaries of what AI can perceive, understand, and reason about. No longer confined to mere text, these models are increasingly adept at integrating information across diverse modalities—vision, speech, and even complex scientific data—to tackle real-world challenges. Recent research highlights a surge in innovation, addressing everything from enhancing spatial reasoning in robots to improving diagnostic capabilities in healthcare, and even deciphering ancient scripts. Let’s dive into some of the latest breakthroughs and what they mean for the future of AI.

The Big Idea(s) & Core Innovations

The overarching theme in recent MLLM research is the relentless pursuit of more nuanced, robust, and generalizable multimodal intelligence. A significant focus lies in improving spatio-temporal reasoning and contextual understanding. For instance, the paper “Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders” by Ali Rasekh et al. from Leibniz University Hannover proposes STAVEQ2 to overcome limitations in capturing complex temporal dynamics in videos, significantly boosting action recognition. Complementing this, “PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity” from Alibaba DAMO Academy introduces a unified framework for fine-grained object referring in both images and videos, crucial for precise visual comprehension.

Another critical area is enabling more interactive and embodied AI. “BLM₁: A Boundless Large Model for Cross-Space, Cross-Task, and Cross-Embodiment Learning” by Wentao Tan et al. from Tongji University presents the first unified multimodal spatial foundation model capable of operating across digital and physical spaces, a major step for robotics. Similarly, “RoboOmni: Proactive Robot Manipulation in Omni-modal Context” by Siyin Wang et al. from Fudan University introduces a framework that allows robots to infer user intent proactively from speech, environmental sounds, and visual cues, fostering more natural human-robot interaction. Further pushing this boundary, “PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments” from Beijing Jiaotong University enables MLLMs to reason in partially observable environments through active information acquisition and sequential physical actions.

Addressing biases and ensuring robustness is also paramount. “Unveiling Intrinsic Text Bias in Multimodal Large Language Models through Attention Key-Space Analysis” by Author One et al. from University of Example provides a novel framework to analyze and mitigate intrinsic text bias, crucial for ethical AI deployment. Meanwhile, “SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space” by Viktoriia Zinkovich et al. reveals vulnerabilities in reasoning segmentation models, driving the need for more robust designs. For efficiency, “SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs” by Jinhong Deng et al. from UESTC proposes a novel visual token pruning strategy that drastically reduces computational overhead without sacrificing semantic completeness, making MLLMs more deployable.

Beyond these core areas, MLLMs are also demonstrating specialized capabilities. “OracleAgent: A Multimodal Reasoning Agent for Oracle Bone Script Research” by Caoshuo Li et al. from Xiamen University is a pioneering AI agent for managing and retrieving ancient Oracle Bone Script information, showcasing MLLMs’ potential in digital humanities. In healthcare, “REMONI: An Autonomous System Integrating Wearables and Multimodal Large Language Models for Enhanced Remote Health Monitoring” by Y. Goldberg et al. integrates wearable sensors with MLLMs for more accurate and context-aware remote health monitoring. In scientific discovery, “Omni-Mol: Multitask Molecular Model for Any-to-any Modalities” from the National University of Singapore offers a generalist AI chemist capable of handling diverse molecular tasks, a significant leap for AI-driven chemistry.

Under the Hood: Models, Datasets, & Benchmarks

These advancements are underpinned by innovative model architectures, comprehensive datasets, and robust benchmarks that collectively push the envelope for MLLM capabilities:

• STAVEQ2 (“Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders”): A novel Video-LLM with stacked temporal attention modules within its vision encoder, achieving state-of-the-art results on SSv2 action recognition. Code available at: https://alirasekh.github.io/STAVEQ2/.
• WOD-E2E (“WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios”): A new dataset by Waymo LLC focused on rare, challenging long-tail driving scenarios for autonomous vehicles, accompanied by the Rater Feedback Score (RFS) metric for human-aligned evaluation.
• OracleAgent (“OracleAgent: A Multimodal Reasoning Agent for Oracle Bone Script Research”): An AI agent system integrating seven specialized tools and a multimodal knowledge base with over 1.4M single-character images and 80K interpretation texts for Oracle Bone Script research. Code available at: https://github.com/lcs0215/OralceAgent.
• Omni-Mol (“Omni-Mol: Multitask Molecular Model for Any-to-any Modalities”): A novel MLLM incorporating Gradient Adaptive LoRA (GAL) and Mixture-of-GAL-Experts (MoGE), trained on the largest molecular instruction-tuning dataset (1.4M samples). Code available at: https://github.com/Omni-Mol-Code.
• Open3D-VQA (“Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Space”): A benchmark by Tsinghua University with over 73K QA pairs across seven tasks for evaluating MLLMs’ 3D spatial reasoning in urban environments, supporting visual and point cloud modalities. Code available at: https://github.com/EmbodiedCity/Open3D-VQA.code.
• Latent Sketchpad (“Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs”): A framework enabling MLLMs to generate internal visual thoughts using a Sketch Decoder, enhancing interpretable multimodal reasoning. Code available for underlying models like Qwen2.5-VL-7B-Instruct: https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct.
• SCOPE (“SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs”): A visual token pruning strategy that jointly models saliency and coverage, achieving a 9x reduction in visual tokens while maintaining performance. Code available at: https://github.com/kinredon/SCOPE.
• MuSaG (“MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations”): The first human-annotated German multimodal sarcasm dataset with text, audio, and video annotations by Karlsruhe Institute of Technology. Available at: https://huggingface.co/datasets/sc0ttypee/MuSaG.
• BLM₁ (“BLM₁: A Boundless Large Model for Cross-Space, Cross-Task, and Cross-Embodiment Learning”): A unified multimodal spatial foundation model for cross-space, cross-task, and cross-embodiment learning, using a two-stage training paradigm for robust control. Project page: https://boundless-large-model.github.io.
• RoboOmni (“RoboOmni: Proactive Robot Manipulation in Omni-modal Context”): A Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs, trained on OmniAction, a large-scale multimodal dataset for proactive intent recognition. Code available at: https://github.com/OpenMOSS/RoboOmni.
• PixelRefer & PixelRefer-2.2M (“PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity”): A unified region-level MLLM for spatio-temporal object referring, introducing Scale-Adaptive Object Tokenizer (SAOT) and the PixelRefer-2.2M dataset. Code available at: https://github.com/alibaba-damo-academy/PixelRefer.
• EgoThinker & EgoRe-5M (“EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT”): A framework for egocentric reasoning with spatio-temporal Chain-of-Thought, using a two-stage SFT-RFT curriculum and the EgoRe-5M dataset with hand-object annotations. Code available at: https://github.com/InternRobotics/EgoThinker.
• Emotional Rationale Verifier (ERV) (“Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier”): A lightweight verifier to assess emotional alignment in MLLM explanations, introducing metrics like EEA, EPC, and FCR for emotional coherence.
• MMTutorBench (“MMTutorBench: The First Multimodal Benchmark for AI Math Tutoring”): The first multimodal benchmark by the University of Notre Dame for AI math tutoring, featuring 685 problems across Insight Discovery, Operation Formulation, and Operation Execution.
• Video-Thinker & Video-Thinker-10K (“Video-Thinker: Sparking”Thinking with Videos” via Reinforcement Learning”): A framework that enables MLLMs to perform video reasoning autonomously through grounding and captioning, trained on the Video-Thinker-10K dataset. Code available at: https://github.com/shijian2001/Video-Thinker.
• VideoTG-R1 (“VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotations”): A novel approach for video temporal grounding using curriculum reinforcement learning with reflected boundary annotations. Code available at: https://github.com/ldong1111/VideoTG-R1.
• Positional Preservation Embedding (PPE) (“Positional Preservation Embedding for Multimodal Large Language Models”): A novel encoding method by Huawei Noah’s Ark Lab that preserves spatiotemporal structure during visual token compression, significantly improving MLLM performance. Code available at: https://github.com/2U1/.
• Windsock & DANCE (“Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation”): A query-aware module for adaptive multimodal retrieval-augmented generation and Dynamic Noise-Resistance (DANCE), an adaptive training strategy. Preprint available at: https://arxiv.org/pdf/2510.22694.
• OFFSIDE (“OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models”): A benchmark by Harbin Institute of Technology for evaluating misinformation unlearning in MLLMs, highlighting challenges like catastrophic forgetting and prompt attacks. Code available at: https://github.com/zh121800/OFFSIDE.
• VisJudge-Bench & VISJUDGE (“VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations”): The first comprehensive benchmark by The Hong Kong University of Science and Technology (Guangzhou) for assessing MLLMs’ visualization aesthetics and quality, alongside VISJUDGE, an optimized model. Preprint available at: https://arxiv.org/pdf/2510.22373.
• VPSG (Vision-PE Shuffle Guidance) (“Mitigating Coordinate Prediction Bias from Positional Encoding Failures”): A training-free, test-time method by The Hong Kong University of Science and Technology (Guangzhou) to correct directional biases in coordinate prediction caused by positional encoding failures. Code available at: https://github.com/taoxj2001/VPSG.
• LightAgent (“LightAgent: Mobile Agentic Foundation Models”): A mobile agentic foundation model by the University of Hong Kong that combines on-device and cloud collaboration for efficient GUI task execution. Code available at: https://github.com/HKUDS/LightAgent.
• DeepOmni (“DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE”): An MoE-based framework by Tencent and Fudan University that addresses catastrophic forgetting in native MLLMs for seamless speech interaction. Preprint available at: https://arxiv.org/pdf/2506.21864.
• CMIE (“CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection”): A framework by Yunnan University that leverages MLLMs and external evidence for explainable out-of-context misinformation detection. Code available at: https://github.com/fanxiao15/CMIE.
• DynamicVL Suite (DVL-Bench, DVL-Instruct, DVLChat) (“DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understanding”): A comprehensive framework by The University of Tokyo for urban dynamics analysis using remote sensing imagery, including a benchmark, an instruction-tuning dataset, and a baseline model. Code available at: https://github.com/weihao1115/dynamicvl.
• KBE-DME (“KBE-DME: Dynamic Multimodal Evaluation via Knowledge Enhanced Benchmark Evolution”): A dynamic evaluation framework by Peking University that evolves static benchmarks using knowledge-enhanced methods to address data contamination and saturation. Preprint available at: https://arxiv.org/abs/2510.21182.
• VOGUE (“VOGUE: A Multimodal Dataset for Conversational Recommendation in Fashion”): A novel dataset by the University of Toronto for multimodal conversational recommendation in fashion, featuring human-human dialogues, visual grounding, and user feedback. Preprint available at: https://arxiv.org/pdf/2510.21151.
• NoisyGRPO (“NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation”): A reinforcement learning framework by ShanghaiTech University that improves multimodal Chain-of-Thought reasoning by injecting noise into visual inputs and using Bayesian estimation. Project page: https://artanic30.github.io/project_pages/NoisyGRPO.
• PhysVLM-AVR & CLEVR-AVR, AVR-152k (“PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments”): Introduces the Active Visual Reasoning (AVR) task, the CLEVR-AVR benchmark, and the AVR-152k dataset for training agents in partially observable environments. Project page: https://anonymous-je99tt.open.science/.
• Sketch2BIM (“Sketch2BIM: A Multi-Agent Human-AI Collaborative Pipeline to Convert Hand-Drawn Floor Plans to 3D BIM”): A human-in-the-loop pipeline by the University of Texas at Arlington that converts hand-drawn floor plans to 3D BIM models using MLLMs within a multi-agent framework. Code available at: https://github.com/UTA-CivilEng/Sketch2BIM.
• InfiniPot-V (“InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding”): A training-free, query-agnostic framework by Hanyang University for memory-constrained streaming video understanding, introducing Temporal-axis Redundancy (TaR) and Value-Norm (VaN) metrics for KV cache compression. Preprint available at: https://arxiv.org/pdf/2506.15745.
• MELLM, MEFlowNet, & MEU-Instruct (“MELLM: Exploring LLM-Powered Micro-Expression Understanding Enhanced by Subtle Motion Perception”): A framework by the University of Science and Technology of China that combines optical flow-based motion perception (MEFlowNet) with LLM reasoning, using the MEU-Instruct dataset for micro-expression understanding. Code available at: https://github.com/zyzhangUstc/MELLM.
• RTV-Bench (“RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video”): A benchmark by Harbin Institute of Technology for evaluating MLLMs in real-time video analysis with Multi-Timestamp QA and Hierarchical Question Structures. Project page: https://ljungang.github.io/RTV-Bench.
• VITA-1.5 (“VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction”): A multimodal LLM by Nanjing University enabling real-time vision and speech interaction through a three-stage training methodology, achieving end-to-end speech output without ASR/TTS. Code available at: https://github.com/VITA-MLLM/VITA.
• ARGenSeg (“ARGenSeg: Image Segmentation with Autoregressive Image Generation Model”): A framework by Ant Group that integrates image segmentation into MLLMs using an autoregressive image generation paradigm for fine-grained pixel-level understanding. Preprint available at: https://arxiv.org/pdf/2510.20803.
• Conan & Conan-91k (“Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence”): A framework by Peking University for multi-step video reasoning that integrates visual evidence with logical deduction, introducing the Conan-91k dataset. Code available at: https://github.com/OuyangKun10/Conan.
• BIOCAP (“BIOCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models”): A biological foundation model by The Ohio State University that leverages synthetic captions from Wikipedia-derived visual information for species classification and text-image retrieval. Code available at: https://github.com/Imageomics/biocap.

Impact & The Road Ahead

The collective impact of this research is profound, painting a picture of MLLMs becoming increasingly capable, trustworthy, and adaptable. These advancements promise more natural human-AI interactions, highly autonomous systems, and specialized AI assistants across various domains. The pursuit of enhanced spatial and temporal reasoning is critical for applications like autonomous driving, robotics, and video analysis, paving the way for AI that understands the world with human-like intuition.

Addressing biases and improving model robustness, as seen in the text bias analysis and adversarial paraphrasing research, is fundamental for deploying AI ethically and securely. Efficient MLLM architectures, through token pruning and memory-constrained processing, are making these powerful models accessible on edge devices, unlocking new possibilities for ubiquitous AI.

Looking forward, several exciting avenues emerge. The integration of active reasoning and human-in-the-loop systems, exemplified by “Sketch2BIM: A Multi-Agent Human-AI Collaborative Pipeline to Convert Hand-Drawn Floor Plans to 3D BIM” from the University of Texas at Arlington, suggests a future where AI and humans collaborate seamlessly. Further research into multi-modal safety, as highlighted by “Beyond Text: Multimodal Jailbreaking of Vision-Language and Audio Models through Perceptually Simple Transformations”, will be crucial as MLLMs become more powerful. The development of dynamic and knowledge-enhanced benchmarks, such as “KBE-DME: Dynamic Multimodal Evaluation via Knowledge Enhanced Benchmark Evolution”, will ensure that evaluations keep pace with rapid innovation, preventing models from simply overfitting to static datasets.

In essence, the field of multimodal large language models is not just expanding; it’s diversifying and specializing, preparing for a future where AI can perceive, understand, and interact with our complex, multimodal world with unprecedented intelligence and versatility. The journey towards truly generalist and reliable multimodal AI is well underway, promising transformative applications across every sector.