Multimodal Large Language Models: A Leap Towards Human-like Perception and Reasoning

Latest 50 papers on multimodal large language models: Sep. 8, 2025

Multimodal Large Language Models (MLLMs) are rapidly evolving, pushing the boundaries of what AI can perceive, understand, and reason across diverse data types. From interpreting complex visual scenes to understanding nuanced emotional cues, these models promise a future where AI interacts with the world with unprecedented sophistication. Recent research showcases remarkable advancements and tackles critical challenges, paving the way for more robust, reliable, and intelligent AI systems.### The Big Idea(s) & Core Innovationscutting edge of MLLM research is characterized by a drive to imbue these models with more human-like perceptual and reasoning abilities, while simultaneously addressing inherent limitations like hallucination and computational efficiency. A significant theme across recent papers is the pursuit of deeper visual understanding. For instance, the “Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Model” paper by She Yifei and Huangxuan Wu from Beijing University of Posts and Telecommunications directly confronts the bottleneck of single vision encoders in MLLMs. They introduce FtZ, a novel framework that dynamically fuses features from multiple expert vision encoders, significantly boosting performance on fine-grained visual tasks. This idea of leveraging multiple specialized visual inputs resonates with “One More Glance with Sharp Eyes: Rethinking Lightweight Captioning as a Practical Visual Specialist” by Junha Song et al. from KAIST, which proposes a “Sharp-Eyed Refinement” mechanism. This lightweight captioning framework mimics human attention by performing a focused second glance, achieving performance comparable to large generalist models with far fewer parameters.perception, improving MLLMs’ reasoning capabilities is paramount. The “Tailored Teaching with Balanced Difficulty: Elevating Reasoning in Multimodal Chain-of-Thought via Prompt Curriculum” paper by Xinglong Yang et al. from Nanjing University of Aeronautics and Astronautics, addresses the instability of Chain-of-Thought prompting. They introduce CAMS, a framework that selects prompt examples based on a balance of easy and hard examples, leading to more stable and high-performing reasoning. This is complemented by “R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning” by Yann Qi from Tencent Hunyuan Team, which develops an auto-thinking MLLM that adaptively switches between “thinking” and “direct answering” modes. This bi-mode annealing, combined with a novel reinforcement learning algorithm, significantly enhances efficiency and reasoning across diverse benchmarks.and reliability are also major concerns. In “Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs”, Guangzong Si et al. from USTC challenge the conventional understanding of object hallucination, identifying two distinct causes (omission and fabrication) and proposing Visual Potential Field Calibration (VPFC) to mitigate them. Similarly, “Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization” by Alberto Compagnoni et al. from the University of Modena and Reggio Emilia introduces CHAIR-DPO, a preference optimization method that leverages the CHAIR metric to reduce visual hallucinations without compromising performance. These works highlight the critical need for more nuanced approaches to hallucination, moving beyond simple fixes to address underlying mechanisms.applications are also seeing breakthroughs. “PreGenie: An Agent Powered by Page Graph” by Weizhi Chen et al. from Zhejiang University, introduces a multi-agent framework that uses page graphs to enhance GUI navigation, demonstrating improved generalization with limited data. For video understanding, “Kwai Keye-VL 1.5 Technical Report” from Kuaishou Group presents a multimodal foundation model with a Slow-Fast video encoding strategy and progressive pre-training, enabling it to efficiently handle complex visual content and long contexts up to 128K tokens. Addressing critical real-world issues, “Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety” by Younggun Kim et al. from the University of Central Florida, introduces Safe-LLaVA and the PRISM benchmark, specifically designed to address and mitigate biometric data leakage in MLLMs, a crucial step towards ethical AI.### Under the Hood: Models, Datasets, & Benchmarksadvancements detailed above are often underpinned by novel architectural designs, specialized datasets, and rigorous benchmarks. Here’s a look at the resources fueling this progress:Architectures & Frameworks:FtZ (Fusion to Enhance): A vision tower framework leveraging Multi-Head Cross-Attention to dynamically fuse features from multiple pre-trained vision experts (from Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Model).Sharp-Eyed Refinement with DeepLens: A lightweight image captioning framework mimicking human visual attention with a focused second glance (One More Glance with Sharp Eyes: Rethinking Lightweight Captioning as a Practical Visual Specialist).R-4B: An auto-thinking MLLM with a bi-mode annealing training paradigm and Bi-mode Policy Optimization (BPO) for adaptive strategy selection (R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning). Code available at https://github.com/yannqi/R-4B and https://huggingface.co/YannQi/R-4B.PreGenie: An agentic framework based on MLLMs for generating high-quality visual presentations, integrating intermediate code review and visual page review (PreGenie: An Agentic Framework for High-quality Visual Presentation Generation). Code available at https://github.com/hkust-ai-pregenie/pregenie.Kwai Keye-VL-1.5: A multimodal foundation model for video understanding, featuring a Slow-Fast video encoding strategy and progressive pre-training (from Kwai Keye-VL 1.5 Technical Report).OmniActor: A generalist GUI and embodied agent utilizing Layer-heterogeneity MoE to balance synergy and conflict between 2D and 3D tasks (OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds). Code available at https://github.com/meituan/OmniActor.ZoomEye: A training-free, model-agnostic tree search algorithm for vision-level reasoning in MLLMs, simulating human-like zooming capabilities (ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration). Code available at https://github.com/om-ai-lab/ZoomEye.PREMIR: A zero-shot multimodal document retrieval framework using cross-modal question generation to mitigate domain shift (Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation). Code available at https://huggingface.co/datasets/allganize/.AVAM: A universal training-free Adaptive Visual Anchoring strategy for efficient visual token compression in multi-image VQA (AVAM: Universal Training-free Adaptive Visual Anch anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering).Key Datasets & Benchmarks:Safe-LLaVA & PRISM: The first privacy-preserving MLLM training dataset, systematically cleaned to remove biometric cues, and a benchmark for evaluating biometric safety (Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety). Dataset and code at https://huggingface.co/datasets/kyh9191/Safe-LLaVA.Misleading ChartQA, Misviz, & Misviz-synth: Benchmarks for detecting and reasoning about misleading chart visualizations, including real-world and synthetic examples (from Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering and Is this chart lying to me? Automating the detection of misleading visualizations). Code available at https://github.com/CinderD/MisleadingChartQA and https://github.com/UKPLab/arxiv2025-misviz.Multimodal Uncertainty Benchmark (MUB): A new benchmark with varying difficulty levels to quantify MLLM response uncertainty under misleading scenarios (Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios). Code available at https://github.com/Yunkaidang/uncertainty.REACHQA: The first fully LLM-synthesized reasoning-intensive chart Q&A dataset focusing on both visual recognition and reasoning abilities (Distill Visual Chart Reasoning Ability from LLMs to MLLMs). Code available at https://github.com/hewei2001/ReachQA.ELV-Halluc: The first long-video hallucination benchmark, focused on Semantic Aggregation Hallucination (SAH) (ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding). Code available at https://github.com/hlsv02/ELV-Halluc.Med-RewardBench: The first comprehensive benchmark for evaluating reward models and judges in medical multimodal LLMs, covering six clinically critical dimensions (Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models).Percept-V: A novel dataset for evaluating basic visual perception abilities using program-generated images (Can Multimodal LLMs Solve the Basic Perception Problems of Percept-V?).MaRVL-QA: A benchmark for mathematical reasoning over visual landscapes, specifically designed for spatial and mathematical reasoning in MLLMs (MaRVL-QA: A Benchmark for Mathematical Reasoning over Visual Landscapes). Code and dataset available on Hugging Face and GitHub.MTMEUR: A multi-turn multimodal emotion understanding and reasoning benchmark with real-world video data and progressive questions (Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark). Code available at https://github.com/MindIntLab-HFUT/MTMEUR.WebMMU: A multilingual benchmark for website visual question answering, code editing, and mockup-to-code generation (WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation).AudioCodecBench & CodecBench: Comprehensive benchmarks for evaluating audio codecs across acoustic and semantic dimensions (AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation and CodecBench: A Comprehensive Benchmark for Acoustic and Semantic Evaluation). Code available at https://github.com/wuzhiyue111/Codec-Evaluation and https://github.com/RayYuki/CodecBench.MulSeT: A benchmark for evaluating MLLMs on multi-view spatial understanding tasks (Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture). Dataset at https://huggingface.co/datasets/WanyueZhang/MulSeT and code at https://github.com/WanyueZhang-ai/spatial-understanding.SAPA-Bench: A benchmark for evaluating privacy awareness in MLLM-powered smartphone agents (Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agents). Resources at https://zhixin-l.github.io/SAPA-Bench.### Impact & The Road Aheadcollective impact of this research is profound, accelerating the development of MLLMs that are not only more powerful but also more reliable, efficient, and aligned with human values. The focus on robust safety mechanisms, interpretability, and privacy-preserving approaches, exemplified by Safe-LLaVA and the new insights into hallucination, is crucial for building trustworthy AI. Advancements in reasoning and perception, from fine-grained visual understanding to multi-turn emotion analysis, unlock new possibilities for AI in diverse fields such as autonomous driving (Vehicle-to-Infrastructure Collaborative Spatial Perception via Multimodal Large Language Models), urban planning (From Drone Imagery to Livability Mapping: AI-powered Environment Perception in Rural China), and medical diagnostics (Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models, Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Study).road ahead for MLLMs involves scaling these innovations. We’ll likely see further research into efficient adaptation techniques, such as those leveraging pixel-level visual prompts (Parameter-Efficient Adaptation of mPLUG-Owl2 via Pixel-Level Visual Prompts for NR-IQA), and training-free frameworks (Scale, Don’t Fine-tune: Guiding Multimodal LLMs for Efficient Visual Place Recognition at Test-Time). The development of more sophisticated synthetic data generation methods, like Code-as-Intermediary Translation for chart reasoning, will also be vital for overcoming data scarcity in complex domains. The future of MLLMs is bright, promising AI systems that not only understand our world but also interact with it in increasingly intelligent and beneficial ways.