Image Segmentation: From Foundational Models to Trustworthy Medical AI and Beyond

Latest 50 papers on image segmentation: Oct. 27, 2025

Image segmentation, the critical task of partitioning an image into meaningful regions, remains a cornerstone of computer vision and a perpetually evolving field in AI/ML. Its applications span from autonomous driving and environmental monitoring to, most notably, medical diagnostics. Recent research showcases a fascinating blend of foundational model adaptation, novel architectural designs, and a strong push towards robustness and interpretability, particularly in high-stakes clinical settings.

The Big Idea(s) & Core Innovations

The latest breakthroughs reveal a concerted effort to enhance segmentation models’ efficiency, generalizability, and trustworthiness. A significant theme is the leveraging and adaptation of large pre-trained models, a trend exemplified by the Segment Anything Model (SAM) and its successors. For instance, ARGenSeg: Image Segmentation with Autoregressive Image Generation Model from Ant Group, introduces a novel framework integrating segmentation into Multimodal Large Language Models (MLLMs) using an autoregressive image generation paradigm. This eliminates the need for task-specific heads, using continuous visual tokens for fine-grained, efficient segmentation. Similarly, SAM2LoRA: Composite Loss-Guided, Parameter-Efficient Finetuning of SAM2 for Retinal Fundus Segmentation effectively adapts SAM2 for retinal fundus segmentation, showcasing parameter-efficient fine-tuning with composite loss functions. Meanwhile, SAM2-3dMed: Empowering SAM2 for 3D Medical Image Segmentation by researchers at Beijing Jiaotong University, bridges the domain gap between video-based SAM2 and 3D medical data through novel modules for spatial dependency modeling and boundary precision.

Another crucial innovation revolves around enhancing robustness and generalization, especially for medical imaging where data scarcity and domain shifts are prevalent. TreeFedDG: Alleviating Global Drift in Federated Domain Generalization for Medical Image Segmentation from institutions including Central South University and Zhejiang University, addresses global drift in federated learning through hierarchical parameter aggregation and style mixing. For semi-supervised scenarios, BARL: Bilateral Alignment in Representation and Label Spaces for Semi-Supervised Volumetric Medical Image Segmentation by Huazhong University of Science and Technology, proposes aligning representations and labels in dual spaces, significantly improving accuracy with limited labeled data. Carnegie Mellon University and other institutions, in DuetMatch: Harmonizing Semi-Supervised Brain MRI Segmentation via Decoupled Branch Optimization, introduce a dual-branch semi-supervised framework using asynchronous optimization and decoupled training to improve robustness and generalization in brain MRI segmentation.

The drive for interpretability and efficiency is also prominent. From Segments to Concepts: Interpretable Image Classification via Concept-Guided Segmentation from Bar-Ilan University introduces SEG-MIL-CBM, which integrates concept-guided segmentation for spatially grounded, interpretable image classification without explicit concept annotations. MedVKAN: Efficient Feature Extraction with Mamba and KAN for Medical Image Segmentation by Shaoxing University, combines Mamba and Kolmogorov-Arnold Networks (KAN) for highly efficient feature extraction, outperforming Transformers on medical datasets. FuseUNet: A Multi-Scale Feature Fusion Method for U-like Networks from Sichuan University, reimagines UNet decoding as an initial value problem, leveraging numerical methods for efficient multi-scale feature fusion with reduced parameters.

Addressing the challenge of annotation scarcity, GenCellAgent: Generalizable, Training-Free Cellular Image Segmentation via Large Language Model Agents from Brookhaven National Laboratory, presents a training-free multi-agent system that uses LLMs for generalizable cellular image segmentation, adapting to novel objects without retraining. Similarly, Uncertainty-Aware Extreme Point Tracing for Weakly Supervised Ultrasound Image Segmentation from the University of Science and Technology, China, uses extreme points to generate high-quality pseudo-labels, drastically reducing annotation costs.

Under the Hood: Models, Datasets, & Benchmarks

These advancements are powered by innovative model architectures, specialized datasets, and rigorous benchmarking:

• ARGenSeg leverages a pre-trained VQ-VAE tokenizer for continuous visual tokens, unifying segmentation into MLLMs. It achieves state-of-the-art results without specialized heads.
• FuseUNet (Code: https://github.com/nayutayuki/FuseUNet) uses linear multistep methods and nmODEs for efficient cross-scale feature fusion in UNet-like networks, demonstrating performance across medical segmentation tasks.
• Training-Free Framework for Open-Vocabulary Segmentation (Paper: https://arxiv.org/pdf/2510.19333) utilizes pre-trained EfficientNet and CLIP models to reduce the need for labeled data in open-vocabulary tasks.
• IC-MoE (Paper: https://arxiv.org/pdf/2510.17684) introduces an Intelligent Communication Mixture-of-Experts for medical image segmentation, enhancing high-level feature representation and employing semantic-guided contrastive learning.
• U-DFA (Code: https://github.com/sajjad/U-DFA) combines DINOv2 and UNet with a Dual Fusion Attention mechanism, achieving state-of-the-art on Synapse and ACDC datasets with high parameter efficiency.
• SAMA-UNet (Code: https://github.com/sqbqamar/SAMA-UNet) from Sohar University and others, integrates Self-Adaptive Mamba-like Attention and Causal-Resonance Learning, outperforming existing CNN, Transformer, and Mamba-based methods on BTCV, ACDC, EndoVis17, and ATLAS23 datasets.
• MedVKAN (Code: https://github.com/beginner-cjh/MedVKAN) substitutes Transformer modules with a hybrid VKAN block, combining Mamba and KAN for efficient medical image segmentation across multiple public datasets.
• CURVAS Challenge (Code: https://curvas.grand-challenge.org/) evaluates models for multi-organ segmentation with a focus on calibration and uncertainty, highlighting multi-rater variability’s importance. It provides a comprehensive framework to assess model performance.
• U-Bench (Code: https://github.com/FengheTan9/U-Bench, https://huggingface.co/FengheTan9/U-Bench) introduces a new benchmark and U-Score metric for evaluating over 100 U-Net variants across 28 datasets, open-sourcing all resources for reproducibility.
• MetaSeg (Code: https://github.com/KVyas/MetaSeg) employs implicit neural representations (INRs) and meta-learning for one-shot medical image segmentation, achieving near state-of-the-art results with minimal fine-tuning on unseen images.
• GenCellAgent (Code: https://github.com/yuxi120407/GenCellAgent) is a training-free multi-agent LLM system for cellular image segmentation, demonstrating significant accuracy gains across benchmarks like LIVECell and TissueNet, even on novel objects.

Impact & The Road Ahead

These advancements collectively pave the way for more robust, efficient, and interpretable AI systems across diverse applications. In medical imaging, the focus on reducing annotation burden, improving cross-domain generalization, and enhancing trustworthiness is critical for clinical adoption. Frameworks like AutoMiSeg (https://arxiv.org/pdf/2505.17931) and WS-ICL (https://arxiv.org/pdf/2510.05899) promise to democratize high-accuracy segmentation by minimizing reliance on extensive labeled data, a bottleneck in healthcare AI. The emphasis on uncertainty quantification, as seen in Progressive Uncertainty-Guided Evidential U-KAN and the CURVAS challenge, fosters trust, enabling AI systems to indicate when they are unsure, a vital feature for diagnostic tools.

Beyond medicine, open-vocabulary and vision-language integration (Refer to Any Segmentation Mask Group With Vision-Language Prompts, DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation, SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation) are expanding segmentation’s reach into more complex, human-like interactions. The ability to segment based on natural language prompts opens doors for more intuitive human-AI collaboration in content creation, robotics, and complex scene understanding. In remote sensing, methods like SAIP-Net (https://arxiv.org/pdf/2504.16564) improve the accuracy of environmental monitoring, crucial for climate change initiatives and urban planning.

The road ahead will likely see continued exploration of large foundation models, further integration of human-in-the-loop strategies, and increasingly sophisticated methods for uncertainty estimation. The burgeoning intersection of classical numerical methods with deep learning, as demonstrated by FuseUNet and MedVKAN, also suggests exciting new architectural paradigms. The commitment to open-sourcing models and benchmarks, championed by initiatives like U-Bench, will accelerate this progress. Image segmentation, far from being a solved problem, is entering a new era of generalizable, reliable, and intelligent systems.