Segment Anything Model (SAM) and Its Evolution: From Generalist to Specialist in AI Vision

Latest 50 papers on segment anything model: Sep. 21, 2025

The Segment Anything Model (SAM), a true marvel in computer vision, has redefined the landscape of image segmentation with its powerful zero-shot capabilities. Initially lauded for its ability to segment anything with remarkable generality, recent research showcases an exciting evolution: how SAM, and its successor SAM2, are being repurposed, enhanced, and specialized to tackle complex, domain-specific challenges. This blog post dives into a collection of recent breakthroughs, exploring how researchers are pushing the boundaries of what these foundation models can achieve.

The Big Idea(s) & Core Innovations

At its heart, the recent research on SAM revolves around adapting its potent, generalist segmentation power to more nuanced and demanding tasks. A central theme is efficiency and precision in domain-specific applications, often addressing challenges where generic segmentation falls short due to semantic ambiguity, data scarcity, or complex physical properties.

For instance, in medical imaging, researchers are making significant strides. The SAMIR: an efficient registration framework via robust feature learning from SAM from Hunan University leverages SAM’s robust features for accurate medical image registration, achieving state-of-the-art performance in cardiac and abdominal CT scans by enhancing anatomical consistency. Similarly, PG-SAM: Multi-Sequence Parotid Gland Lesion Segmentation via Expert Text-Guided Segment Anything Model by Sun Yat-Sen University integrates expert diagnostic reports as prompts, eliminating the need for extensive manual annotations and improving multi-sequence parotid gland lesion segmentation. This emphasis on integrating domain knowledge through intelligent prompting is echoed in AutoSAME: Think as Cardiac Sonographers: Marrying SAM with Left Ventricular Indicators Measurements According to Clinical Guidelines, which combines SAM with clinical guidelines for automated left ventricular quantification in echocardiography. And for more nuanced segmentation, A Probabilistic Segment Anything Model for Ambiguity-Aware Medical Image Segmentation from the University of Kentucky introduces Probabilistic SAM, which captures segmentation ambiguity through a learned latent space, providing uncertainty-aware outputs crucial for clinical reliability.

Beyond medicine, optimizing SAM for efficiency and fine-grained tasks is another major focus. Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation by Zhejiang University’s Xiaobo Yang and Xiaojin Gong proposes the Semantic Visual Projector (SVP), drastically reducing visual token redundancy in Multimodal Large Language Models (MLLMs) for referring image segmentation while preserving semantic clarity. In a similar vein, EdgeSAM: Prompt-In-the-Loop Distillation for SAM by Meta AI and Apple Inc. introduces a dynamic prompt-in-the-loop distillation strategy, enabling SAM to operate in real-time on edge devices with significant speed improvements.

The challenge of zero-shot generalization in new domains is addressed by several papers. U-SAM: Repurposing SAM for User-Defined Semantics Aware Segmentation from the University of California, Riverside, allows SAM to generate semantic masks for user-defined categories without manual supervision, leveraging synthetic or web-crawled images. For scientific imaging, Zenesis: Foundation Models for Zero-Shot Segmentation of Scientific Images without AI-Ready Data by Lawrence Berkeley National Laboratory offers a no-code platform for zero-shot and interactive segmentation on raw scientific data, showcasing the power of lightweight multimodal adaptation.

Finally, the integration of SAM with other advanced AI models is yielding powerful hybrid solutions. ABS-Mamba: SAM2-Driven Bidirectional Spiral Mamba Network for Medical Image Translation from an anonymized author combines SAM2’s global semantic understanding with Mamba’s efficient state-space modeling for high-fidelity medical image translation. CLUE: Leveraging Low-Rank Adaptation to Capture Latent Uncovered Evidence for Image Forgery Localization by Shenzhen University integrates LoRA-tuned Stable Diffusion 3 with SAM for robust image forgery detection, shifting focus from artifact detection to understanding generative principles of forgery.

Under the Hood: Models, Datasets, & Benchmarks

These innovations are built upon sophisticated models, novel datasets, and rigorous benchmarks:

• SAM/SAM2-based Architectures: Many papers directly build upon or adapt the Segment Anything Model (SAM) and its successor, SAM2. Examples include Probabilistic SAM, EdgeSAM, U-SAM, ABS-Mamba, FS-SAM2, CAV-SAM, and GeoSAM, each tailoring the foundation model for specific tasks.
• Hybrid Models: The trend towards combining SAM with other powerful models is evident:
  • YOLOv8 & SAM: Used in Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication for robust surgical tool tracking.
  • Faster R-CNN & SAM: Integrated in a Synthetic Data-Driven Multi-Architecture Framework for Automated Polyp Segmentation for enhanced polyp detection.
  • Stable Diffusion 3 (SD3) & SAM: Leveraged by CLUE for image forgery localization, utilizing SD3’s generative process and SAM’s semantic context.
  • Mamba & SAM2: Combined in ABS-Mamba for efficient medical image translation, blending global semantic understanding with long-range contextual modeling.
  • CLIP & SAM/SAM2: Used in EMeRALDS (https://arxiv.org/pdf/2509.11714) for zero-shot lung nodule segmentation and detection via text prompts, and in CLAPS (https://arxiv.org/pdf/2509.08618) for unified auto-prompt segmentation in retinal imaging.
  • Grounding DINO & SAM: Utilized by IAPF (https://arxiv.org/pdf/2508.06904) for training-free camouflaged object segmentation, generating instance-level masks from task-generic prompts.
• Novel Datasets: To address data scarcity and improve generalization, new datasets are emerging:
  • SA1B-Matte & MicroMat-3K: Introduced by ZIM: Zero-Shot Image Matting for Anything for micro-level matte labels and fine-grained zero-shot matting evaluation.
  • ReSOS dataset: Developed by SOPSeg: Prompt-based Small Object Instance Segmentation in Remote Sensing Imagery as the first large-scale instance segmentation benchmark for remote sensing small objects.
  • Osprey-724K: Created by Osprey: Pixel Understanding with Visual Instruction Tuning for mask-text instruction tuning, enhancing pixel-level understanding in MLLMs.
• Benchmarks & Evaluation: Papers consistently report superior performance on established benchmarks like PASCAL VOC, COCO, ACDC, abdomen CT datasets, CD-FSS, and various camouflaged object detection (COD) and remote sensing change detection datasets, often with significant percentage improvements in mIoU, F1-score, or C-index.

Impact & The Road Ahead

These advancements profoundly impact various fields, from medical diagnostics (faster, more accurate analyses of lung nodules, cardiac function, and parotid gland lesions) and robotics (improved human-robot interaction and precise robotic manipulation) to environmental monitoring (accurate olive tree segmentation, remote sensing change detection) and digital forensics (robust image forgery detection). The ability to perform zero-shot segmentation on novel data, often without extensive retraining or manual annotation, is a game-changer for domains with limited labeled data, such as scientific imaging and specialized industrial tasks. The development of efficient, deployable models like EdgeSAM ensures that powerful AI can run on resource-constrained edge devices, broadening accessibility and real-time application.

Looking ahead, the research points towards increasingly specialized yet adaptable foundation models. The emphasis on parameter-efficient fine-tuning (PEFT), knowledge distillation, and multimodal integration will continue to drive down computational costs while boosting performance. The emergence of physics-guided rewards in SAM* (https://arxiv.org/pdf/2509.07047) for microscopy segmentation, and uncertainty-aware models like Probabilistic SAM and E-BayesSAM (https://arxiv.org/pdf/2508.17408) for medical imaging, highlights a crucial shift towards more reliable and domain-informed AI. As we move forward, expect to see SAM and its descendants becoming even more embedded in real-world applications, offering intelligent, adaptable, and increasingly interpretable solutions to complex visual challenges across diverse industries.